CN106954060B - A kind of three-dimensional video-frequency generation method and auxiliary filming apparatus towards smart phone - Google Patents

Abstract

The invention discloses a kind of three-dimensional video-frequency generation method and auxiliary filming apparatus towards smart phone.It includes five steps that three-dimensional video-frequency of the present invention, which generates scheme: three-dimensional video-frequency assists filming apparatus building and installation；Three-dimensional video-frequency assists filming apparatus parameter automatic Calibration；Perspective geometry transformation matrix calculates and the estimation of grey level compensation mapping function；Video data capture；Three-dimensional video-frequency generates.It includes: stainless steel plane mirror, plane mirror fixed frame, bearing attachment part, bracket holder, mobile phone connecting component, fastener that three-dimensional video-frequency, which assists filming apparatus,.The method of the present invention can avoid the asynchronous caused left and right picture problem of disharmony of two cameras, and three-dimensional video-frequency of the present invention auxiliary filming apparatus have many advantages, such as it is of simple structure and low cost, easy to carry and easy to use, facilitate ordinary consumer and obtain the stronger three-dimensional video-frequency of feeling of immersion using smart phone, enriches smart phone visual form, promotes shooting and ornamental enjoyment.

Description

A stereoscopic video generation method and auxiliary shooting device for smart phone

technical field

The invention relates to the field of stereoscopic video imaging, in particular to a stereoscopic video generation method and an auxiliary shooting device for smart phones.

Background technique

Vision is an important source of information for humans to perceive the external environment, so images and videos play an important role in various application scenarios such as entertainment, education, and social interaction. However, ordinary images and videos are essentially two-dimensional projections of the three-dimensional world, and the depth information of the scene is lost during the imaging process, so it is difficult for ordinary images and videos to bring an immersive feeling to the observer. In recent years, stereoscopic (stereoscopy) video imaging technology has developed rapidly, which greatly enhances the three-dimensionality and realism of scenes. Stereoscopic video can present left and right views with parallax to the observer's left and right eyes, and these two views can provide the observer with complete scene depth cues after being synthesized by the observer's brain. At present, people can watch pre-recorded stereoscopic videos on smartphones with the help of auxiliary devices such as virtual reality glasses, which brings users an unprecedented sense of immersion and an excellent experience of watching movies.

Existing stereoscopic image and stereoscopic video acquisition technologies can be roughly divided into the following four categories: using computer vision and image processing methods to extract scene depth information from ordinary single-camera videos, and then generating left and right views with parallax on this basis; Use two side-by-side cameras with fixed relative positions for simultaneous shooting; use dual-lens cameras for stereoscopic video shooting; use a single camera with auxiliary devices for shooting. Each of these technical means has problems such as high cost, complex structure, and difficult operation, and cannot meet the needs of ordinary consumers to shoot stereoscopic videos with ordinary smartphones.

Aiming at the problems of high cost, complex structure and difficult operation of stereoscopic video shooting in the prior art, there is currently no effective solution.

SUMMARY OF THE INVENTION

In view of this, the purpose of the embodiments of the present invention is to provide a smart phone-oriented stereoscopic video generation method and auxiliary shooting device, which enables ordinary consumers to use ordinary smart phones to shoot stereoscopic videos, reduces cost, simplifies structure, and facilitates operation.

Based on the above purpose, an embodiment of the present invention provides a method for generating a stereoscopic video for a smartphone, including:

Build a stereoscopic video assist camera and mount it on a smartphone;

Calibrate the installation parameters of the stereoscopic video auxiliary shooting device;

Use smartphone to calculate projection geometric transformation matrix and grayscale compensation mapping function;

Capture video data with a smartphone;

Convert the captured video data into a stereoscopic video.

In some embodiments, the stereoscopic video auxiliary shooting device includes a stainless steel plane mirror for acquiring a virtual image of the scene, the plane where the stainless steel plane mirror is located is perpendicular to the back of the smartphone, the plane where the stainless steel plane mirror is located intersects with the back of the smartphone at the inner edge of the stainless steel plane mirror, and the stainless steel plane mirror is located at the inner edge of the stainless steel plane mirror. The inner edge of the plane mirror is perpendicular to the side of the smartphone; at the same time, the stainless steel plane mirror is fixed by the plane mirror fixing frame, the plane mirror fixing frame is connected to the bracket fixing frame through the bracket connecting part, the bracket fixing frame is connected with the smartphone through the mobile phone connecting part, and the plane mirror fixing frame Fasteners are used to fix or adjust the relative position and angle between the bracket connecting part and the bracket fixing frame and the mobile phone connecting part.

In some embodiments, the calibration of the installation parameters of the stereoscopic video auxiliary shooting device includes:

Use a smartphone to shoot an image, where the shooting scene corresponding to the image contains rich texture and color information, and cannot be a single-color scene such as a wall or desktop;

Extract multiple sparse feature points from the image and perform feature point matching to generate a feature point matching set;

Repeatedly select two pairs of feature points at random for matching, and generate the plane mirror normal vector estimation determined by the above-mentioned feature point matching;

The normal vector of the plane mirror is determined as the installation parameter by estimating the median value of each component according to the plurality of normal vectors.

In some embodiments, extracting a plurality of sparse feature points from the image and performing feature point matching, and generating a feature point matching set includes:

Extract a plurality of sparse feature points and a description vector of each sparse feature point from the image according to the image local feature detection and description algorithm;

Match each sparse feature point with other feature points in the image according to the distance ratio test feature matching method to generate matching feature point pairs;

A feature point matching set including at least 100 feature point pairs is established according to the plurality of sparse feature point pairs.

In some embodiments, the multiple repetitions of randomly selecting two pairs of feature points for matching, and generating the plane mirror normal vector determined by the above-mentioned feature point matching includes:

Determine the number of times of randomly selecting two pairs of feature points for matching according to the number of feature point pairs;

Two pairs of feature points are randomly selected for matching each time, and the normal vector determined by the above feature point matching is generated according to the image coordinates of the two pairs of feature points and the focal length of the smartphone camera.

In some embodiments, calculating the projection geometric transformation matrix and the grayscale compensation mapping function using a smartphone includes:

Generate an image transformation matrix according to the length and width of the video frame and the normal vector of the plane mirror;

Obtain the consistency index of all feature point pairs in the feature point matching set according to the focal length of the smartphone camera;

Collect all feature point pairs whose consistency index is greater than the pre-specified threshold, and establish a set of valid feature point matching pairs;

Obtain the image gray level of each feature point position in the effective feature point matching pair set, and fit the gray level compensation mapping function with the image gray level of each feature point position according to the cubic polynomial curve fitting algorithm.

In some implementation manners, when using a smartphone to capture video data, the angle between the lateral side of the mobile phone and the horizontal plane does not exceed 30 degrees.

In some embodiments, converting the captured video data into a stereoscopic video includes:

Decompose the video data by frame, and use the image transformation matrix for each frame of image to perform geometric correction and crop out blank areas;

Divide each frame of image into left view and right view of the same size, and perform left and right mirror transformation on the right view;

Color compensation for each pixel in the right view according to the grayscale compensation map;

The left and right views of each frame are adaptively processed according to the stereoscopic video observation mode, and the left and right views of each frame are merged into a stereoscopic video frame, and then all the stereoscopic video frames are merged into a stereoscopic video.

In some embodiments, the stereoscopic video observation mode includes at least one of the following: VR glasses, red and blue glasses, naked eyes; the adaptive processing of the left view and right view of each frame according to the stereoscopic video observation mode is based on the stereoscopic video The video resolution and aspect ratio specified by the viewing mode are adaptively processed for the left and right views of each frame.

Based on the above purpose, an embodiment of the present invention provides a stereoscopic video auxiliary shooting device oriented to a smartphone, including a stainless steel plane mirror for acquiring a virtual image of a scene, the plane where the stainless steel plane mirror is located is perpendicular to the back of the smartphone, and the plane where the stainless steel plane mirror is located is parallel to the smartphone. The back intersects with the inner edge of the stainless steel plane mirror, and the inner edge of the stainless steel plane mirror is perpendicular to the side of the smartphone; at the same time, the stainless steel plane mirror is fixed by the plane mirror fixing frame, and the plane mirror fixing frame is connected to the bracket fixing frame through the bracket connecting part, and the bracket fixing frame is connected by the mobile phone connecting part. It is connected with the smart phone, and the relative position and angle are fixed or adjusted by fasteners between the plane mirror fixing frame and the bracket connecting part, and between the bracket fixing frame and the mobile phone connecting part.

It can be seen from the above that the stereoscopic video generation method for smartphones provided by the embodiments of the present invention only needs one camera to shoot stereoscopic videos, so the problem of uncoordinated left and right images caused by the two cameras being out of synchronization can be avoided, and the present invention The stereoscopic video auxiliary shooting device of the invention has the advantages of simple structure, low cost, easy portability and easy use. Therefore, the method and device of the present invention can help ordinary consumers to obtain a stereoscopic video with a strong sense of immersion by using a smart phone, enrich the video form of the smart phone, and improve the fun of shooting and viewing.

Description of drawings

In order to explain the embodiments of the present invention or the technical solutions in the prior art more clearly, the following briefly introduces the accompanying drawings that need to be used in the description of the embodiments or the prior art. Obviously, the accompanying drawings in the following description are only These are some embodiments of the present invention. For those of ordinary skill in the art, other drawings can also be obtained according to these drawings without creative efforts.

1 is a flow chart of a method for generating a stereoscopic video for a smartphone provided by the present invention;

2 is a structural diagram of a stereoscopic video auxiliary shooting device in a smartphone-oriented stereoscopic video generation method provided by the present invention;

3 is an imaging principle diagram of a smartphone-oriented stereoscopic video generation method provided by the present invention;

4 is an estimated distribution diagram of a plane mirror normal vector in an embodiment of the smartphone-oriented stereoscopic video generation method provided by the present invention;

5 is a graph of a fitting result of a grayscale compensation function in an embodiment of a smartphone-oriented stereoscopic video generation method provided by the present invention;

6 is a diagram of a video frame and a geometric correction result captured by a smartphone equipped with a stereoscopic video auxiliary shooting device in an embodiment of the smartphone-oriented stereoscopic video generation method provided by the present invention;

Fig. 7 is an embodiment of the stereoscopic video generation method for smartphones provided by the present invention, the left view obtained by cutting out the geometric correction video, the right view without color compensation, and the right view after color compensation;

FIG. 8 is a screenshot of a cross-eye stereoscopic video suitable for naked-eye viewing and a left-right stereoscopic video suitable for viewing with VR glasses, in an embodiment of the smartphone-oriented stereoscopic video generation method provided by the present invention.

Detailed ways

In order to make the objectives, technical solutions and advantages of the present invention more clearly understood, the embodiments of the present invention will be further described in detail below with reference to the specific embodiments and the accompanying drawings.

It should be noted that all expressions using "first" and "second" in the embodiments of the present invention are for the purpose of distinguishing two entities with the same name but not the same or non-identical parameters. It can be seen that "first" and "second" It is only for the convenience of expression and should not be construed as a limitation to the embodiments of the present invention, and subsequent embodiments will not describe them one by one.

In an embodiment of the present invention, FIG. 1 shows a schematic flowchart of a first embodiment of a method for generating stereoscopic video for a smartphone provided by the present invention.

As shown in Figure 1, the stereoscopic video generation method for smartphones includes:

Step S101, constructing a stereoscopic video auxiliary shooting device and installing it on a smartphone;

Step S103, calibrating the installation parameters of the stereoscopic video auxiliary shooting device;

Step S105, use the smartphone to calculate the projection geometric transformation matrix and the grayscale compensation mapping function;

Step S107, using a smartphone to capture video data;

In step S109, the captured video data is converted into a stereoscopic video.

In some embodiments, the stereoscopic video auxiliary shooting device includes a stainless steel plane mirror 1 for acquiring a virtual image of a scene, the plane where the stainless steel plane mirror 1 is located is perpendicular to the back of the smartphone 0, and the plane where the stainless steel plane mirror 1 is located intersects with the back of the smartphone 0 on the stainless steel plane. The inner edge of the plane mirror 1, the inner edge of the stainless steel plane mirror 1 is perpendicular to the side of the smartphone 0; at the same time, the stainless steel plane mirror 1 is fixed by the plane mirror fixing frame 2, and the plane mirror fixing frame 2 is connected to the bracket fixing frame 4 through the bracket connecting part 3, and the bracket fixing frame 4 is connected to the smartphone 0 through the mobile phone connecting part 5, and the relative position and angle are fixed or adjusted with fasteners 6 between the plane mirror fixing frame 2 and the bracket connecting part 3, and between the bracket fixing frame 4 and the mobile phone connecting part 5.

In some embodiments, the calibration of the installation parameters of the stereoscopic video auxiliary shooting device includes:

Use a smartphone to shoot an image, where the shooting scene corresponding to the image contains rich texture and color information, and cannot be a single-color scene such as a wall or desktop;

Extract multiple sparse feature points from the image and perform feature point matching to generate a feature point matching set;

Repeatedly select two pairs of feature points at random to match, and generate the plane mirror normal vector determined by the above-mentioned feature point matching;

The normal vector of the plane mirror is determined as the installation parameter by estimating the median value of each component according to the matching normal vector of multiple feature points.

In some embodiments, extracting a plurality of sparse feature points from the image and performing feature point matching, and generating a feature point matching set includes:

Extract a plurality of sparse feature points and a description vector of each sparse feature point from the image according to the image local feature detection and description algorithm;

Match each sparse feature point with other feature points in the image according to the distance ratio test feature matching method to generate matching feature point pairs;

A feature point matching set including at least 100 feature point pairs is established according to the plurality of sparse feature point pairs.

In some embodiments, the multiple repetitions of randomly selecting two pairs of feature points for matching, and generating the plane mirror normal vector determined by the above-mentioned feature point matching includes:

Determine the number of times of randomly selecting two pairs of feature points for matching according to the number of feature point pairs;

Two pairs of feature points are randomly selected for matching each time, and the normal vector determined by the above feature point matching is generated according to the image coordinates of the two pairs of feature points and the focal length of the smartphone camera.

In some embodiments, calculating the projection geometric transformation matrix and the grayscale compensation mapping function using a smartphone includes:

Generate an image transformation matrix according to the length and width of the video frame and the normal vector of the plane mirror;

Obtain the consistency index of all feature point pairs in the feature point matching set according to the focal length of the smartphone camera;

Collect all feature point pairs whose consistency index is greater than the pre-specified threshold, and establish a set of valid feature point matching pairs;

Obtain the image gray level of each feature point position in the effective feature point matching pair set, and fit the gray level compensation mapping function with the image gray level of each feature point position according to the cubic polynomial curve fitting algorithm.

In some implementation manners, when using a smartphone to capture video data, the angle between the lateral side of the mobile phone and the horizontal plane does not exceed 30 degrees.

In some embodiments, converting the captured video data into a stereoscopic video includes:

Decompose the video data by frame, and use the image transformation matrix for each frame of image to perform geometric correction and crop out blank areas;

Divide each frame of image into left view and right view of the same size, and perform left and right mirror transformation on the right view;

Color compensation for each pixel in the right view according to the grayscale compensation map;

The left and right views of each frame are adaptively processed according to the stereoscopic video observation mode, and the left and right views of each frame are merged into a stereoscopic video frame, and then all the stereoscopic video frames are merged into a stereoscopic video.

In some embodiments, the stereoscopic video observation mode includes at least one of the following: VR glasses, red and blue glasses, naked eyes; the adaptive processing of the left view and right view of each frame according to the stereoscopic video observation mode is based on the stereoscopic video The video resolution and aspect ratio specified by the viewing mode are adaptively processed for the left and right views of each frame.

It can be seen from the above that the stereoscopic video generation method for smartphones provided by the embodiments of the present invention only needs one camera to shoot stereoscopic videos, so the problem of uncoordinated left and right images caused by the two cameras being out of synchronization can be avoided, and the present invention The stereoscopic video auxiliary shooting device of the invention has the advantages of simple structure, low cost, easy portability and easy use. Therefore, the method and device of the present invention can help ordinary consumers to obtain a stereoscopic video with a strong sense of immersion by using a smart phone, enrich the video form of the smart phone, and improve the fun of shooting and viewing.

In another embodiment of the present invention, the smartphone-oriented stereoscopic video generation method includes:

In step S101, a stereoscopic video auxiliary shooting device is constructed and installed on a smart phone.

The structure diagram of the stereoscopic video auxiliary shooting device for smartphones according to the embodiment of the present invention is shown in FIG. 2 , including: a stainless steel plane mirror 1 , a plane mirror fixing frame 2 , a bracket connecting part 3 , a bracket fixing frame 4 , a mobile phone connecting part 5 , a fastener 6. Among them, the stainless steel plane mirror 1 is used to obtain the virtual image of the scene; the plane mirror fixing frame 2 is used to fix the stainless steel plane mirror 1; the bracket connecting part 3 is used to connect the plane mirror fixing frame 2 and the bracket fixing frame 4; the bracket fixing frame 4 is used to fix the bracket connecting part 3; the mobile phone connecting part 5 is used to fix the stereoscopic video auxiliary shooting device on the smart phone 0; the fastener 6 is used to fix and adjust the relative position and angle of the fixing frame 4 and the connecting part 5.

The mobile phone used in the embodiment of the present invention is the iphone 4s mobile phone, and the specific method is as follows: first, the stereoscopic video auxiliary shooting device is fixed on the back of the mobile phone through the mobile phone connecting part; then, the plane mirror, the bracket connecting part and the Bracket holder; ensure that the plane where the plane mirror is located is roughly perpendicular to the back of the phone, the edge of the plane mirror is as close as possible to the back of the phone, the intersection line between the plane mirror and the back of the phone is about 1-3 mm from the center of the camera on the back of the phone, and the plane mirror and the back of the phone The intersecting line is roughly perpendicular to the long side of the phone. Through the above connection, fixation, and adjustment methods, it is finally ensured that the real image and the virtual image of the scene with substantially the same range and size can be observed from the mobile phone at the same time.

Step S103, calibrating installation parameters of the stereoscopic video auxiliary shooting device.

FIG. 3 shows a schematic diagram of stereoscopic video shooting and imaging for a smartphone according to an embodiment of the present invention. Due to possible errors in the installation of the auxiliary camera, the normal vector n of the plane mirror and the X-axis direction of the camera (that is, the direction of the long side of the mobile phone) cannot be completely coincident. Calibration is performed. The calibration process only needs to be performed once. After the calibration is completed, multiple video data can be captured. The calibration is not required until the stereoscopic video auxiliary shooting device is adjusted manually, or the lighting conditions or scene content have changed greatly. The specific steps of normal vector calibration of the plane mirror are as follows:

First, use a smartphone equipped with a stereoscopic video auxiliary camera to capture an image, denoted as The shooting scene corresponding to the above image needs to contain rich texture and color information, and cannot be a scene with a single color such as a wall or a desktop.

Secondly, using SIFT image local feature detection and description algorithm to extract sparse feature points and a description vector for each feature point. For each feature point, use the distance ratio test feature matching method to obtain its For matching feature points in all feature points (excluding the current feature point itself), the value of the distance ratio threshold τ is 0.6. Note that the final number of feature point matching is N, and the matching set of feature points is in, is the image feature point coordinates (unit is pixel), the origin of the image coordinate system is located at the center of the image, the x-axis points to the right along the long side of the image, and the y-axis points down along the short side of the image. If N<100, then use the smartphone equipped with the stereoscopic video auxiliary shooting device to shoot an image again, and perform this step again.

Again, repeatedly perform N _it times the plane mirror candidate normal vector estimation operation, the jth normal direction estimation operation includes the following two steps (N _it =min(N(N-1)/2,1000), j={1,2 ,…N _it }): First, randomly select two pairs of feature points to match, denoted as Second, calculate the normal vector determined by the above feature point matching Calculated as follows:

Among them, f represents the focal length of the camera (unit is pixel), and the symbol "×" represents the cross product of the vector.

Finally, the final normal vector n of the plane mirror is calculated, and the calculation formula is as follows:

Here, the notation "[·] _k " denotes the kth element of the vector. If [n] ₁ < 0, let n←-n. FIG. 4 shows one embodiment of plane mirror normal vector estimation. As can be seen from Figure 4, there are many noises and errors in the candidate normal vector estimation, but the middle regions are concentrated.

Step S105, use the smartphone to calculate the projection geometric transformation matrix and the grayscale compensation mapping function.

First, calculate the image transformation matrix H, the calculation formula is as follows:

Among them, w and h represent the width and height of the video frame (in pixels), respectively, and R represents the rotation around the rotation axis (0,[n] ₃ ,-[n] ₂ ) ^T rotation angle arccos([n] ₁ ) matrix.

Secondly, the estimation of the grayscale compensation mapping function includes the following two steps: First, obtain a set of valid feature point matching pairs First, let Initialize the set of valid feature point matching pairs. Then, each feature point matching pair is calculated according to the following formula The consistency index δ of (i={1,2,...,N}):

if will from deleted in . Among them, the value of the parameter θ is 3°. Record the final valid feature point matching pair set Second, construct the virtual-real grayscale compensation mapping function: First, create N _in two-tuples (j=1,2,..., _Nin ). Among them, g(·) represents the gray value of the image at the position of the image feature point. Using a cubic polynomial curve fitting algorithm, according to the two-tuple Fit grayscale compensation mapping function The value range of s is 0 to 255. The value range is 0 to 255. FIG. 5 shows an embodiment of the fitting result of the grayscale compensation function. As shown in Figure 5, Present nonlinear characteristics, and different scenarios correspond to vary.

Step S107, using a smartphone to capture video data.

The video is shot by a smartphone equipped with a stereoscopic video auxiliary shooting device. After the parameter calibration is completed, the camera focal length of the camera and the relative position of each component of the stereoscopic video auxiliary shooting device are kept unchanged. The smartphone camera is used to shoot video data. The angle between the long side of the mobile phone and the horizontal plane is less than 30°.

In step S109, the captured video data is converted into a stereoscopic video.

Perform geometric correction, transformation, and cropping operations on each frame of the captured video to obtain the left and right views of the stereoscopic video frame, and synthesize the device-related stereoscopic video. The specific methods are:

First, for any frame in the video (denoted as ), follow the steps below to generate video frames Corresponding left and right view

First, create a geometrically rectified video frame of width w _p and height h _p w _p and h _p are respectively calculated by the following formulas:

Among them, u' _ll =H(-w/2,h/2,1) ^T , u' _ul =H(-w/2,-h/2,1) ^T , u' _lr =H(w/2 , h/2,1) ^T , u′ _ur =H(w/2,-h/2,1) ^T . image The color value of any pixel position (m,n) in is equal to middle pixel position color value. in,

The mobile phone used in this embodiment is an iphone 4s mobile phone, and the aspect ratio of the video frame is 4:3. The left picture of Fig. 6 is an example of a video frame captured by a smartphone equipped with an auxiliary photographing device of the present invention. It can be seen from the left picture of Fig. 6 that since the matching lines of the feature points of the installation error are not parallel, it cannot be rotated by a simple image. Realize geometric correction. The right picture of FIG. 6 shows the result of geometric correction of video frames realized by the method of the present invention, and a video frame with almost no horizontal parallax is obtained.

Second, create video frames Corresponding left and right view and and corresponding to the image The left and right halves of the Perform left and right mirror transformations.

Third, for For each pixel of , assuming that its RGB color values are R, G, and B, respectively, the RGB color values of the pixel after color compensation are where t=0.299R+0.587G+0.114B. Figure 7 shows examples of left and right views obtained by cutting from a geometrically rectified video, wherein the left picture is the left view, the middle picture is the right view without color compensation, and the right picture is the right view after color compensation. It can be seen from FIG. 7 that the color correction method of the present invention can effectively compensate for the loss of light reflection.

Then, for stereoscopic video observation methods (such as VR glasses, naked-eye observation, red and blue glasses), the left and right views are compared according to the video resolution, aspect ratio and other parameters. and Crop, then merge the left and right views of each frame into corresponding stereoscopic video frames, and finally combine all stereoscopic video frames into a stereoscopic video. FIG. 8 shows an example of a stereoscopic video frame with an aspect ratio of 16:9 generated according to the solution of the present invention. The left picture of FIG. 8 is a cross-eye stereoscopic video suitable for naked eye observation, and the right picture of FIG. 8 is a left and right stereoscopic video suitable for viewing with VR glasses.

It can be seen from the above that the stereoscopic video generation method for smartphones provided by the embodiments of the present invention only needs one camera to shoot stereoscopic videos, so the problem of uncoordinated left and right images caused by the two cameras being out of synchronization can be avoided, and the present invention The stereoscopic video auxiliary shooting device of the invention has the advantages of simple structure, low cost, easy portability and easy use. Therefore, the method and device of the present invention can help ordinary consumers to obtain a stereoscopic video with a strong sense of immersion by using a smart phone, enrich the video form of the smart phone, and improve the fun of shooting and viewing.

In yet another embodiment of the present invention, the structure diagram of the stereoscopic video auxiliary shooting device for smartphones is shown in FIG. 2 , including: a stainless steel plane mirror 1 , a plane mirror fixing frame 2 , a bracket connecting part 3 , a bracket fixing frame 4 , a mobile phone connection Part 5, Fastener 6. Among them, the stainless steel plane mirror 1 is used to obtain the virtual image of the scene; the plane mirror fixing frame 2 is used to fix the stainless steel plane mirror 1; the bracket connecting part 3 is used to connect the plane mirror fixing frame 2 and the bracket fixing frame 4; the bracket fixing frame 4 is used to fix the bracket connecting part 3; the mobile phone connecting part 5 is used to fix the stereoscopic video auxiliary shooting device on the smart phone 0; the fastener 6 is used to fix and adjust the relative position and angle of the fixing frame 4 and the connecting part 5.

It can be seen from the above that the stereoscopic video generation method for smartphones provided by the embodiments of the present invention only needs one camera to shoot stereoscopic videos, so the problem of uncoordinated left and right images caused by the two cameras being out of synchronization can be avoided, and the present invention The stereoscopic video auxiliary shooting device of the invention has the advantages of simple structure, low cost, easy portability and easy use. Therefore, the method and device of the present invention can help ordinary consumers to obtain a stereoscopic video with a strong sense of immersion by using a smart phone, enrich the video form of the smart phone, and improve the fun of shooting and viewing.

Claims (8)

1. A stereoscopic video generation method facing to a smart phone is characterized by comprising the following steps:

constructing a stereoscopic video auxiliary shooting device and installing the stereoscopic video auxiliary shooting device on a smart phone;

calibrating the installation parameters of the stereoscopic video auxiliary shooting device;

calculating a projection geometric transformation matrix and a gray compensation mapping function by using a smart phone;

shooting video data by using a smart phone;

converting the shot video data into a three-dimensional video;

the converting the shot video data into the stereoscopic video includes:

decomposing video data according to frames, performing geometric correction on each frame of image by using an image transformation matrix, and cutting off blank areas;

dividing each frame of image into a left view and a right view with the same size, and performing left-right mirror transformation on the right view;

performing color compensation on each pixel in the right view according to the gray compensation mapping;

adaptively processing each frame of left view and right view according to a stereo video observation mode, merging each frame of left view and right view into a stereo video frame, and merging all stereo video frames into a stereo video;

supplementary shooting device of stereoscopic video is including being used for acquireing stainless steel level crossing (1) of scene virtual image, and the plane of stainless steel level crossing (1) place is crossing in stainless steel level crossing (1) inward flange with smart mobile phone (0) back: through connection, fixation and adjustment, the scene real images and the virtual images with basically the same range size can be observed from the smart phone at the same time;

the calibrating of the installation parameters of the auxiliary shooting device for the stereoscopic video comprises the following steps:

shooting an image by using a smart phone, wherein a shooting scene corresponding to the image contains abundant texture and color information and cannot be a scene with a single color;

extracting a plurality of sparse feature points from the image and performing feature point matching to generate a feature point matching set;

repeatedly and randomly selecting two pairs of feature points for matching for many times to generate a plane mirror normal vector estimation determined by matching the feature points;

and estimating the median of each component according to the multiple normal vectors to determine a plane mirror normal vector as an installation parameter.

2. The method according to claim 1, characterized in that the plane of the stainless steel flat mirror (1) is perpendicular to the back of the smartphone (0), the inner edge of the stainless steel flat mirror (1) is perpendicular to the side of the smartphone (0): meanwhile, the stainless steel plane mirror (1) is fixed by a plane mirror fixing frame (2), the plane mirror fixing frame (2) is connected to a support fixing frame (4) through a support connecting part (3), the support fixing frame (4) is connected with the smart phone (0) through a mobile phone connecting part (5), and the relative position and the relative angle between the plane mirror fixing frame (2) and the support connecting part (3) and between the support fixing frame (4) and the mobile phone connecting part (5) are fixed or adjusted through a fastener (6).

3. The method of claim 1, wherein extracting a plurality of sparse feature points from the image and performing feature point matching to generate a feature point matching set comprises:

extracting a plurality of sparse feature points and a description vector of each sparse feature point from the image according to an image local feature detection and description algorithm;

matching each sparse characteristic point with other characteristic points in the image according to a distance ratio test characteristic matching method to generate a matched characteristic point pair;

and establishing a characteristic point matching set comprising at least 100 characteristic point pairs according to the plurality of sparse characteristic point pairs.

4. The method of claim 1, wherein the step of repeating the step of randomly selecting two pairs of feature points for matching to generate a normal vector of the plane mirror determined by matching the feature points comprises:

determining the times of randomly selecting two pairs of feature points for matching according to the number of the feature point pairs;

and randomly selecting two pairs of feature points for matching each time, and generating a normal vector determined by matching the feature points according to the image coordinates of the two pairs of feature points and the focal length of the camera of the smart phone.

5. The method of claim 1, wherein the computing the projection geometry transformation matrix and the gray-scale compensation mapping function using the smartphone comprises:

generating an image transformation matrix according to the length and the width of the video frame and the normal vector of the plane mirror;

obtaining consistency indexes of all characteristic point pairs in the characteristic point matching set according to the focal length of the camera of the smart phone;

collecting all characteristic point pairs with consistency indexes larger than a preassigned broad value, and establishing an effective characteristic point matching pair set;

and obtaining the image gray of each feature point position in the effective feature point matching pair set, and fitting a gray compensation mapping function with the image gray of each feature point position according to a cubic polynomial curve fitting algorithm.

6. The method of claim 1, wherein the angle between the lateral side of the smartphone and the horizontal plane is not more than 30 degrees when the smartphone is used to capture video data.

7. The method of claim 1, wherein the stereoscopic video viewing mode comprises at least one of: VR glasses, red and blue glasses, bore hole: and adaptively processing each frame of left view and right view according to the stereoscopic video observation mode, wherein the left view and the right view of each frame are adaptively processed according to the video resolution and the aspect ratio specified by the stereoscopic video observation mode.

8. A stereoscopic video auxiliary shooting device applied to the method according to any one of claims 1 to 7, characterized by comprising a stainless steel plane mirror (1) for acquiring a virtual image of a scene, wherein the plane of the stainless steel plane mirror (1) is perpendicular to the back of the smartphone (0), the plane of the stainless steel plane mirror (1) and the back of the smartphone (0) intersect at the inner edge of the stainless steel plane mirror (1), and the inner edge of the stainless steel plane mirror (1) is perpendicular to the side of the smartphone (0): meanwhile, the stainless steel plane mirror (1) is fixed by a plane mirror fixing frame (2), the plane mirror fixing frame (2) is connected to a support fixing frame (4) through a support connecting part (3), the support fixing frame (4) is connected with the smart phone (0) through a mobile phone connecting part (5), and the relative position and the relative angle between the plane mirror fixing frame (2) and the support connecting part (3) and between the support fixing frame (4) and the mobile phone connecting part (5) are fixed or adjusted through a fastener (6).