CN114640801B - A vehicle-side panoramic view assisted driving system based on image fusion - Google Patents

Abstract

A car-side panoramic view assisted driving system based on image fusion, including: an image acquisition module for collecting 360° road information around the vehicle body, an embedded image processing device for real-time processing of images output by the image acquisition module, and a display Image display device for vehicle-side panoramic images. Among them, a physical connection is established between the embedded image processing device and the other two devices using cables; images are collected using three fisheye cameras installed at different locations on the vehicle with a viewing angle of 180°, and a video encoder and video capture are used The card integrates multiple analog video images into one. The embedded image processing device passes the integrated digital video image through the fisheye image processing module and the panoramic image splicer, splicing the digital video images from three different angles into a panoramic image, and displays the spliced panorama through the Web panoramic player. The image is displayed on the image display device. The invention can eliminate blind spots in the field of vision when large special vehicles are traveling.

Description

A vehicle-side panoramic view assisted driving system based on image fusion

Technical field

The invention relates to the field of safe driving of large special vehicles, and specifically to a vehicle-side panoramic view assisted driving system based on image fusion.

Background technique

In recent years, with the continuous advancement of my country's urbanization process, more and more large-scale special vehicles have appeared on urban roads, including but not limited to city buses, cement tankers, muck trucks, etc. The emergence of large-scale special vehicles has greatly facilitated people's daily production and life. However, these large special-purpose vehicles often have the characteristics of long and high body, which makes these vehicles have a wide range of blind spots. When driving on the road, especially when turning, serious traffic accidents will occur due to the large blind spots. accidents, which brings greater safety risks to other vehicles or pedestrians walking on the road. At present, major cities across the country have implemented the policy of "turning right and must stop" for large special vehicles. However, traffic accidents caused by the blind spots of large special vehicles have been occurring. Therefore, in order to improve the driving safety of large special-purpose vehicles and reduce the risk of other road vehicles or pedestrians as much as possible, it is necessary to reduce or even completely eliminate the blind spots in the field of vision that exist when large special-purpose vehicles are driving on the road.

In order to achieve the above goals, various systems for reducing blind spots in the vehicle's field of vision while driving have been researched and developed. Among them, Wang Lujie et al. proposed a car rearview mirror front-mounted probe vehicle-mounted system (Wang Lujie; Ma Feilong; Stone Carving; et al. Car rearview mirror front-mounted probe vehicle-mounted system [P]. Chinese patent: CN112776729A, 2021-05-11 .), by installing foreign object detection components on the car body to solve the problem of blind spots in the car's field of vision, but there are limitations such as complex design and high cost. Ye Shengjuan et al. proposed a car imaging device that eliminates blind spots in the car's field of vision (Ye Shengjuan; Wang Haifeng; Yang Yingfei; et al. A car imaging device that eliminates blind spots in the car's field of view [P]. Chinese Patent: CN213948279U, 2021-08-13.), The problem of blind spots in the car's field of view is solved by installing a fixed display on the vehicle and adjusting the fixed angle of the display. However, there is a drawback that the viewing angle is fixed, the display angle needs to be manually adjusted, and the surrounding environment information of the vehicle body cannot be fully displayed.

Contents of the invention

In order to overcome the above-mentioned problems of the prior art, the present invention provides a vehicle-side panoramic view assisted driving system based on image fusion, aiming to reduce or even completely eliminate the blind spots of large special vehicles, thereby improving the driving safety of large special vehicles. .

The present invention provides a vehicle-side panoramic view assisted driving system based on image fusion, including an image acquisition device, an embedded image processing device and an image display device.

The image acquisition equipment uses three fisheye cameras to collect the 360° road environment around the vehicle body, uses a video encoder to integrate the output images of the three fisheye cameras into one analog video, and uses the video capture card to convert the analog video Convert it into digital video and transmit it through cables to the fisheye image processing module in the embedded image processing device for image processing;

The embedded image processing device includes a fisheye image processing module; a Web panoramic player; a panoramic image splicer;

The fisheye image processing module is used to process the original fisheye image output by the video capture card. In order to capture a larger field of view, the original fisheye camera will cause severe distortion of the pixel information around the image. Therefore, it is necessary to use latitude and longitude expansion to correct the original fisheye image into a surround view to improve the final panoramic stitching effect. This method mainly transforms the pixel coordinates in the 2D Cartesian coordinate system into the spherical Cartesian coordinate system by making a series of changes to the pixel coordinates in the fisheye image, and finally converts the coordinates in the spherical Cartesian coordinate system into After that, the pixel points are mapped based on the longitude and latitude coordinates to achieve the purpose of converting the fisheye image into a surround view. The specific steps are as follows:

1) After obtaining the original fisheye image, write a circular mask function based on the center and radius of the three-view fisheye imaging to intercept the target image area, where the coordinate range of the pixels in the intercepted image area is as follows Formula (1) shows:

x∈[0,cols-1],y∈[0,rows-1] (1)

Among them, x and y are the abscissa and ordinate of the pixel coordinates of the intercepted image respectively, cols is the horizontal width of the original fisheye image, and rows is the vertical width of the original fisheye image;

2) In order to control the final video resolution after image fusion, it is necessary to control the size of the output image in step 1);

3) Convert the pixel coordinate point (x, y) of the intercepted image area from the 2D Cartesian coordinate system to the standard coordinate A (x _A , y _A ). The conversion relationship is as shown in formula (2):

Among them, x and y are the abscissa and ordinate of the pixel coordinates of the intercepted image respectively, cols is the horizontal width of the original fisheye image, and rows is the vertical width of the original fisheye image;

4) Convert the standard coordinates A (x _A , y _A ) into spherical three-dimensional Cartesian coordinates P (x _p , y _p , z _p ). The conversion formula is as shown in formulas (3) and (4):

P(p,φ,θ) (3)

Among them, P is the radial distance of the line OP between the coordinates of a point on the sphere and the origin O, θ is the angle between OP and the z-axis, φ is the angle between the projection of OP on the xOy plane and the x-axis, r is the radius of the ball, F is the focal length of the fisheye camera, and the spherical coordinate system is converted into a Cartesian coordinate system according to formula (5):

x _p = psinθcosφ, y _p = psinθsinφ, z _p = pcosθ (5)

5) Convert the spatial coordinate system P into longitude and latitude coordinates. The conversion relationship is as shown in formula (6):

Among them, x _p , y _p , z _p are the coordinates of point P, latitude is the longitude coordinate, and longitude is the latitude coordinate;

6) Convert and map the longitude and latitude coordinates in step 5) to the pixel coordinates (x _o , y _o ) of the expanded image. The mapping relationship is as shown in formula (7):

Among them, x ₀ represents the abscissa of the pixel in the expanded image, and y ₀ represents the ordinate of the pixel in the expanded image;

7) After completing the pixel mapping, black gaps that are not mapped to pixels will appear in the picture. These black areas are filled with the cubic interpolation algorithm to achieve a complete output image effect.

Panoramic image splicer is used to splice three fisheye images processed by the fisheye image processing module into a panoramic image. In order to ensure the coherence of the field of view after splicing, the three fisheye cameras in different directions need to be numbered in a fixed order, and the order will always remain unchanged in subsequent operations. In the image processing process, the SIFT algorithm is used to calculate the feature points of each image, which are used as local invariant description operators of the image that remain unchanged in scale space, scaling, rotation and affine transformation. It is also necessary to find matching feature points between adjacent images, and use the RANSAC method to further filter out the feature matching points, so as to calculate the homography matrix by finding the mapping relationship between the feature matching points. Finally, the perspective changes are performed on the image according to the calculated homography matrix, and finally the perspective transformed images are spliced to realize the function of splicing panoramic images on the vehicle side. The specific steps are as follows:

1) After obtaining the fisheye image processed by the fisheye image processing module, first fixedly number the images from different angles in sequence, and maintain the consistency of subsequent image numbering;

2) Use the SIFT algorithm that comes with OpenCV to calculate the feature points of each image, and use them as local invariant description operators of the image that remain unchanged in scale space, scaling, rotation and affine transformation;

3) Image splicing also requires finding matching feature points between adjacent images. Therefore, in the present invention, the method of calculating the Euclidean distance measure is used to roughly match the fisheye images from three viewing angles, and then the nearest neighbor Euclidean distance and the second neighbor Euclidean distance are compared. The SIFT matching method of distance is used to filter the feature points of the two images. When the ratio of the nearest neighbor Euclidean distance to the next neighbor Euclidean distance is less than 0.8, the matching point is selected;

4) Use the RANSAC method to further screen out the mismatching points from the rough matching points processed in step 3), thereby improving the accuracy of subsequent image processing, and then find out the mapping relationship between the feature points to calculate the homography matrix;

5) Use the homography matrix calculated in step 4) to perform perspective transformation on the fisheye image processed by the fisheye image processing module, then splice the perspective transformed images, and finally synthesize the video stream to realize the panoramic splicing function.

Web panoramic player is used to display the panoramic image output by the panoramic image splicer on the web page. In order to reduce the delay in video display, the panoramic player uses the rtc.js player plug-in to build a front-end player to play the video. In order to enable the front end to support the playback of panoramic videos, the technology of three.js+video tag+rtc.js is used to implement the panoramic player. The panoramic player mainly builds a spherical model through three.js, and uses the video tag as the surface rendering material of the sphere to map the sphere, so as to achieve the effect of projecting the panoramic video onto the sphere. The panoramic player mainly builds a spherical model through three.js, and uses the video tag as the surface rendering material of the sphere to map the sphere, so as to achieve the effect of projecting the panoramic video onto the sphere. It is installed and browsed on the embedded image processing device. You can browse the panoramic image on the image display device by using the device;

The image display device establishes a physical connection with the embedded image processing device and is used to display the panoramic image presented by the Web player.

Compared with the existing technology, the beneficial effects of the present invention are: using three fisheye cameras, 360° environmental information around the vehicle body can be obtained, and the three-channel analog video data can be integrated into one digital channel through a video encoder and a video capture card. Video data, the digital video data is input to the embedded image processing device for further processing, which greatly saves the port resources of the embedded device, and the overall design cost is relatively low. At the same time, embedded devices integrated with AI chips are used to improve the real-time processing capabilities and image output capabilities of videos, and a self-designed panoramic player is used to play panoramic videos, significantly reducing or even completely eliminating the presence of large-scale special vehicles when driving. It solves the problem of blind spots in a relatively wide range of vision and has a good assisted driving effect.

Description of the drawings

Figure 1 is an overall framework diagram of the system of the present invention;

Figure 2 is a schematic diagram of the installation of the camera of the present invention;

Figure 3 is a flow chart of fisheye image processing of the present invention;

Figure 4 is a processing flow chart of image splicing and fusion according to the present invention.

Detailed ways

The examples of the present invention will be further described in detail below in conjunction with the accompanying drawings:

As shown in Figure 1, a car-side panoramic view assisted driving system based on image fusion consists of three parts: image acquisition equipment, embedded image processing equipment and image display equipment. Among them, the embedded image processing equipment, image acquisition equipment and image display equipment are physically connected with cables. The image acquisition equipment mainly realizes the output analog video images of multi-channel fisheye cameras through video encoders, video capture cards and other hardware. The device uses hardware encoding to integrate multiple channels of analog video into one channel of digital video. While reducing the amount of data transmission, the digital video data is input into the embedded image processing device for image processing. After the embedded image processing device obtains the input digital image, it first processes the severely distorted original fisheye image into a ring view using a coordinate change method to obtain relatively rich image information, and then processes the three channels into a ring view. Fisheye images are stitched together to form a video stream. Finally, the spliced panoramic image is presented on the image display device on the Web through the Web panoramic player, thus achieving the function of significantly reducing or even completely eliminating the wide range of blind spots when large special vehicles are driving, and improving the safety of large special vehicles. .

As shown in Figure 2, the angle of view of the fisheye camera used in the present invention is 180°. In order to achieve the collection of 360° environmental information around the vehicle body and a better image stitching effect, three fisheye cameras need to be installed on the same at different heights, and each fisheye camera needs to be spaced 120° apart. According to the installation diagram in Figure 2, and by adjusting the installation positions of the three cameras on the large special vehicle according to the final image display effect, 360° panoramic environmental image information around the vehicle body can be obtained, thereby achieving good assisted driving effects.

As shown in Figure 3, the fisheye image processing module achieves the optimal effect of the output image by performing a series of transformations on the pixel coordinates in the fisheye image, mapping of pixels, and image filling of gap points. The main implementation steps are as follows:

1) After obtaining the original fisheye image, write a circular mask function based on the center and radius of the three-view fisheye imaging to intercept the target image area, where the coordinate range of the pixels in the intercepted image area is as follows Formula (1) shows:

x∈[0,cols-1],y∈[0,rows-1] (1)

Among them, x and y are the abscissa and ordinate of the pixel coordinates of the intercepted image respectively, cols is the horizontal width of the original fisheye image, and rows is the vertical width of the original fisheye image;

2) In order to control the final video resolution after image fusion, it is necessary to control the size of the output image in step 1);

3) Convert the pixel coordinate point (x, y) of the intercepted image area from the 2D Cartesian coordinate system to the standard coordinate A (x _A , y _A ). The conversion relationship is as shown in formula (2):

Among them, x and y are the abscissa and ordinate of the pixel coordinates of the intercepted image respectively, cols is the horizontal width of the original fisheye image, and rows is the vertical width of the original fisheye image;

4) Convert the standard coordinates A (x _A , y _A ) into spherical three-dimensional Cartesian coordinates P (x _p , y _p , z _p ). The conversion formula is as shown in formulas (3) and (4):

P(p,φ,θ) (3)

Among them, P is the radial distance of the line OP between the coordinates of a point on the sphere and the origin O, θ is the angle between OP and the z-axis, φ is the angle between the projection of OP on the xOy plane and the x-axis, r is the radius of the ball, F is the focal length of the fisheye camera, and the spherical coordinate system is converted into a Cartesian coordinate system according to formula (5):

x _p = psinθcosφ, y _p = psinθsinφ, z _p = pcosθ (5)

5) Convert the spatial coordinate system P into longitude and latitude coordinates. The conversion relationship is as shown in formula (6):

Among them, x _p , y _p , z _p are the coordinates of point P, latitude is the longitude coordinate, and longitude is the latitude coordinate;

6) Convert and map the longitude and latitude coordinates in step 5) to the pixel coordinates (x _o , y _o ) of the expanded image. The mapping relationship is as shown in formula (7):

Among them, x ₀ represents the abscissa of the pixel in the expanded image, and y ₀ represents the ordinate of the pixel in the expanded image;

7) After completing the pixel mapping, black gaps that are not mapped to pixels will appear in the picture. These black areas are filled with the cubic interpolation algorithm to achieve a complete output image effect.

As shown in Figure 4, the technical method of the panoramic image splicer includes extraction and calculation of image feature points, matching of feature points between adjacent images, finding the mapping relationship between feature points, calculation of the homography matrix and perspective of the image. Transformation and image splicing, etc., the main implementation steps are as follows:

1) After obtaining the fisheye image processed by the fisheye image processing module, first fixedly number the images from different angles in sequence, and maintain the consistency of subsequent image numbering;

2) Use the SIFT algorithm that comes with OpenCV to calculate the feature points of each image, and use them as local invariant description operators of the image that remain unchanged in scale space, scaling, rotation and affine transformation;

3) Image splicing also requires finding matching feature points between adjacent images. Therefore, in the present invention, the method of calculating the Euclidean distance measure is used to roughly match the fisheye images from three viewing angles, and then the nearest neighbor Euclidean distance and the second neighbor Euclidean distance are compared. The SIFT matching method of distance is used to filter the feature points of the two images. When the ratio of the nearest neighbor Euclidean distance to the next neighbor Euclidean distance is less than 0.8, the matching point is selected;

4) Use the RANSAC method to further screen out the mismatching points from the rough matching points processed in step 3), thereby improving the accuracy of subsequent image processing, and then find out the mapping relationship between the feature points to calculate the homography matrix;

5) Use the homography matrix calculated in step 4) to perform perspective transformation on the fisheye image processed by the fisheye image processing module, then splice the perspective transformed images, and finally synthesize the video stream to realize the panoramic splicing function.

The content described in the embodiments of this specification is only an enumeration of the implementation forms of the inventive concept. The protection scope of the present invention should not be considered to be limited to the specific forms stated in the embodiments. The protection scope of the present invention also extends to those skilled in the art. Equivalent technical means that a person can think of based on the concept of the present invention.

Claims (3)

1. An image fusion-based vehicle-end panoramic view angle assisted driving system is characterized in that: comprising the following steps: the system comprises an image acquisition module, an embedded image processing device and an image display device, wherein the image acquisition module is used for acquiring 360-degree road environment information around a vehicle body, the embedded image processing device is used for processing an image output by the image acquisition module in real time, and the image display device is used for displaying a panoramic image of a vehicle end;

the image acquisition module acquires a road environment of 360 degrees around a vehicle body by adopting three fisheye cameras, integrates output images of three fisheye cameras into one path of analog video by using a video encoder, converts the one path of analog video into digital video by using a video acquisition card, and transmits the digital video to a fisheye image processing module in the embedded image processing equipment for image processing by using a cable;

the embedded image processing device comprises a fisheye image processing module, a Web panoramic player and a panoramic image splicer;

the fish-eye image processing module is used for processing the original fish-eye image output by the video acquisition card; the original fisheye image is corrected into a circular view by adopting a longitude and latitude unfolding mode to improve the effect of final panoramic stitching: through a series of changes to the pixel coordinates in the fisheye image, the pixel coordinates in the 2D Cartesian coordinate system are transformed into a spherical Cartesian coordinate system, the coordinates in the spherical Cartesian coordinate system are finally converted into longitude and latitude coordinates, and then, the pixel points are mapped based on the longitude and latitude coordinates, so that the fisheye image is converted into the annular view, and the specific operation steps are as follows:

1) After an original fisheye image is obtained, a circular mask function is written according to the circle center and the radius of the three-view fisheye imaging to intercept an image region of a target, wherein the coordinate range of pixel points of the intercepted image region is shown as a formula (1):

x is E [0, cols-1], y is E [0, rows-1] (1), wherein x and y are respectively the abscissa and ordinate of the pixel coordinates of the intercepted image, and cols is

The lateral width of the original fisheye image, rows is the longitudinal width of the original fisheye image;

2) In order to control the resolution of the video after final image fusion, it is necessary to control the output in step 1)

The size of the picture;

3) Converting pixel coordinate points (x, y) of the truncated image area from a 2D Cartesian coordinate system to a standard

Coordinates A (x) _A ,y _A ) The conversion relationship is shown in formula (2):

wherein x and y are respectively the abscissa and the ordinate of the pixel point coordinates of the intercepted image, and cols is

The lateral width of the original fisheye image, rows is the longitudinal width of the original fisheye image;

4) Standard coordinates a (x _A ,y _A ) Three-dimensional Cartesian coordinates P (x) _p ,y _p ,z _p ) Conversion of

The formulas are shown as formulas (3) and (4):

P(p,φ,θ)(3)

wherein P is the radial distance of a connecting line OP between the P coordinate of a point on the spherical surface and the origin O, θ is the included angle between the OP and the z axis, φ is the included angle between the projection of the OP on the xOy plane and the x axis, r is the radius of the sphere, F is the focal length of the fisheye camera, and the spherical coordinate system is converted into Cartesian coordinate according to formula (5)

The system is as follows:

x _p ＝psinθcosφ,y _p ＝psinθsinφ,z _p ＝pcosθ (5)

5) Converting the space coordinate system P into longitude and latitude coordinates, wherein the conversion relation is shown in a formula (6):

wherein x is _p ,y _p ,z _p Is the coordinates of the P point, latitudes are longitude coordinates, longitudes are latitudes

Coordinates;

6) Mapping to pixel coordinates (x) of the expanded graph according to the longitude and latitude coordinate conversion in step 5) _o ,y _o ) Mapping of

The radial relationship is shown in formula (7):

wherein x is ₀ Represented by the pixel abscissa, y in the expanded view ₀ Representing pixels in an expanded view

An ordinate;

7) After the pixel mapping is completed, black void points which are not mapped by the pixel points appear in the picture,

filling the black areas by using a cubic interpolation algorithm to achieve the effect of outputting complete images;

the panoramic image splicer carries out panoramic image's concatenation with three fisheye images after fisheye image processing module handles, includes: in order to ensure the continuity of the spliced visual field, the three fisheye cameras in different directions need to be numbered according to a fixed sequence, and the sequence is kept unchanged all the time in the subsequent operation; in the image processing process, calculating the characteristic point of each image by using a SIFT algorithm, and taking the characteristic point as an image local invariant description operator with the scale space, scaling, rotation and affine transformation kept unchanged; the matching feature points between the adjacent images are required to be found, and the feature matching points are further screened out by using a RANSAC method, so that a homography matrix is calculated by finding the mapping relation between the feature matching points; finally, performing perspective change on the images according to the homography matrix obtained by calculation, and finally splicing the images after perspective change to realize the function of splicing panoramic images at the vehicle end; the specific operation steps are as follows:

(1) When the fisheye images processed by the fisheye image processing module are obtained, firstly, the images with different angles are sequentially and fixedly numbered, and the consistency of the serial numbers of the subsequent images is maintained;

(2) Calculating the characteristic points of each image by using an SIFT algorithm carried by OpenCV, and taking the characteristic points as an image local invariant description operator with the scale space, scaling, rotation and affine transformation kept unchanged;

(3) The image stitching also needs to search matching characteristic points between adjacent images, the fisheye images of three visual angles are subjected to rough matching by adopting a method for calculating Euclidean distance measure, then screening is carried out in the characteristic points of the two images by adopting a SIFT matching mode for comparing the nearest neighbor Euclidean distance with the secondary neighbor Euclidean distance, and when the ratio of the nearest neighbor Euclidean distance to the secondary neighbor Euclidean distance is smaller than 0.8, the matching points are selected;

(4) Further screening out mismatching points from the rough matching points processed in the step (3) by using a RANSAC method, thereby improving the precision of subsequent image processing, and then finding out the mapping relation between the characteristic points, so as to calculate a homography matrix;

(5) Performing perspective transformation on the fisheye image processed by the fisheye image processing module by using the homography matrix calculated in the step (4), then splicing the images after perspective transformation, and finally synthesizing a video stream to realize the panoramic splicing function;

the Web panorama player displays the panoramic image output by the panoramic image splicer on the webpage; in order to reduce the delay of video display, the Web panorama player adopts an rtc.js player plug-in to construct a front-end player to play the video; in order to enable the front end to support the playing of panoramic video, a technology of three.js+video tag+rtc.js is adopted to realize panoramic playing; the Web panoramic player establishes a spherical model through three. Js, and uses the video tag as a rendering material on the surface of the sphere to map the sphere, so that the effect of projecting panoramic video onto the sphere is achieved; installing a browser on the embedded image processing device, and browsing the panoramic image on the image display device;

the image display device is in physical connection with the embedded image processing device, and displays the panoramic image presented by the Web panoramic player.

2. The vehicle-end panoramic viewing angle assisted driving system based on image fusion according to claim 1, wherein: the embedded image processing device selects an Atlas 200 acceleration module as an AI computing chip.

3. The vehicle-end panoramic viewing angle assisted driving system based on image fusion according to claim 1, wherein: the angle of view of the fisheye cameras is 180 °, three fisheye cameras are installed at different positions at the same height, and a space of 120 ° is required between each fisheye camera.