JP2010109451A - Vehicle surrounding monitoring device, and vehicle surrounding monitoring method - Google Patents

Abstract

A vehicle surroundings monitoring apparatus and a vehicle surroundings monitoring method are provided that allow a driver to more accurately recognize the positional relationship of three-dimensional objects around the vehicle.
A vehicle surroundings monitoring apparatus 100 combines a three-dimensional object detection unit 430 that detects a three-dimensional object existing around the vehicle and a bird's-eye view image around the vehicle based on images taken by a plurality of cameras 200 to generate a combined bird's-eye view image. A composite overhead image generation unit 440 that generates and corrects the solid object reflected in the generated overhead image based on the detection result of the three-dimensional object detection unit 430 to the one rotated in the projection direction viewed from a predetermined viewpoint. .
[Selection] Figure 1

Description

  The present invention relates to a vehicle periphery monitoring device and a vehicle periphery monitoring method for synthesizing a bird's-eye view image that enables a bird's-eye view of a vehicle periphery from captured images of a plurality of cameras that capture the vehicle periphery.

  2. Description of the Related Art In recent years, vehicle surroundings monitoring devices that display a bird's-eye view image looking down around a vehicle from a virtual viewpoint above the vehicle have been spreading. Many vehicle surrounding monitoring devices generate a bird's-eye view image using a plurality of cameras so that the vehicle surroundings can be displayed in a wide range (see, for example, Patent Documents 1 and 2).

  FIG. 33 is a diagram illustrating an example of a bird's-eye view image obtained from a camera arrangement and a captured image of each camera in the vehicle surrounding monitoring device (hereinafter referred to as “conventional device”) described in Patent Literatures 1 and 2.

  As shown in FIG. 33, the vehicle 10 is provided with first to fourth cameras 20-1 to 20-4. The first to fourth cameras 20-1 to 20-4 respectively photograph the road surface of the vehicle 10 on the front side, the rear side, the left side, and the right side in an obliquely looking down direction. From the captured images of the first to fourth cameras 20-1 to 20-4, first to fourth camera overhead images 30-1 to 30-approximated to images when looking down from a virtual viewpoint above the vehicle. 4 can be obtained.

  It is desirable that the overhead image to be displayed is an image that is continuous over as wide a range as possible. Therefore, the conventional apparatus synthesizes a single bird's-eye view image from images captured by a plurality of cameras. Hereinafter, a bird's-eye view image synthesized from images captured by a plurality of cameras is referred to as a “synthesized bird's-eye view image”.

  The conventional apparatus combines the captured images of the first to fourth cameras 20-1 to 20-4 with the road surface as a reference, and generates a combined overhead image that is continuous over the entire circumference of the vehicle 10. Further, in the conventional apparatus, the shooting ranges of the adjacent cameras 20 are overlapped so as not to cause omission in the synthesized overhead view image. Hereinafter, an area where the imaging ranges overlap is referred to as an “overlap area”.

  The positions of objects (non-three-dimensional objects) having substantially no height, such as road markings and shades of overlapping areas, coincide in the camera overhead image 30 of both cameras 20. However, the position of the three-dimensional object located in the overlapping area and having a height is different for each camera overhead image. This is because the projection direction of the three-dimensional object when viewed from the camera 20 is different for each camera 20.

  FIG. 34 shows the third state in the situation where the first and second three-dimensional objects 40-1 and 40-2 are located in the overlapping area of the second camera 20-2 and the third camera 20-3. It is a figure which illustrates camera overhead view image 30-3. FIG. 35 is a diagram illustrating an example of the second camera overhead image 30-2 in the same situation as described above.

  In FIG. 34, the first and second three-dimensional objects 40-1 and 40-2 are divided into base portions 41-1 and 41-2 and solid portions 42-1 and 42-2. The base portion is a portion having substantially no height with respect to the road surface in each three-dimensional object, and the three-dimensional portion is a portion having a height component with respect to the road surface in each three-dimensional object. In the third camera bird's-eye view image 30-3, the three-dimensional parts 42-1 and 42-2 are visually recognized by the driver in a state of extending backward from the bases 41-1 and 41-2 with respect to the vehicle 10. That is, in the third camera bird's-eye view image 30-3, the first and second three-dimensional objects 40-1 and 40-2 are in a state of falling toward the rear direction of the vehicle 10. On the other hand, as shown in FIG. 35, in the second camera overhead image 30-2, the first and second three-dimensional objects 40-1 and 40-2 are in a state of falling down toward the left direction of the vehicle 10. ing.

  As described above, the direction in which the three-dimensional object in the overlapping area falls (projection direction) is different for each camera overhead image. When a composite overhead image is created by simply superimposing two camera overhead images in the overlapping area, two stereoscopic objects appear for one stereoscopic object, resulting in a double image. Therefore, how to process the three-dimensional object in the overlapping area becomes a problem.

Therefore, as shown in FIGS. 36 and 37, the conventional apparatus adopts only the three-dimensional object 40 of one camera overhead image with the boundary 60 as a boundary to generate a composite overhead image 50. More specifically, the conventional apparatus sets a boundary 60 inside the overlapping area as shown in FIGS. Thereby, it can prevent that a solid thing is displayed repeatedly.
JP 2008-48317 A JP 2007-180720 A

  However, in the conventional device, for example, as shown in FIGS. 38 and 39, two parallel objects that are actually parallel may be displayed in different directions on both sides of the boundary 60. In addition, as shown in FIG. 39, two solid objects that do not actually overlap may be displayed in an image with the boundary 60 overlapping. That is, there are cases in which a three-dimensional object is displayed with an arrangement relationship that is significantly different from an actual arrangement relationship between a plurality of three-dimensional objects (hereinafter simply referred to as “three-dimensional object arrangement relationship”). Therefore, the conventional device has a problem of causing the driver to erroneously recognize the arrangement relationship of the three-dimensional object around the vehicle.

  In the above description, the example of two solid objects has been described for easy understanding of the problem. However, the present invention is not limited to this, and the same problem occurs even in the case of one solid object.

  The present invention has been made in view of such a point, and an object of the present invention is to provide a vehicle periphery monitoring device and a vehicle periphery monitoring method that allow a driver to easily recognize the arrangement relationship of three-dimensional objects around the vehicle. To do.

  The vehicle periphery monitoring device of the present invention generates a combined overhead image by combining a three-dimensional object detection unit that detects a three-dimensional object existing around the vehicle and a bird's-eye view image around the vehicle based on images captured by a plurality of cameras. A first image generation unit configured to correct a three-dimensional object shown in the generated combined overhead image based on a detection result by the object detection unit to a rotation in a projecting direction viewed from a predetermined viewpoint;

  According to a vehicle surrounding monitoring method of the present invention, a first step of detecting a three-dimensional object existing around a vehicle and a bird's-eye view image around the vehicle based on captured images of a plurality of cameras are generated to generate a combined bird's-eye view image. And a second step of correcting the three-dimensional object shown in the generated synthesized overhead image to the one rotated in the projecting direction viewed from a predetermined viewpoint based on the detection result in the first step.

  According to the present invention, since the projection direction of the three-dimensional object is aligned, the driver can more accurately recognize the arrangement relationship of the three-dimensional object around the vehicle.

  Hereinafter, an embodiment of the present invention will be described in detail with reference to the drawings.

  FIG. 1 is a block diagram of a vehicle surroundings monitoring apparatus according to an embodiment of the present invention.

  In FIG. 1, the vehicle surrounding monitoring apparatus 100 is roughly provided with first to fourth cameras 200-1 to 200-4, an image processing unit 400, and a monitor 500. Since the first to fourth cameras 200-1 to 200-4 have the same configuration, they are collectively described as the camera 200 as appropriate.

  The camera 200 captures the surroundings of the vehicle. The camera 200 includes an imaging unit 210 and a frame switching unit 230 having a pair of frame memories 220.

  The imaging unit 210 includes a CCD (Charge Coupled Device) that is a solid-state imaging device, and an optical system that forms an optical image on the imaging surface of the CCD (none of which is shown), and an image captured by the CCD is framed. Output to the memory 220. The frame memory 220 temporarily stores the image captured by the imaging unit 210. The frame switching unit 230 alternately switches between the input destination of the image captured by the imaging unit 210 and the reference destination by the image processing unit 400 between the two frame memories 220.

  The camera 200 may employ a complementary metal oxide (CMOS) device as an image sensor. In this case, the frame memory 220 can be omitted by providing the CMOS device with a frame memory function.

  FIG. 2 is a perspective view showing an example of attachment positions of the first to fourth cameras 200-1 to 200-4 of the vehicle surrounding monitoring apparatus of FIG.

  As shown in FIG. 2, the first camera 200-1 is attached to the front center of the vehicle 600 in order to photograph the front of the vehicle 600. The second camera 200-2 is attached to the center of the rear part of the vehicle 600 in order to photograph the rear of the vehicle 600. The third camera 200-3 is attached to the left side mirror of the vehicle 600 in order to photograph the left side of the vehicle 600. The fourth camera 200-4 is attached to the right side mirror of the vehicle 600 in order to image the right side of the vehicle 600.

  The first to fourth camera imaging ranges 200a-1 to 200a-4, which are imaging ranges of the first to fourth cameras 200-1 to 200-4, all include the ground around the vehicle 600. Further, adjacent camera imaging ranges 200a (for example, the third camera imaging range 200a-3 and the second camera imaging range 200a-2) overlap each other. That is, there is an overlapping area between adjacent camera imaging ranges 200a. Thereby, the 1st-4th cameras 200-1-200-4 can image 360 degrees around the vehicle 600 as a whole.

  FIG. 3 is a diagram illustrating an example of captured images of the first to fourth cameras 200-1 to 200-4.

  As shown in FIG. 3, first to fourth camera captured images 710-1 to 710-4 captured by the first to fourth cameras 200-1 to 200-4 are respectively in front of the vehicle 600. The situation on the rear, left side, and right side is shown. The first to fourth camera captured images 710-1 to 710-4 are stored in the frame memories of the first to fourth cameras 200-1 to 200-4 in a state where they can be read from the image processing unit 400. .

  Refer to FIG. 1 again. The image processing unit 400 includes a mapping table reference unit 410, a camera overhead image generation unit 420, a three-dimensional object detection unit 430, a combined overhead image generation unit 440, and a timing generation unit 450.

  The mapping table reference unit 410 stores two types of mapping tables (MPT: Mapping Table), and generates two types of mapping data by referring to these mapping tables. One mapping table is a table in which pixel positions of captured images stored in the frame memory 220 of each camera 200 are associated with pixels of a camera overhead image described later. The other mapping table is a table in which pixels of each camera overhead image are associated with pixels of a composite overhead image described later. One mapping data is data instructing which pixel value of which captured image is assigned for each pixel of each camera overhead image. The other mapping data is data instructing which pixel value in which camera overhead image is assigned to each pixel of the synthesized overhead image. The mapping table reference unit 410 outputs one mapping data to the three-dimensional object detection unit 430 and the camera overhead image generation unit 420, and outputs the other mapping data to the composite overhead image generation unit 440.

  The mapping table stored in the mapping table reference unit 410 may be created manually, or may be created by the vehicle surrounding monitoring apparatus 100 executing calculations such as geometric transformation. When the mapping table is provided as firmware, the mapping table reference unit 410 may acquire the mapping table from the outside using a data transfer unit such as a communication line or a disk drive.

  The camera overhead image generation unit 420 reads out each captured image stored in the frame memory 220 of each camera 200 according to one mapping data output from the mapping table reference unit 410, and performs mapping, thereby obtaining a camera overhead image. Generate. Specifically, the camera bird's-eye view image generation unit 420 assigns the designated captured image pixel as the pixel value of the camera bird's-eye view image based on the one mapping data output from the mapping table reference unit 410, and A camera overhead image is generated for each captured image.

  Here, the camera-captured image shows the camera viewpoint, that is, the situation of the camera imaging range when viewed from the camera installation position. On the other hand, the camera overhead view image is obtained by converting the viewpoint of the camera-captured image, and the camera overhead view image is a camera when viewed from the viewpoint of a virtual camera (virtual viewpoint) installed above the vehicle. The whole or part of the imaging range is shown.

  Here, FIG. 4 is an example of the first to fourth camera overhead images 724-1 to 724-4 generated from the first to fourth camera captured images 710-1 to 710-4 shown in FIG. 3. FIG. The first to fourth camera overhead images 724-1 to 724-4 each include an overlapping area with the adjacent camera overhead image 724.

  Although the details will be described later, the three-dimensional object detection unit 430 shown in FIG. 1 performs image recognition processing on the camera-captured images obtained from the respective cameras 200 and detects a three-dimensional object existing around the vehicle from the camera-captured images. To do. In addition, the three-dimensional object detection unit 430 uses one of the mapping data from the mapping table reference unit 410 to know which pixel in each camera captured image corresponds to which pixel in the camera overhead image. Using the data, a set of coordinate values of an area (hereinafter referred to as “three-dimensional object area”) in which the detected three-dimensional object is reflected in the camera overhead image is specified. The three-dimensional object detection unit 430 further acquires the position closest to the vehicle in the detected three-dimensional object area as the base position of the three-dimensional object. The collection of coordinate values of the three-dimensional object area and the base position of the three-dimensional object are sent to the synthesized overhead image generation unit 440 as three-dimensional object position information.

  The combined overhead image generation unit 440 receives each camera overhead image from the camera overhead image generation unit 420 and receives the other mapping data from the mapping table reference unit 410. The synthesized overhead image generation unit 440 performs mapping on each received camera overhead image according to the received mapping data, and synthesizes the synthesized overhead image. Specifically, the synthesized overhead image generation unit 440 allocates the designated camera overhead image pixel as the pixel value of the synthesized overhead image based on the mapping data, thereby generating the synthesized overhead image. .

  Here, the synthesized bird's-eye view image differs from the camera bird's-eye view image in that the entire periphery of the vehicle when viewed from the virtual camera viewpoint is contained in one frame.

  FIG. 5 is a diagram illustrating an example of a combined overhead image generated by the combined overhead image generation unit 440. In the example of FIG. 5, the position of the virtual viewpoint is set right above the vehicle, and an icon 721 indicating the vehicle is arranged at the center of the synthesized overhead image 720. When a three-dimensional object exists around the vehicle, the three-dimensional object 723 is drawn at a corresponding position in the composite overhead image 720.

  Here, since each camera bird's-eye view image is taken from the position of each camera 200, the direction of collapse of the three-dimensional object shown in each camera is different (see FIG. 4). Therefore, also in the composite bird's-eye view image generated on the basis of such a camera bird's-eye view image, the three-dimensional object 723 has a different falling direction (see FIG. 5). If such a composite bird's-eye view image is displayed as it is, the driver who sees this image feels uncomfortable. In particular, in the overlapping area, as described in the “Background art” column, the arrangement relationship of the three-dimensional object is difficult for the driver to understand.

  From the above viewpoint, first, the synthesized overhead image generation unit 440 receives the three-dimensional object corresponding to the three-dimensional object position information from the camera overhead image generation unit 420 based on the three-dimensional object position information from the three-dimensional object detection unit 430. The camera is cut out from the overhead image and rotated (image rotation processing). Thereafter, the synthesized overhead image generation unit 440 synthesizes the rotated three-dimensional object with the synthesized overhead image generated by itself (three-dimensional object synthesis process).

  The image rotation process is performed when the three-dimensional object position information is received, and the three-dimensional object is extracted from the three-dimensional object area specified by the three-dimensional object position information in the camera overhead view image. The extracted three-dimensional object is rotated around the base position specified by the three-dimensional object position information, from the projection direction viewed from the viewpoint of the camera 200 that captured the three-dimensional object, to the projection direction viewed from a predetermined viewpoint.

  In the three-dimensional object synthesis process, the rotated three-dimensional object is synthesized on the synthesized overhead image. Here, since the other mapping data indicating the correspondence between the camera overhead image and the synthesized overhead image is input to the synthesized overhead image generation unit 440, the synthesized overhead image generation unit 440 is configured to display the rotated stereoscopic image. Based on this mapping data, it is specified at which position on the combined overhead image the object is to be combined.

  The above image rotation process and three-dimensional object synthesis process are performed for each received three-dimensional object position information. In particular, for the overlapping area, the same three-dimensional object is shown in a plurality of camera overhead images, and from which camera overhead image the three-dimensional object is extracted will be described later.

  Then, the synthesized overhead image generation unit 440 generates a video signal for displaying a moving image series from the synthesized overhead image for each frame, and outputs the generated video signal to the monitor 500.

  The timing generation unit 450 includes frame switching timings of the first to fourth cameras 200-1 to 200-4, mapping data output timings of the mapping table reference unit 410, and operation timings of the combined overhead image generation unit 440. A timing signal for controlling is generated. Based on this timing signal, a video signal of a composite overhead image that projects an image around the vehicle 600 in real time is output to the monitor 500.

  The monitor 500 displays the synthesized overhead image according to the video signal received from the synthesized overhead image generation unit 440. The monitor 500 is a display device having a liquid crystal display, for example, and is attached at a position where it can be seen from the driver's seat of the vehicle 600. The monitor 500 may be shared with other systems or devices such as a vehicle-mounted GPS (Global Positioning System) terminal (so-called car navigation system).

  Although not shown, the vehicle surroundings monitoring apparatus 100 is not shown in the figure, but a CPU (Central Processing Unit), a ROM (Read Only Memory) that stores a control program, and an EEPROM (Electrically Erasable and Programmable Read Only Memory) that stores various data including a mapping table. ) And a working memory such as a RAM (Random Access Memory). The functions of the above-described units are realized, for example, when the CPU executes a control program.

  According to the vehicle surrounding monitoring apparatus 100 as described above, the composite overhead image can be displayed to the driver, and the projection direction of the three-dimensional object of the composite overhead image is aligned with a direction based on a predetermined viewpoint. Can do. Moreover, since a mapping table is used, a composite overhead image can be generated at high speed.

  Next, details of the above-described image rotation processing will be described.

  FIG. 6 is a diagram illustrating an example of a combined overhead image before image rotation processing is performed. FIG. 6 schematically shows a combined overhead image when a plurality of three-dimensional objects (for example, pylon) placed on the road surface around the vehicle are located.

  Here, the position of the viewpoint of the camera 200 is referred to as “camera viewpoint”. As shown in FIG. 6, in the synthesized overhead image 720 before the image rotation processing is performed, the three-dimensional object 723 is projected from the camera viewpoint 731 of the camera 200 that captured the image with the base position 732 of the three-dimensional object as a base point. It falls into a direction (hereinafter referred to as “camera projection direction”) 733.

  For example, the first three-dimensional object 723-1 located in the portion close to the second camera overhead image 724-2 of the third camera overhead image 724-3 is the camera viewpoint 731 of the third camera 200-3. 3 falls in the first projecting direction 733-1 (rearward direction) as seen from FIG. On the other hand, the second three-dimensional object 723-2 located in the portion close to the third camera overhead image 724-3 of the second camera overhead image 724-2 is the camera viewpoint 731 of the second camera 200-2. -2 as viewed from the second projection direction 733-2 (leftward).

  Since the first three-dimensional object 723-1 and the second three-dimensional object 723-2 are originally located at a short distance from each other, the compatible objects 723-1 and 723-2 are in a state where the driver falls down in the same direction. Is preferably visually recognized. However, since the first three-dimensional object 723-1 and the second three-dimensional object 723-2 are taken from different camera viewpoints 731-3 and 731-2, the composite overhead image 720 before the image rotation process is performed. Is visually recognized by the driver as if it is falling in a significantly different direction.

  As described above, when the image rotation process is not performed, the projection direction of the three-dimensional object 723 is large in the vicinity of the boundary line of the camera overhead image 724 even though the two three-dimensional objects are actually located at a short distance. Different. This means that the projection direction of the three-dimensional object 723 is greatly different on both sides of the boundary line of the camera overhead image 724 even though they are actually the same three-dimensional object. Therefore, the composite overhead image 720 that is not subjected to the image rotation process causes a recognition error with respect to the arrangement relationship of the three-dimensional object of the driver as described above. Further, in the composite overhead image 720 that is not subjected to image rotation processing, the three-dimensional object 723 is displayed with discontinuous movement when the boundary line is exceeded by the movement of the vehicle or the movement of the three-dimensional object 723.

  Therefore, the vehicle surrounding monitoring apparatus 100 according to the present embodiment rotates the three-dimensional object 723 extracted from the camera overhead image 724 in the projection direction viewed from one viewpoint, and then synthesizes the three-dimensional object 723 into each composite overhead image 720. The projection direction of the object 723 is aligned. Thus, the recognition error and the discontinuous movement of the three-dimensional object are reduced. Hereinafter, a single viewpoint serving as a reference for rotation is referred to as a “composite viewpoint”.

  FIG. 7 is a diagram illustrating an example of a synthesized overhead image after image rotation processing is performed, and corresponds to FIG.

  As shown in FIG. 7, in the synthesized overhead image 720 after the image rotation processing is performed, the three-dimensional object 723 is projected from the synthesized viewpoint 734 with the base position 732 as a base point (hereinafter referred to as “viewpoint projection direction”). It is in a state of falling to 735. Here, the composite viewpoint 734 is set to the position of the viewpoint of the driver of the vehicle. The composite overhead image generation unit 440 generates such a composite overhead image 720 by rotating the three-dimensional object 723 around the base position 732 from the camera projection direction 733 to the viewpoint projection direction 735.

  For example, the projection directions of the first and second three-dimensional objects 723-1 and 723-2 are substantially parallel and closer to the actual arrangement relationship. Therefore, by performing the image rotation process, it is possible to prevent a discontinuous movement of the three-dimensional object 723 on the boundary line and a recognition error with respect to the arrangement relation of the three-dimensional object.

  As described above, by performing the image rotation process, it is possible to provide the driver with a combined overhead image in which the three-dimensional object 723 is arranged in an arrangement relationship that is closer to the actual arrangement relationship of the three-dimensional object. Further, since the composite viewpoint 734 is matched with the driver's viewpoint, the three-dimensional object can be brought down in the same direction as when the three-dimensional object is actually viewed from the driver. Therefore, it is possible to make the driver more accurately recognize the arrangement relationship of the actual three-dimensional object.

Here, in the coordinate system excluding the height component from the vehicle, the coordinates of the installation position of the camera 200 (the coordinates of the camera viewpoint) are C (C _x , C _y ), and the coordinates of the base position 732 of the three-dimensional object are O (O _x , O _y ), and the coordinates of the composite viewpoint are set as P (P _x , P _y ). At this time, the rotation angle θ from the camera projection direction 733 to the viewpoint projection direction is determined by the camera projection direction direction vector V _C (O _x −C _x , O _y −C _y ) and the viewpoint projection direction direction vector V _P (O _x -P _x, using _O y -P _y), can be calculated using the following equation (1).
θ = arccos {(V _P · V _C ) / (| V _P | · | V _C |)} (1)

  The rotation angle θ is uniquely determined for each coordinate of the synthesized overhead image. Therefore, by calculating and storing the rotation angle θ for each coordinate in advance, the image rotation process can be speeded up.

  Hereinafter, the operation of the vehicle surrounding monitoring apparatus 100 according to the present embodiment will be described.

  FIG. 8 is a flowchart showing an example of the operation of the vehicle surrounding monitoring apparatus 100.

  First, in step S1100, the camera overhead image generation unit 420 and the three-dimensional object detection unit 430 receive the timing signal from the timing generation unit 450 and receive the first signal from the first to fourth cameras 200-1 to 200-4. The fourth camera-captured image is acquired (see FIG. 3).

  Next, in step S1200, the camera overhead image generation unit 420 generates the first to fourth cameras from the first to fourth camera captured images based on one mapping data input from the mapping table reference unit 410. An overhead image is generated (see FIG. 4). Then, the camera overhead image generation unit 420 outputs each generated camera overhead image to the combined overhead image generation unit 440.

  Next, in step S <b> 1300, the composite overhead image generation unit 440 receives each camera overhead image from the camera overhead image generation unit 420 and receives the other mapping data from the mapping table reference unit 410. The synthesized overhead image generation unit 440 performs mapping on each received camera overhead image according to the received mapping data, and synthesizes the synthesized overhead image.

  Next, in step S1400, the three-dimensional object detection unit 430 performs a three-dimensional object detection process by performing an image recognition process on the first to fourth camera captured images acquired in step S1100.

  As the image recognition processing, for example, pattern matching processing, edge extraction processing, or stereo processing can be employed.

  In the case of pattern matching processing, the three-dimensional object detection unit 430 stores in advance characteristics of the shape of the appearance of a three-dimensional object (for example, a pylon or a pole) to be detected, and an image portion highly correlated with the shape is stored as a three-dimensional object. Detect as an object. In the case of edge extraction processing, the three-dimensional object detection unit 430 extracts and extracts only line segments in a direction close to a specific direction (for example, a direction perpendicular to the road surface) characteristic of the line segments constituting the three-dimensional object to be detected. The image portion where the line segments are concentrated is detected as a three-dimensional object. For example, a three-dimensional object such as a person, a pole, or a box has many line segments in a direction close to the vertical direction, and is easy to detect. In the case of stereo processing, the three-dimensional object detection unit 430 extracts a portion that is not a road surface image, that is, a three-dimensional object, from the movement of the vehicle with respect to the road surface and the changes in the first to fourth camera captured images accompanying the movement. Details of the stereo processing will be described later.

  Note that the three-dimensional object detection unit 430 determines a portion having a large luminance or color difference between a plurality of camera overhead images as a three-dimensional portion for the overlapping area, and the image portion and the luminance and color are continuously matched. The image to be determined is determined as a three-dimensional object.

  In addition, the three-dimensional object detection unit 430 may determine that a rough region including the three-dimensional object is a three-dimensional object. This is effective when a three-dimensional object cannot be extracted. In this case, the three-dimensional object detection unit 430 may make the three-dimensional object have a simple outer shape such as a rectangle or an ellipse.

  In step S1500, the three-dimensional object detection unit 430 determines whether at least one three-dimensional object to be subjected to image processing in a subsequent step has been detected in step S1400. If the three-dimensional object detection unit 430 has detected a three-dimensional object to be processed (S1500: YES), the process proceeds to step S1600.

  FIG. 9 is a diagram illustrating an example of a composite overhead image when a three-dimensional object to be processed exists. Here, the three-dimensional object is a rod-shaped member extending in a direction perpendicular to the road surface. As shown in FIG. 9, the three-dimensional object 723 falls down in the camera projection direction 733 starting from the camera viewpoint 731 that captured the image.

  In step S1600 of FIG. 8, the three-dimensional object detection unit 430 refers to one mapping data and converts the position of the three-dimensional object in the first to fourth camera captured images into position information in the camera overhead image. The three-dimensional object detection unit 430 outputs the three-dimensional object position information obtained in this manner to the composite overhead image generation unit 440.

  FIG. 10 is a diagram illustrating the three-dimensional object position information. As described above, the three-dimensional object position information is information indicating a collection of coordinates of the three-dimensional object area and the base position. The base position 732 is a position where the three-dimensional object 723 contacts the road surface. The three-dimensional object area 736 is specified by a set of coordinates where the three-dimensional object 723 is drawn.

  The three-dimensional object detection unit 430 employs the coordinates of the position closest to the camera viewpoint 731 of the camera 200 in the three-dimensional object area 736 as the base position 732.

  On the other hand, if the three-dimensional object detection unit 430 cannot detect the three-dimensional object to be processed (step S1500: NO), the process proceeds to step S1700.

  In step S1700, the three-dimensional object detection unit 430 outputs information indicating that the three-dimensional object to be processed has not been detected to the composite overhead image generation unit 440, and proceeds to step S1800.

  In step S1800 of FIG. 8, the synthesized overhead image generation unit 440 determines whether or not the three-dimensional object position information has been received. If the three-dimensional object position information is received (S1800: YES), the process proceeds to step S1900.

  Next, in step S1900, the synthesized overhead image generation unit 440 sets a synthesized viewpoint for rotating the three-dimensional object. Here, it is assumed that the synthesized overhead image generation unit 440 sets the synthesized viewpoint at the image position of the driver's seat of the vehicle (see FIG. 7).

  In step S2000, the synthesized overhead image generation unit 440 calculates a set of coordinates of the three-dimensional object area after rotating the three-dimensional object by θ obtained by the above equation (1) around the base of the three-dimensional object. . As described with reference to FIGS. 6 and 7, this rotation is a rotation from the camera projection direction around the base position to the viewpoint projection direction.

  FIG. 11 is a diagram illustrating how the three-dimensional object rotates. As shown in FIG. 11, the three-dimensional object 723 is rotated around the base 732 from the camera projection direction 733 to the viewpoint projection direction 735.

  Note that the three-dimensional object in the overlapping area has the same tilting direction after rotation, regardless of which camera overhead image is used for generation. However, the size of the three-dimensional object, that is, the length in the camera projection direction (hereinafter referred to as “projection length”) differs depending on which camera overhead image is used.

  FIG. 12 is a diagram illustrating a three-dimensional object of a certain three-dimensional object when a third camera photographed image is used. FIG. 13 is a diagram illustrating the same three-dimensional object when the second camera overhead image is used. As shown in FIGS. 12 and 13, the projection lengths of the three-dimensional objects 723-3 and 723-2 are different. This is because the distance from the camera 200 to the three-dimensional object is different.

FIG. 14 is a diagram illustrating the relationship between the distance from the camera 200 and the projection length. As shown in FIG. 14, the ratio between the distance d from the camera 200 to the three-dimensional object 737 and the projection length L is the height h _L of the three-dimensional object 737 and the distance h from the upper end of the three-dimensional object 737 to the height of the camera 200. _It is proportional to the ratio with _U. Therefore, the projection length L is proportional to the distance d from the camera 200 to the three-dimensional object 737. This means that the longer the distance from the camera 200, the longer the projection length of the three-dimensional object 737, the greater the distortion of the three-dimensional object 737 with respect to the actual appearance of the three-dimensional object, and the more portions that disappear from the display range. Means that.

  Therefore, it is desirable that the combined overhead image generation unit 440 uses a camera image having a shorter length to the position of the camera 200 for the three-dimensional object in the overlapping area. The distance can be obtained from the base position of the three-dimensional object and the position of the camera 200, for example.

  Then, in step S2100 of FIG. 8, the synthesized overhead image generation unit 440 draws a three-dimensional object in the synthesized overhead image in the area after rotation calculated in step S2000.

  Then, only by drawing a three-dimensional object in the rotated area, the composite overhead image remains the same three-dimensional object but the one before rotation. In step S2200, the composite overhead image generation unit 440 The road surface is drawn in the area before the three-dimensional object is rotated. As a result, the three-dimensional object is corrected to be rotated from the camera projection direction to the viewpoint projection direction around the base position in the synthesized overhead image.

  Here, an example of detailed processing in step S2200 will be described. Behind the three-dimensional object as viewed from the camera 200 (camera projection direction side) is a blind spot for the camera 200. Therefore, in the camera-captured image of the camera 200, the road surface color data to be drawn in the area before the rotation is missing. On the other hand, the area before rotation (hereinafter referred to as “data missing area”) changes with the movement of the viewpoint of the camera 200 (movement of the vehicle) and the movement of the three-dimensional object.

  Therefore, the synthesized overhead image generation unit 440 stores a predetermined number of frames of synthesized overhead images generated in the past, and supplements the data missing area of the current frame using the past frames.

  FIG. 15 is a diagram illustrating how data missing areas are complemented.

After the composite bird's-eye view image 720 _n of the nth frame shown in FIG. 15 (A) is acquired, the vehicle moves backward for garage entry or the like, and the composite bird's-eye view image 720 of the n + a frame shown in FIG. 15 (B). _{Assume that n + a} is acquired.

In the synthesized bird's-eye view image 720 _n of the n-th frame, the three-dimensional object 723 rotates from a certain data missing area 738 _n to a rotated area (hereinafter referred to as “rotation destination area”) 739 _n . The road surface image of the rotation destination area 739 _n is included in the synthesized overhead image 720 _n of the base n-th frame.

Then, in the combined overhead image 720 _{n + a} of the ( _{n + a) th} frame, the three-dimensional object 723 further rotates from the data missing area 738 _{n + a} to another rotation destination area 739 _{n + a} . Here, for simplification of description, it is assumed that the data missing area 738 _{n + a in the n + a} frame coincides with the rotation destination area 739 _n in the nth frame. In this case, the road surface image of the data missing area 738 _{n + a in} the _{n + a} frame is included as the road surface image of the rotation destination region 739 _n in the synthesized overhead image 720 _n of the nth frame.

Therefore, the combined overhead image generation unit 440 first combines the frames between the frames based on the image processing such as the moving amount and moving direction of the vehicle and / or the optical flow obtained from the steering angle amount and the vehicle speed. Correlate image coordinates. Then, as indicated by an arrow 740, the synthesized overhead image generation unit 440 uses the data missing area 738 _{n + a} in the synthesized overhead image 720 _{n + a} of the _{n + a} frame as the rotation destination region 739 in the synthesized overhead image 720 _n of the nth frame. Complement with _n road surface images.

  If there is no movement of the vehicle or the three-dimensional object, the data missing area cannot be complemented. Therefore, the synthesized overhead image generation unit 440 detects the data missing area only when the movement of the vehicle is detected in a state where the three-dimensional object to be detected exists and when the entry of the three-dimensional object is detected in the camera shooting range. You may make it complement.

  Further, the synthesized overhead image generation unit 440 may complement the data missing area using a synthesized overhead image of a later frame. However, images of several frames are buffered, and a slight delay occurs in the synthesized overhead image displayed on the monitor 500.

  After execution of step S2200, the process proceeds to step S2300 shown in FIG. In step S2300, the combined overhead image generation unit 440 outputs a video signal indicating the corrected combined overhead image to the monitor 500 to display the combined overhead image. The composite overhead image to be displayed is an image obtained by rotating the three-dimensional object when there is a three-dimensional object to be processed.

  In step S2400, timing generation unit 450 determines whether an instruction to end the process of displaying the synthesized overhead image is given by a user operation or the like. When the timing generation unit 450 is not instructed to end the processing (S2400: NO), the timing generation unit 450 outputs a timing signal and returns to Step S1100. When the timing generation unit 450 is instructed to end the processing (S2400: YES), the timing signal Is stopped and a series of processing is terminated.

  If there is no three-dimensional object to be subjected to image processing (S1800: NO), the synthesized overhead image generation unit 440 proceeds directly to step S2300 and monitors the synthesized overhead image without performing image rotation processing. 500.

  By such an operation, the vehicle surroundings monitoring apparatus 100 can provide the driver with a composite bird's-eye view image in which the projection directions of the three-dimensional objects around the vehicle are aligned.

  In the above description of the embodiment, the distance from the camera image received by the three-dimensional object detection unit 430 to the three-dimensional object and the three-dimensional object is detected. However, the present invention is not limited to this. It does not matter if the distance to the object is detected.

  In addition, when the synthetic | combination bird's-eye view image which consists of a several camera bird's-eye view image is used (refer FIG. 4), the solid object located in an overlap area may be reflected on a some camera bird's-eye view image in a different form. Therefore, in the synthesized bird's-eye view image, these different shapes may be synthesized so that one solid object is reflected.

  In addition, it is possible to obtain a three-dimensional object that is easier to see by correcting the projection length of the three-dimensional object shown in each camera overhead image to the projection length when viewed from the combined viewpoint.

  In addition, it is mainly in the overlapping area that the arrangement relationship of the three-dimensional object shown in the composite bird's-eye view image is significantly different from the actual arrangement relation of the three-dimensional object. Therefore, it is possible to reduce the processing load of the vehicle periphery monitoring device 100 by narrowing the processing target to a three-dimensional object located in the vicinity of the overlapping area.

  Further, when image processing is limited to overlapping areas, it is possible to more easily complement the data missing area by using a composite overhead image composed of a plurality of camera overhead images.

  In addition, when a certain three-dimensional object is difficult to see behind the other three-dimensional object, it is possible to make the three-dimensional object that has been shaded easily visible by changing the composite viewpoint.

  Hereinafter, these aspects will be described in order.

  First, the aspect which synthesize | combines the solid thing reflected in a different form is demonstrated in detail.

  FIG. 16 is a diagram illustrating an example of a combined overhead image when only a three-dimensional object after rotation obtained from the third camera overhead image is employed, and corresponds to FIG. FIG. 17 is a diagram illustrating an example of a composite overhead image when only a three-dimensional object after rotation (same as the three-dimensional object in FIG. 16) obtained from the second camera overhead image is employed. To do. 18 is a diagram illustrating an example of a combined overhead image when a three-dimensional object obtained by combining the three-dimensional object illustrated in FIG. 16 and the three-dimensional object illustrated in FIG. 17 is employed.

  A solid object 723-3 shown in FIG. 16 and a solid object 723-2 shown in FIG. 17 are images viewed from one viewpoint, respectively. Accordingly, in a three-dimensional object in which it is difficult to visually recognize the entire image when observed only from a certain direction, when the direction coincides with the shooting direction of the camera, the three-dimensional object 723 is an image in which it is difficult to grasp the three-dimensional object. For example, this is the case when a plate-shaped signboard is photographed from the side.

  Therefore, for example, the synthesized overhead image generation unit 440 combines the three-dimensional object 723-3 and the three-dimensional object 723-2 to draw the three-dimensional object 723 as illustrated in FIG. As a result, the driver can more easily grasp the three-dimensional object 723. Note that the synthesized overhead image generation unit 440 may synthesize only the three-dimensional object after rotation, or may synthesize the camera overhead image after rotating the three-dimensional object.

  Next, an aspect in which the projection lengths of the same three-dimensional object (hereinafter, referred to as “two three-dimensional objects” for convenience) reflected in different forms in different camera overhead images will be described.

This will be described with reference to FIG. The projection of the three-dimensional object 737 from the height h _L of the three-dimensional object 737 specified by the three-dimensional object detection and the distance d to the three-dimensional object 737 and the height H = h _L + h _{U of} the camera 200 (or the combined viewpoint 734). The length L can be calculated using the following equation (2).
L = (h _L · d) / (H−h _L ) (2)

The synthesized overhead image generation unit 440 calculates, for example, the projection length of the two three-dimensional objects to be synthesized and the projection length when viewed from the synthesis viewpoint 734 using Expression (2). Then, the combined overhead image generating unit 440, each of the two three-dimensional object, the ratio L obtained by dividing the projection length L ₂ when the projection length L ₁ when viewed from the synthesis viewpoint 734 as viewed from the camera 200 in ₁ / _{L 2,} it is reduced to the projection direction. Thereby, the projection length of the three-dimensional object can be aligned with the projection length when viewed from the composite viewpoint 734.

  When the height of the composite viewpoint 734 is the same as the height of the camera viewpoint 731, the projection lengths of the three-dimensional objects can be more easily aligned.

FIG. 19 is a diagram illustrating an example of the relationship between the base position of the three-dimensional object, the image position of the camera 200, and the composite viewpoint. As shown in FIG. 19, in the camera overhead image 724, the distance between the camera viewpoint 731 of the base position 732 and the camera 200 of the three-dimensional object 723 is _{d 1,} the distance between the same base position 732 with the synthetic viewpoint 734 _{d 2} Suppose that

FIG. 20 is a diagram illustrating the relationship between the distance from the viewpoint to the three-dimensional object and the projection length. As shown in FIG. 20, the original projection length and L _1, a projection length when the camera 200 is horizontally moved in the synthesis viewpoint and L _2. In this case, the ratio of the projection length _{L 2} for the projection length _{L 1} is consistent with the ratio of _{d 2} with respect to the distance _{d 1.} Therefore, the combined overhead image generation unit 440, for example, saw the projection length of the three-dimensional object from the combined viewpoint by expanding and contracting the three-dimensional object in the projection direction at a ratio r represented by the following expression (3). Correct to the projected length of time.
r = L ₁ · d ₂ / d ₁ (3)

  FIG. 21 is a diagram illustrating an example of a three-dimensional object after projection length correction obtained from the third camera overhead view image, and corresponds to FIG. FIG. 22 is a diagram illustrating an example of a three-dimensional object after projection length correction obtained from the second camera overhead image for the same three-dimensional object as illustrated in FIG. 21, and corresponds to FIG. 13. FIG. 23 is a diagram illustrating an example of a three-dimensional object obtained by synthesizing the three-dimensional object illustrated in FIG. 21 and the three-dimensional object illustrated in FIG. 22.

As illustrated in FIG. 21, the synthesized overhead image generation unit 440, for example, uses the projection length L ₁ ′ to the projection length L _{2 of the} three-dimensional object 723-3 obtained from the third camera overhead image. Correct to '. Further, as illustrated in FIG. 22, the synthesized overhead image generation unit 440 projects, for example, the three-dimensional object 723-2 obtained from the second camera overhead image from the projection length L ₁ ″ with the synthesized viewpoint 734 as a reference. Correct to length L ₂ ″. Since the projection lengths L ₂ ′ and L ₂ ″ are both corrected using the composite viewpoint 734 as a reference, they have the same value.

  As shown in FIG. 23, the three-dimensional object 723 obtained by synthesizing the three-dimensional objects 723-3 and 723-4 becomes closer to the actual three-dimensional object and becomes easier to see. Further, by aligning with the projected length when viewed from the composite viewpoint 734, the correlation between the height of the actual three-dimensional object and the projected length on the display is increased, and the driver can adjust the height of the actual three-dimensional object. It becomes possible to recognize more accurately.

  Next, a mode in which the target of image processing is narrowed down to a three-dimensional object near the overlapping area will be described.

  FIG. 24 is a diagram illustrating a state of rotation of the three-dimensional object when only the three-dimensional object overlapping the region near the joint portion of the camera overhead image is a processing target, and corresponds to FIG. In this pattern, the synthesized overhead image generation unit 440 rotates only the three-dimensional object 723 that overlaps the region 741 that is set so as to surround the joint between the adjacent camera overhead images 724 in the three-dimensional object 723.

  FIG. 25 is a diagram illustrating a state of rotation of the three-dimensional object when only the three-dimensional object overlapping the joint portion of the camera overhead image is a processing target, and corresponds to FIG. 11. In this pattern, the combined overhead image generation unit 440 rotates only the three-dimensional object 723 that overlaps the joint 742 between the adjacent camera overhead images 724 among the three-dimensional objects 723.

  FIG. 26 is a diagram illustrating a state of rotation of the three-dimensional object when only the three-dimensional object overlapping the overlapping area is a processing target, and corresponds to FIG. 11. In this pattern, of the three-dimensional object 723, only the three-dimensional object 723 that overlaps the overlapping area 743 of the camera overhead image 724 is rotated.

  FIG. 27 is a diagram illustrating a state of rotation of a three-dimensional object when only a three-dimensional object having a base position in the overlapping area is a processing target, and corresponds to FIG. 11. In this pattern, only the three-dimensional object 723 having the base position 732 in the overlapping area 743 of the camera overhead image 724 is rotated.

  Comparing the processing target determination patterns shown in FIGS. 11 and 24 to 27, the change in the projection direction when the relative position between the vehicle and the three-dimensional object changes due to the movement of the vehicle and the three-dimensional object is shown in FIG. It will be continuous in the pattern shown and will be relatively smooth and natural. On the other hand, it is the pattern shown in FIG. 27 that minimizes the number of processing targets and can reduce the apparatus load most. In the pattern shown in FIG. 27, since the area before rotation and the area after rotation are in the overlapping area, it is possible to complement the data missing area according to the mode described below. From the balance between the tolerance for the apparatus load and the tolerance for the change in the projection direction, an appropriate pattern can be adopted in whole or in part.

  In addition, as shown in FIGS. 24 to 27, when the area to be processed is limited, the three-dimensional object detection unit 430 may be configured to detect a three-dimensional object only for the area.

  Next, a mode of complementing the data missing area using the camera overhead image of the overlapping area will be described.

  FIG. 28 is a diagram illustrating a state of complementing the data missing area using the camera overhead image of the overlapping area. As illustrated in FIG. 28, the combined overhead image generation unit 440, for example, in the overlapping area 743 of the third camera overhead image 724-3 and the second camera overhead image 724-2, -3 to rotate it. In this case, a data missing area 738 occurs in the third camera overhead image 724-3. However, the road surface is projected in an area 744 corresponding to the data missing area 738 of the second camera overhead image 724-2. This is because the camera projection direction differs for each camera overhead image 724.

  Therefore, the synthesized bird's-eye view image generation unit 440 supplements the data missing area 738 with the image of the corresponding region 744 of the second camera bird's-eye view image 724-2. Since the data missing area can be complemented from the combined overhead image of one frame, the processing load and memory load can be reduced as compared with the complementing method described in FIG.

  Next, an aspect of changing the composite viewpoint will be described.

  FIG. 29 is a diagram illustrating an example of a composite overhead image before image rotation processing is performed. FIG. 30 is a diagram illustrating an example of a synthesized overhead image after image rotation processing is performed on the synthesized overhead image illustrated in FIG. 29. FIG. 31 is a diagram illustrating an example of a composite overhead image in which the composite viewpoint is shifted from the composite overhead image illustrated in FIG. 30.

  As shown in FIG. 30, the base position 732-1 of the first three-dimensional object 723-1 and the base position 732-2 of the second three-dimensional object 723-2 are positioned substantially in a straight line with respect to the composite viewpoint 734. is doing. In this case, the second three-dimensional object 723-2 on the side far from the composite viewpoint 734 may be in the shadow of the first three-dimensional object 723-1 and difficult to view.

  Therefore, as shown in FIG. 31, the synthesized overhead image generation unit 440 makes it possible to move the synthesized viewpoint 734 and make it easier to visually recognize the second three-dimensional object 723-2 that is shaded.

  This movement may be performed automatically by determining whether or not the image correction 440 has a plurality of base positions arranged in a straight line as viewed from the combined viewpoint, or may be performed in response to a user operation. . In addition, a fixed position such as the center of the vehicle may be determined in advance as the movement destination, or may be set at an arbitrary position according to the detection result of the base position or a user operation. As a result, as shown in FIG. 29, it is possible to prevent a situation in which the three-dimensional object that originally did not overlap is overlapped by the rotation process, and the visibility is lowered.

  Next, details of a technique for detecting a three-dimensional object by image processing on a camera photographed image will be described.

  As described above, detection of a three-dimensional object by image processing is possible by, for example, pattern matching processing and stereo processing.

  In the case of pattern matching processing, the three-dimensional object detection unit 430 detects, as a three-dimensional object, an image portion having a high correlation with the shape of the external appearance of the three-dimensional object stored in advance, and determines the position of the three-dimensional object from the detected image position of the three-dimensional object. Is detected.

  In the case of stereo processing, the three-dimensional object detection unit 430 uses two images photographed from different positions, and detects the position of the three-dimensional object from the variation in the photographing position and the difference between the two photographed images. Stereo processing includes stereo processing using a binocular camera and motion stereo processing using a monocular camera.

  Here, as a motion stereo process using a monocular camera, a method of detecting a three-dimensional object using a captured image of the camera 200 will be described. Unlike the case of using a binocular camera, when a monocular camera is used, only one captured image can be obtained at a certain time. Therefore, the movement amount of the feature point of the same pattern is obtained using two captured images with different shooting times, and the distance to the three-dimensional object is calculated from the ratio calculation with the movement amount of the vehicle 600.

FIG. 32 is a diagram illustrating an outline of a technique for detecting a three-dimensional object using a camera image. As shown in FIG. 32, it is assumed that the vehicle 600 moves while looking at the three-dimensional object 751 to the left. Then, the three-dimensional object detection unit 430, a first image 752-1 acquires at time _{t 1,} and to obtain a second image 752-2 at time _{t 2.} The first and second images 752-1 and 752-2 are third camera captured images. The three-dimensional object detection unit 430 uses a corner detection filter process, an edge detection filter, or the like to add points (edge points, corners, etc.) that are characteristic of the three-dimensional object from the first image 752-1 to the feature points 753. Detect as. Then, the detected feature point 753 is searched for in the second image 752-2.

The three-dimensional object detection unit 430 calculates the inter-pixel distance D [mm] between the position of the feature point 753 in the first image 752-1 and the position of the feature point 753 in the second image 752-2. It is calculated from the inter-pixel distance [pixel] and the camera parameters of the third camera 200-3. The camera parameter is a parameter that is set when the lens of the camera 200 is designed, and is a parameter that associates the inter-pixel distance in pixel units with the inter-pixel distance in length units. Then, the three-dimensional object detection unit 430, and the focal length of the third camera 200-3 f, the moving distance of the vehicle 600 from time _{t 1} to time _{t 2} is X, the distance from the camera 200 to the three-dimensional object 751 d is calculated using the following equation (4).
d = f · X / D (4)

  Thus, the three-dimensional object detection unit 430 can detect a three-dimensional object only by image processing.

  Further, when detecting the feature point of the three-dimensional object as described above, the three-dimensional object detection unit 430 groups the plurality of detected feature points, thereby generating a composite overhead image surrounded by the grouped feature points. The region may be detected as the coordinates of the three-dimensional object.

  As described above, according to the present embodiment, the projection direction of the three-dimensional object is aligned based on the predetermined viewpoint, so that the driver can more accurately recognize the arrangement relation of the three-dimensional object around the vehicle. Note that the number of cameras and the range of the synthesized overhead image are not limited to the contents of the embodiment described above.

  INDUSTRIAL APPLICABILITY The vehicle surrounding monitoring apparatus and the vehicle surrounding monitoring method according to the present invention are useful as a vehicle surrounding monitoring apparatus and a vehicle surrounding monitoring method that allow a driver to more accurately recognize the arrangement relationship of three-dimensional objects around the vehicle.

1 is a block diagram of a vehicle periphery monitoring device according to an embodiment of the present invention. The perspective view which shows an example of the attachment position of the camera in this Embodiment The figure which shows an example of the picked-up image of the camera in this Embodiment. The figure which shows an example of the camera bird's-eye view image in this Embodiment The figure which shows an example of the synthetic | combination overhead image produced | generated by the camera overhead image production | generation part in this Embodiment. The figure which shows the 1st example of the synthetic | combination overhead image before performing the image rotation process in this Embodiment. The figure which shows an example of the synthetic | combination overhead image after performing the image rotation process in this Embodiment The flowchart which shows an example of operation | movement of the vehicle periphery monitoring apparatus which concerns on this Embodiment. The figure which shows the 2nd example of the synthetic | combination overhead image before performing the image rotation process in this Embodiment. The figure which shows the solid-object position information in this Embodiment. The figure which shows the mode of rotation of the solid object in this Embodiment The figure which shows the solid object at the time of using the 3rd camera bird's-eye view image in this Embodiment. The figure which shows the solid object at the time of using the 2nd camera bird's-eye view image in this Embodiment. The figure which shows the relationship between the distance from the camera and projection length in this Embodiment The figure which shows the mode of a complement of the data missing area in this Embodiment The figure which shows an example of the synthetic | combination overhead image at the time of employ | adopting only the solid object obtained from the 3rd camera overhead image in this Embodiment. The figure which shows an example of the synthetic | combination overhead image at the time of employ | adopting only the solid object obtained from the 2nd camera overhead image in this Embodiment. The figure which shows an example of the synthetic | combination overhead image at the time of employ | adopting the solid object which synthesize | combined the solid object shown in FIG. 16 and the solid object shown in FIG. 17 in this Embodiment. The figure which shows an example of the relationship between the base position of the three-dimensional object in this Embodiment, the image position of a camera, and a synthetic | combination viewpoint. The figure which shows the relationship between the distance from the viewpoint to a solid object, and projection length in this Embodiment The figure which shows an example of the solid thing after the projection length correction | amendment obtained from the 3rd camera bird's-eye view image in this Embodiment. The figure which shows an example of the solid object after the projection length correction | amendment obtained from the 2nd camera overhead view image in this Embodiment. The figure which shows an example of the solid object which synthesize | combined the solid object shown in FIG. 21 and the solid object shown in FIG. 22 in this Embodiment. The figure which shows the mode of rotation of a solid object at the time of setting only the solid object which overlaps the area | region of the junction part vicinity of the camera overhead view image in this Embodiment as a process target. The figure which shows the mode of rotation of a solid object at the time of setting only the solid object which overlaps the junction part of the camera overhead view image in this Embodiment as a process target. The figure which shows the mode of rotation of the solid object at the time of setting only the solid object which overlaps the overlap area in this Embodiment as a process target. The figure which shows the mode of rotation of the solid object at the time of setting only the solid object which has a base position in the overlapping area in this Embodiment as a process target. The figure which shows the mode of a complement of a data missing area using the camera overhead view image of the duplication area in this Embodiment The figure which shows an example of the synthetic | combination overhead image before performing the image rotation process in this Embodiment The figure which shows an example of the synthetic | combination overhead image after performing an image rotation process with respect to the synthetic | combination overhead image shown in FIG. 29 in this Embodiment. The figure which shows an example of the synthetic | combination overhead view image which shifted the synthetic | combination viewpoint from the synthetic | combination overhead image shown in FIG. 30 in this Embodiment. The figure which shows the outline | summary of the method of detecting a solid object using the camera picked-up image in this Embodiment. The figure which shows an example of the camera arrangement | positioning in the conventional vehicle surroundings monitoring apparatus, and a camera bird's-eye view image The figure which shows an example of a 3rd camera overhead image when a solid object is located in the duplication area in the conventional vehicle periphery monitoring apparatus. The figure which shows an example of a 2nd camera bird's-eye view image when a solid object is located in the duplication area in the conventional vehicle periphery monitoring apparatus. The figure which shows the 1st example of the synthetic | combination overhead view image in the conventional vehicle periphery monitoring apparatus. The figure which shows the 2nd example of the synthetic | combination overhead view image in the conventional vehicle periphery monitoring apparatus. The figure which shows the 3rd example of the synthetic | combination overhead view image in the conventional vehicle periphery monitoring apparatus. The figure which shows the 4th example of the synthetic | combination overhead view image in the conventional vehicle periphery monitoring apparatus.

Explanation of symbols

DESCRIPTION OF SYMBOLS 100 Vehicle surroundings monitoring apparatus 200 Camera 210 Imaging part 220 Frame memory 230 Frame switching part 400 Image processing part 410 Mapping table reference part 420 Camera overhead view image generation part 430 Three-dimensional object detection part 440 Synthetic overhead view image generation part 450 Timing generation part 500 Monitor

Claims (10)

A three-dimensional object detection unit for detecting a three-dimensional object existing around the vehicle;
Based on the images taken by a plurality of cameras, a bird's-eye view image around the vehicle is synthesized to generate a synthesized bird's-eye view image, and based on the detection result by the three-dimensional object detection unit, the three-dimensional object shown in the generated synthesized bird's-eye image is viewed from a predetermined viewpoint. A first image generation unit that corrects the image to be rotated in the viewed projection direction;
A vehicle surroundings monitoring device comprising: The predetermined viewpoint is a viewpoint position of a driver of the vehicle;
The vehicle surrounding monitoring apparatus according to claim 1. There are overlapping areas that overlap each other in the shooting range of the plurality of cameras,
The first image generation unit corrects at least the three-dimensional object existing in the overlap area in the composite overhead image rotated in a projecting direction viewed from a predetermined viewpoint.
The vehicle surrounding monitoring apparatus according to claim 1. A second image generation unit that generates a camera overhead view of the surroundings of the vehicle for each captured image by the plurality of cameras;
The first image generation unit includes:
Generating a combined overhead image based on the plurality of camera overhead images generated by the second image generation unit;
A three-dimensional object is extracted from a plurality of camera overhead images generated by the second image generation unit based on a detection result by the three-dimensional object detection unit, and the extracted three-dimensional object is rotated in a projection direction viewed from a predetermined viewpoint. In the state of letting you draw on the generated synthetic overhead image,
The vehicle surrounding monitoring apparatus according to claim 1. The first image generation unit further corrects the length of the three-dimensional object shown in the generated composite overhead image so as to be a projection length on the road surface when the three-dimensional object is viewed from the predetermined viewpoint.
The vehicle surrounding monitoring apparatus according to claim 1. The first image generation unit corrects all three-dimensional objects reflected in the synthesized overhead image to those rotated in a projection direction viewed from a predetermined viewpoint,
The vehicle surrounding monitoring apparatus according to claim 1. The first image generation unit further draws a road surface using a camera overhead image on the portion of the three-dimensional object originally reflected in the generated synthesized overhead image.
The vehicle surrounding monitoring apparatus according to claim 4. The first image generation unit further draws a road surface using a composite overhead image generated in the past on a part of the three-dimensional object originally reflected in the currently generated composite overhead image.
The vehicle surrounding monitoring apparatus according to claim 4. A monitor that displays the combined overhead image generated by the first image generation unit;
The vehicle surrounding monitoring apparatus according to claim 1. A first step of detecting a three-dimensional object existing around the vehicle;
Based on the captured images of the plurality of cameras, a bird's-eye view image around the vehicle is combined to generate a combined bird's-eye view image. Based on the detection result in the first step, a three-dimensional object reflected in the generated combined bird's-eye view image is obtained from a predetermined viewpoint. A second step for correcting to a rotated view in the projected direction;
A vehicle surroundings monitoring method comprising:

Images