JP2003132349A - Drawing equipment - Google Patents

Abstract

(57) [Summary] [PROBLEMS] To provide a drawing device for generating a driving support image that does not easily give a driver an uncomfortable feeling. A drawing device detects three-dimensional coordinates of an obstacle in an image acquired by an imaging unit based on measurement values from a vehicle speed sensor and a steering angle sensor. In the drawing apparatus 100, the plane model extraction unit 105 calculates a plane approximating the area where the obstacle is present. The drawing apparatus 100 sets the imaging unit 101 on the calculated plane.
The synthesized image is created by projecting the image acquired in step (1) and displayed on the display unit 107.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a drawing device,
More specifically, it relates to a drawing device that can be incorporated in a driving support device. More specifically, the present invention relates to a drawing device that generates a driving assistance image representing a situation around a vehicle based on a captured image captured by an imaging device fixed to the vehicle.

[0002]

2. Description of the Related Art As an example of a conventional driving support device, there is a vehicle surroundings monitoring device disclosed in International Publication WO00 / 07373. The vehicle periphery monitoring device disclosed in International Publication WO00 / 07373 includes a plurality of image capturing devices for capturing an image of the entire periphery of the vehicle.
Different imaging areas are assigned to the respective imaging devices. Here, the shooting area is a part of the periphery of the vehicle, and all the shooting areas include at least the rear direction of the vehicle. Each of the above imaging devices captures an image (hereinafter, referred to as a captured image) representing the situation of its own imaging region.

The vehicle surroundings monitoring device disclosed in International Publication WO00 / 07373 discloses a vehicle surroundings (including a rearward direction of the vehicle) from directly above, based on a photographed image captured by each imaging device. Generate one peripheral image that you see. More specifically, the vehicle surroundings monitoring device performs a viewpoint conversion process on the captured image captured by each image capturing device to generate a peripheral image of the vehicle surroundings viewed from directly above. In the viewpoint conversion process, in order to reduce the processing load on the vehicle surroundings monitoring device, it is assumed that all three-dimensional objects shown in each captured image are present on the road surface. The vehicle periphery monitoring device projects spatial objects for capturing a peripheral image by projecting a three-dimensional object in a captured image onto a road surface with the image capturing device as a viewpoint. Based on the above spatial data, the vehicle periphery monitoring device synthesizes one periphery image from a plurality of captured images.

The user operates the vehicle (parking at the back, etc.) while referring to the peripheral image (including the image in the rear direction of the vehicle) displayed by the vehicle surroundings monitoring device.

[0005]

However, the above-described vehicle surroundings monitoring device has a problem that a three-dimensional object existing around the vehicle may not be drawn correctly on the surrounding image. More specifically, as shown in FIG. 30, the road surface Frd
Assume there is a solid B having an inverted "L" shaped cross section above. When the above three-dimensional object B is photographed by the imaging device 2001, the vertical portion Bv and the horizontal portion B of the three-dimensional object B are captured.
h is reflected in the captured image S3001. In the viewpoint conversion process described above, the three-dimensional object B is projected onto the road surface Frd with the imaging device 2001 as the viewpoint. In FIG. 30, the imaging device 20 is further provided.
A projection portion Bv ′ in which the above portion Bv is projected on the road surface Frd is drawn with 01 as a viewpoint. Therefore, by the above-mentioned viewpoint conversion processing, the vehicle periphery monitoring device treats the virtual object B ′ composed of the portion Bh and the projection portion Bv ′ as a peripheral image viewed from directly above and synthesizes the peripheral image.

However, the virtual object B ′ looks much deformed compared to the object B seen from the image pickup apparatus 2001. Specifically, the virtual object B ′ is generated as if the vertical portion Bv of the object B was greatly stretched. As a result, the vehicle periphery monitoring device disclosed in International Publication WO00 / 07373 cannot correctly draw a three-dimensional object existing in the periphery of the vehicle on the periphery image, and generates a periphery image that gives the driver a sense of discomfort. Was.

[0007] Therefore, an object of the present invention is to provide a drawing device for generating a driving support image which does not give a driver a feeling of strangeness.

[0008]

A first aspect of the present invention is a drawing device for creating a display image for driving assistance of a vehicle, which is provided from a camera installed in the vehicle to the periphery of the vehicle. Image acquisition means for acquiring the image of the three-dimensional object, and a three-dimensional object detection means for detecting the existence area of the three-dimensional object around the vehicle based on the image acquired by the captured image acquisition means, and the three-dimensional object detected by the three-dimensional object detection means. By projecting the image acquired by the captured image acquisition means onto the plane extracted by the plane model extracting means for extracting the plane approximating the existing area, and the plane model extracting means,
And an image synthesizing unit for creating a synthetic image to be displayed.

According to the first aspect of the present invention, the drawing device can extract a plane approximating the existence area of the three-dimensional object existing around the vehicle, and by projecting the imaged image on the extracted plane, It is possible to reduce the distortion of the composite image.

A second invention is according to the first invention, wherein the image synthesizing section projects a mapping table for projecting the image acquired by the captured image acquiring means onto the plane extracted by the surface model extracting means. And a mapping table correcting means for correcting the mapping table based on the planes sequentially extracted by the three-dimensional object detecting means, and an image synthesizing means for sequentially creating a synthetic image based on the mapping table.

According to the second aspect of the present invention, the drawing apparatus sequentially sets the mapping table for projecting the imaged image on the plane that approximates the existing area of the three-dimensional object, so that the image is projected on the plane. It is possible to create a composite image without enormous calculation.

A third aspect of the invention is an aspect according to the second aspect of the invention, in which the image synthesizing unit pastes a CG for pasting in an area on the synthetic image where data is treated as not existing in the mapping table. The image synthesizing unit further includes a CG data creating unit that creates an image, and the image synthesizing unit creates a composite image by pasting the CG image created by the CG data creating unit onto an area of the composite image that is treated as having no data. It is characterized by doing.

According to the third aspect of the invention, the drawing apparatus creates a CG image to be pasted in an area on the composite image which is treated as having no data in the mapping table, and creates the composite image. , It is possible to create a composite image of the vehicle surroundings that is easier for the driver to see.

A fourth invention is an invention according to the first invention, further comprising a vehicle movement amount detecting means for detecting a movement amount of the vehicle, wherein the three-dimensional object detecting means is detected by the vehicle movement amount detecting means. Based on the amount of movement of the vehicle, the image before movement and the image after movement are compared to calculate the distance to the three-dimensional object captured in the image acquired by the captured image acquisition means, The present invention is characterized by detecting the existence area of a three-dimensional object.

According to the fourth aspect, the drawing device can indirectly detect the existence area of the three-dimensional object based on the moving amount of the vehicle and the captured images before and after the movement. Therefore, the drawing device does not need to use an active range-finding sensor such as an ultrasonic sensor or a laser radar sensor that directly transmits a radio wave or the like to the three-dimensional object and reflects it to measure the distance to the three-dimensional object. It is possible to reduce the number of sensors used. Further, since the active distance measuring sensor and the like need to be installed in the vicinity of the vehicle, the appearance of the vehicle is impaired. However, in the fourth invention, the three-dimensional object around the vehicle is not impaired without impairing the appearance of the vehicle. It becomes possible to detect the existing area.

A fifth aspect of the invention is an invention according to the fourth aspect of the invention, wherein the vehicle movement amount detection means is based on the measured values of a vehicle speed sensor and a steering angle sensor installed in the vehicle, Is detected.

According to the fifth aspect of the invention, when the drawing device obtains the moving amount of the vehicle from the vehicle speed sensor and the steering angle sensor installed in the vehicle, the drawing image indirectly captures the existence area of the three-dimensional object. Can be detected.

A sixth invention is an invention according to the fourth invention, wherein the future existence of the three-dimensional object detected by the three-dimensional object detecting means is detected based on the moving amount of the vehicle detected by the vehicle moving amount detecting means. The plane model extracting means further includes a predicting means for predicting the area, and the plane model extracting means extracts the plane based on the existing area of the three-dimensional object predicted by the predicting means.

According to the sixth aspect of the present invention, the future area of existence of the three-dimensional object can be predicted. Therefore, even if the three-dimensional object detecting means cannot sufficiently detect the area of existence of the three-dimensional object, the prediction is possible. The plane can be determined based on the area. Therefore, it is possible to provide the driver with a display image that is not interrupted on the way.

A seventh invention is an invention according to the first invention, wherein the three-dimensional object detecting means determines the three-dimensional coordinates of points on the edge forming the contour of the three-dimensional object to determine the existence area of the three-dimensional object. And the plane model extracting means extracts a plane that approximates the existing area of the three-dimensional object, based on the three-dimensional coordinates of the points on the edge determined by the three-dimensional object detecting means.

According to the seventh aspect, the drawing apparatus extracts the plane based on the existence area of the three-dimensional object represented by the three-dimensional coordinates. Therefore, a more accurate plane approximates the existence area of the three-dimensional object. You can ask. Therefore, the driver
It is possible to accurately perform a driving operation such as parking by looking at the displayed composite image.

An eighth invention is an invention according to the first invention, wherein the image synthesizing section pastes a model diagram showing the own vehicle to create a synthetic image.

According to the eighth aspect of the invention, since the drawing device pastes the model diagram of the own vehicle on the composite image, the driver can clearly recognize the positional relationship between the own vehicle and the three-dimensional object. It is possible to create a composite image around the vehicle that is easier to see.

A ninth aspect of the present invention is a drawing device for producing a display image for driving assistance of a vehicle, and a picked-up image obtaining means for obtaining a peripheral image of the vehicle from a camera installed in the vehicle, Based on the image acquired by the captured image acquisition means, a three-dimensional object detection means for detecting the existence area of the three-dimensional object around the vehicle, and an area for extracting the area when the three-dimensional object detected by the three-dimensional object detection means is projected on the road surface. Based on the extraction means, the area extracted by the area extraction means, a CG image creation means for creating a CG image showing the existing area of the three-dimensional object projected on the road surface, and the image acquired by the captured image acquisition means on the road surface. C created by the CG image creating means on the image projected and obtained by projection
An image combining unit that overlaps the G images and creates a combined image to be displayed.

According to the ninth aspect of the invention, the drawing device extracts a region when the three-dimensional object is projected on the road surface, creates a CG image showing the region, and creates a composite image to be displayed. . This allows the driver to recognize the existence area of the three-dimensional object as an image viewed from above the vehicle.

A tenth invention is an invention according to the ninth invention, further comprising a vehicle movement amount detecting means for detecting a movement amount of the vehicle, wherein the three-dimensional object detecting means is detected by the vehicle movement amount detecting means. Based on the moving amount of the vehicle, the distance to the three-dimensional object captured in the image acquired by the captured image acquisition unit is calculated, and the existence area of the three-dimensional object around the vehicle is detected.

An eleventh invention is an invention dependent on the tenth invention, wherein the vehicle movement amount detecting means is based on measured values of a vehicle speed sensor and a steering angle sensor installed in the vehicle.
It is characterized in that the amount of movement of the vehicle is detected.

A twelfth aspect of the invention is an invention dependent on the tenth aspect of the invention, wherein the presence region of the three-dimensional object detected by the three-dimensional object detection means is detected based on the amount of movement of the vehicle detected by the vehicle movement amount detection means. Further including a predicting means for predicting, the area extracting means, based on the existence area of the three-dimensional object predicted by the predicting means,
It is characterized by extracting a region when a three-dimensional object is projected on a road surface.

According to the twelfth aspect of the present invention, since the future existence area of the three-dimensional object can be predicted, even if the three-dimensional object detection means cannot sufficiently detect the existence area of the three-dimensional object, the prediction is possible. The area of the three-dimensional object projected on the road surface can be obtained based on the area. Therefore, it is possible to provide the driver with a display image that is not interrupted on the way.

A thirteenth invention is an invention according to the ninth invention, and further, a vehicle area for extracting an existing area of another vehicle existing around the vehicle based on the image acquired by the captured image acquisition means. The CG image creating unit includes an extracting unit and a vehicle type determining unit that determines the vehicle type of the other vehicle based on the existing region of the other vehicle extracted by the vehicle region extracting unit. It is characterized in that a CG image is created by pasting a model diagram of a vehicle according to the vehicle type determined by the vehicle type determination means.

According to the thirteenth aspect of the invention, the drawing device creates a composite image by making a model diagram that makes it easy to see other vehicles existing in the vicinity of the vehicle. Therefore, it is possible to create a composite image that does not make the driver feel uncomfortable. Becomes

A fourteenth invention is an invention according to the ninth invention, wherein the three-dimensional object detecting means determines the three-dimensional coordinates of points on the edge forming the contour of the three-dimensional object to determine the existence area of the three-dimensional object. And the area extracting means extracts the area when the three-dimensional object is projected on the road surface based on the three-dimensional coordinates of the points on the edge determined by the three-dimensional object detecting means.

According to the fourteenth aspect, the drawing device extracts the area projected on the road surface based on the existence area of the three-dimensional object represented by the three-dimensional coordinates. Therefore, the area on the road surface is more accurate. You can ask. Therefore, the driver
It is possible to accurately perform a driving operation such as parking by looking at the displayed composite image.

A fifteenth aspect of the present invention is a drawing method for creating a display image for supporting driving of a vehicle, which comprises a captured image obtaining step of obtaining a peripheral image of the vehicle from a camera installed in the vehicle, Based on the image acquired in the captured image acquisition step, a three-dimensional object detection step that detects a three-dimensional object existing area around the vehicle, and a surface model that extracts a plane that approximates the three-dimensional object existing area detected in the three-dimensional object detection step An extraction step and an image combination step of projecting the image acquired in the captured image acquisition step onto the plane extracted in the surface model extraction step to create a combined image to be displayed.

A sixteenth aspect of the present invention is a drawing method for creating a display image for driving assistance of a vehicle, comprising a captured image obtaining step of obtaining a peripheral image of the vehicle from a camera installed in the vehicle, Based on the image acquired in the captured image acquisition step, a three-dimensional object detection step that detects the existence area of the three-dimensional object around the vehicle, and an area that extracts the area when the three-dimensional object detected in the three-dimensional object detection step is projected on the road surface. Based on the extraction step and the area extracted in the area extraction step, the CG image creation step of creating a CG image showing the existing area of the three-dimensional object projected on the road surface, and the image acquired in the captured image acquisition step on the road surface. Image composition for projecting and superimposing the CG image created in the CG image creating step on the image obtained by projection to create a composite image to be displayed And a step.

A seventeenth aspect of the present invention is a recording medium in which a program for creating a display image for supporting driving of a vehicle is recorded, which is an image pickup for acquiring a peripheral image of the vehicle from a camera installed in the vehicle. An image acquisition step, based on the image acquired in the captured image acquisition step, a three-dimensional object detection step of detecting the existence area of the three-dimensional object around the vehicle,
A plane model extraction step that extracts a plane that approximates the existing area of the three-dimensional object detected in the three-dimensional object detection step, and the image acquired in the captured image acquisition step should be projected and displayed on the plane extracted in the surface model extraction step. An image combining step of creating a combined image.

An eighteenth aspect of the present invention is a recording medium in which a program for creating a display image for driving assistance of a vehicle is recorded, and is an image pickup for acquiring a peripheral image of the vehicle from a camera installed in the vehicle. An image acquisition step, based on the image acquired in the captured image acquisition step, a three-dimensional object detection step of detecting the existence area of the three-dimensional object around the vehicle,
A region extraction step of extracting a region when the three-dimensional object detected in the three-dimensional object detection step is projected onto a road surface, and a CG image showing the existing region of the three-dimensional object projected onto the road surface based on the region extracted in the region extraction step The image acquired in the captured image acquisition step and the image acquired in the captured image acquisition step are projected onto the road surface, and the image obtained by projection is overlapped with the CG image created in the CG image creation step to create a composite image to be displayed. An image synthesizing step.

A nineteenth aspect of the present invention is a program for creating a display image for driving support of a vehicle, comprising: a captured image acquisition step of acquiring a peripheral image of the vehicle from a camera installed in the vehicle; Based on the image acquired in the image acquisition step, a three-dimensional object detection step that detects a three-dimensional object existing area around the vehicle, and a surface model extraction that extracts a plane that approximates the three-dimensional object existing area detected in the three-dimensional object detection step And an image composition step of projecting the image acquired in the captured image acquisition step onto the plane extracted in the surface model extraction step to create a combined image to be displayed.

A twentieth aspect of the invention is a program for creating a display image for driving assistance of a vehicle, a captured image acquisition step of obtaining a peripheral image of the vehicle from a camera installed in the vehicle, and a captured image acquisition step. Based on the image obtained in, the three-dimensional object detection step of detecting the existence area of the three-dimensional object around the vehicle, the area extraction step of extracting the area when the three-dimensional object detected in the three-dimensional object detection step is projected on the road surface, Based on the area extracted in the area extraction step, a CG image creation step of creating a CG image showing the existing area of the three-dimensional object projected on the road surface, and an image acquired in the captured image acquisition step are projected on the road surface and projected. The image composition step for superimposing the CG image created in the CG image creation step on the image obtained by Provided with a door.

[0040]

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS (First Embodiment) FIG. 1 is a block diagram showing a functional configuration of a drawing apparatus according to a first embodiment of the present invention. In FIG. 1, the drawing apparatus 100 is
Imaging unit 101, vehicle speed sensor 102, steering angle sensor 1
03, an obstacle detection unit 104, and a surface model extraction unit 105.
, Mapping table correction unit 108, and image composition unit 1
06 and the display unit 107.

The image pickup section 101 includes a plurality of cameras 10 mounted around the vehicle. Here, the camera 10 (# number) is used to distinguish a plurality of cameras. Each camera will be generically referred to as camera 10. Each camera 10 is typically a color or monochrome digital camera having a fixed image sensor such as a CCD or CMOS device. The camera 10 can capture an image around the vehicle. In FIG. 1, the imaging unit 101 includes two cameras 10
(# 1) and 10 (# 2) are included. The number of cameras 10 may be two or more, or less than two.

FIG. 2 shows two cameras 10 (# 1) and 10 (# 1).
It is a figure showing an example of the attachment position in the vehicle of (# 2). As shown in FIG. 2, the camera 10 (# 1) is attached to a position where the right rear portion of the vehicle can be imaged. The camera 10 (# 2) is attached to a position where the left rear part of the vehicle can be imaged. It should be noted that the camera 10 may be attached to a position that can image the surroundings of the vehicle such as a side surface portion or a front surface portion of the vehicle in addition to the rear portion of the vehicle. The image capturing unit 101 sequentially acquires the data of the image around the vehicle captured by the camera 10 (hereinafter, referred to as image data) from the obstacle detecting unit 10.
4 and the image synthesis unit 106.

The vehicle speed sensor 102 sequentially measures the wheel speed of the vehicle equipped with the drawing apparatus 100, and sends the measured speed to the obstacle detection unit 104. Steering angle sensor 103
Sequentially measures the steering wheel steering angle of the vehicle equipped with the drawing apparatus 100, and sends the measured steering wheel steering angle to the obstacle detection unit 104.

The obstacle detection unit 104 is a vehicle speed sensor 102.
Wheel speed of the vehicle given from the steering angle of the vehicle given from the steering angle sensor 103 and the imaging unit 101
Based on the image data of the surroundings of the vehicle given from the above, the coordinate data of the points forming the obstacle (three-dimensional object) around the vehicle shown in the image captured by the imaging unit 101 is calculated. Here, the coordinate data of the points forming the obstacle (three-dimensional object) are coordinate values in a coordinate system with the camera 10 as the origin (hereinafter, referred to as camera coordinates). The obstacle detection unit 104 sends the calculated coordinate data to the surface model extraction unit 105.

The plane model extraction unit 105 includes an obstacle detection unit 1
Based on the coordinate data of the three-dimensional obstacle given from 04,
The surface area of a plane perpendicular to the road surface that approximates the area where the steric obstacle exists is calculated. The surface model extraction unit 105 sends the calculated data regarding the surface area of the flat surface to the mapping table correction unit 108.

The mapping table modification unit 108 includes a standard mapping table 1081 and a modified mapping table 1082. Standard mapping table 1081
Is a table for performing a process of converting the position coordinates of pixels included in the image data into the position coordinates of pixels in the composite image at high speed. Standard mapping table 1081
Defines a conversion for processing and synthesizing a captured image into an image viewed from the upper part of the vehicle under the assumption that all images captured by the camera 10 are present on the road surface.

FIG. 3 shows a standard mapping table 1801.
It is a figure which shows an example. Standard mapping table 180
1 shows how the pixel values at all coordinates (i, j) of the composite image are determined. Specifically, for any coordinate (i, j), the X coordinate, the Y coordinate, and the necessity degree are set in association with the camera number. Here, the X coordinate and the Y coordinate are coordinates in an image captured by the corresponding camera. The necessity degree is a coefficient indicating how much the pixel value corresponding to the X coordinate and the Y coordinate is used. For example, in FIG. 3, the pixel value of the coordinate (i1, j1) of the composite image is obtained by converting the pixel value of the coordinate (12, 45) of the image captured by the camera 10 (# 1) with the necessity degree of "1". Will be things. The necessity level being “1” means that the pixel value of the captured image is copied as it is and used. Also, the coordinates (i2, j2) of the composite image
The pixel value of is captured by the camera 10 (# 2) by multiplying the pixel value of the coordinates (10, 10) of the image captured by the camera 10 (# 1) by the necessity degree "0.3". The sum of the pixel value of the image coordinates (56, 80) multiplied by the necessity degree "0.5" is the sum of the respective necessity degrees, in this case 0.8.
It is the value divided by.

The modified mapping table 1082 is a table in which the contents of the standard mapping table 1081 have been modified. The mapping table correction unit 108 corrects the standard mapping table 1081 based on the surface area data sent from the surface model extraction unit 105, and creates a corrected mapping table 1802.

FIG. 4 shows the standard mapping table 1081.
FIG. 9 is a diagram for intuitively understanding a modified mapping table 1802. FIG. 4A is a diagram for intuitively understanding the standard mapping table 1081. In FIG. 4A, the pixel at each coordinate of the image gb captured by the camera 10 (# 1) is the standard mapping table 10
By 81, it becomes the pixel of the coordinate of the corresponding composite image ga. The pixels of the image gb are the pixels within the frame of the dotted line of the image ga.

Image gb taken by the camera 10 (# 1)
Is a rectangle on the road (for example, a white line indicating a parking frame)
Is imaged. The image gb shows the camera 10 (#
Since the image is taken from the viewpoint 1), the rectangle on the road surface becomes a distorted rhombus. Standard mapping table 180
1 defines the conversion for processing and synthesizing the image captured by the camera 10 into the image viewed from the upper part of the vehicle under the assumption that all the images captured by the camera 10 exist on the road surface. Therefore, the rectangle (the distorted rhombus of the image gb) present on the road surface imaged by the camera 10 (# 1) becomes a non-distorted rectangle in the composite image ga by the conversion in the standard mapping table 1801.

FIG. 4B shows the modified mapping table 1
FIG. 8 is a diagram for intuitively understanding 802. As described in the related art, when the standard mapping table 1801 is used to process and combine an image of an obstacle existing above the road surface, the image becomes a distorted image, and the image cannot be accurately processed and combined. . The further away from the road surface, the more distorted the image.

The modified mapping table 1802 defines transformations for compressing and projecting an object above the road surface to create a composite image. In FIG. 4B, the image gbs captured by the camera 10 (# 1) includes
One pillar is imaged as an obstacle. By the conversion defined in the correction mapping table 1802, the one surfaces A and B of the pillar imaged in the image gbs are compressed and drawn in the composite image gas. The other side of the pillar that is reflected in the image gbs that cannot be seen from the camera 10 is treated as having no data, and the composite image ga
In s, a black image is obtained.

The image synthesizing unit 106 receives the image data from the image pickup unit 101, and then the modified mapping table 10
With reference to 82, the image data is processed to create a composite image. The image composition unit 106 sends the created composite image to the display unit 107. The display unit 107 displays the combined image sent from the image combining unit 106 and presents it to the user.

FIG. 5 shows a drawing apparatus 1 according to the first embodiment.
FIG. 10 is a diagram for intuitively understanding how the captured image is processed until 00 creates a composite image. FIG. 5A is a diagram showing an example of an image captured by the camera 10 (# 1) of the image capturing unit 101. FIG. 5A shows that one passenger car is parked in the white line parking frame. The portion Jsh shown at the bottom of the image in FIG.
Is a vehicle equipped with the camera 10 (# 1). The drawing device 100 displays a flat area (a parked passenger vehicle existence area) in which the parked passenger vehicle is completely covered by the camera 10 (# 1) from the captured image shown in FIG. Area of the plane that approximates

FIG. 5B is a diagram showing a plane area determined by the drawing apparatus 100. In FIG. 5B, two planes C and D indicated by diagonal lines are planes that completely cover the passenger car. After determining the plane area, the drawing apparatus 100 compresses and projects the image data on the plane area according to the modified mapping table 1082 to complete the composite image. The area on the other side of the plane area that cannot be seen from the camera 10 (# 1) is treated as having no data, and a composite image is formed as a black pixel. FIG. 5C is a diagram showing a composite image finally created by the drawing apparatus 100. In FIG. 5C, the camera 10
The part surrounded by the dotted line from (# 1) is the camera 10 (#
The image pickup area of 1) is shown. As shown in FIG. 5C, in the composite image, the image of the vehicle parked in the parking lot is compressed and projected on the surface perpendicular to the road surface. As a result, the composite image is created without the vehicle parked in the parking lot being significantly distorted. In addition, the other side of the plane (camera 10
(Beyond the vehicle when viewed from (# 1))
The image becomes black.

Here, the coordinate system used in the following description will be described. FIG. 6 is a diagram for explaining the camera coordinate system and the vehicle coordinate system. As shown in FIG. 6, the coordinate system centered on the camera 10 will be referred to as a camera coordinate system. In the camera coordinate system, the coordinate value is Pc
It will be represented by (xc, yc, zc). A coordinate system centered on a certain point on the road surface under the vehicle is called a vehicle coordinate system. In the vehicle coordinate system, the coordinate value is Pv (x
v, yv, zv). The point (xv, yv, zv) in the vehicle coordinate system is replaced by the point (xc, y in the camera coordinate system.
To convert to (c, zc), (xc, yc, zc) = R (xv, yv, zv) + T (1) is used. Where R is a 3 × 3 rotation matrix,
T is a three-dimensional translation vector.

FIG. 7 shows a drawing apparatus 1 according to the first embodiment.
10 is a flowchart showing an operation of 00. Below, FIG.
The operation of the drawing apparatus 100 will be described with reference to FIG. First, the drawing apparatus 100 captures an image around the vehicle and acquires image data around the vehicle (step S101). Next, the drawing apparatus 100 detects coordinate data in the camera coordinate system indicating the area where the obstacle exists from the acquired image data (step S102). Next, the drawing apparatus 100, based on the acquired coordinate data indicating the existence region of the obstacle,
The area in which the obstacle is present is approximated, and a plurality of plane surface areas that completely cover the obstacle are extracted (step S10).
3).

Next, the drawing apparatus 100 corrects the standard mapping table 1081 based on the extracted surface area, and creates the modified mapping table 1082 in which the conversion for projecting the imaging data on the extracted surface area is defined. Create (step S104). After that, the drawing apparatus 100
Based on the created correction mapping table 1082,
The part of the image pickup unit 101 that projects the image data acquired by each camera 10 onto the surface area is projected onto the surface area, and the portion onto the road surface is projected onto the road surface as it is and combined to create and display a display image. (Step S105).

FIG. 8 is a flowchart showing in more detail the obstacle coordinate data detection processing (step S102) in FIG. Hereinafter, the obstacle coordinate data detection processing will be described with reference to FIG. 7. First, the drawing apparatus 100 extracts the edge of the acquired image by the processing of the 3 × 3 Sobel filter (step S20).
1). Here, the term “edge” refers to a boundary between image regions existing in the acquired image. The processing by the Sobel filter is processing for extracting edges that are generally used in the field of image processing. Briefly described, in the processing by the Sobel filter, nine pixel values in the upper, lower, left, and right centering on a certain coordinate are multiplied by a predetermined coefficient to calculate the sum, and the coordinate whose calculation result fluctuates greatly is edged. To be In the processing by the Sobel filter, the above edge detection is performed in the vertical direction and the horizontal direction.

Next, the drawing apparatus 100 extracts the inflection point of the extracted edge (step S202). The extracted inflection points are called corresponding points of the edges. Next, the drawing apparatus 100 measures the camera movement amount (Rc, Tc) based on the wheel speed and the steering wheel steering angle detected by the vehicle speed sensor 102 and the steering angle sensor 103 (step S203). Here, Rc is from a certain time t to a time (t
+1) is a 3 × 3 rotation matrix indicating the rotation direction in which the camera 10 has moved in space. Tc is a three-dimensional movement vector indicating the parallel movement direction of the camera 10 moving in space from a certain time t to a time (t + 1). A method of measuring the camera movement amount (Rc, Tc) will be described in detail later.

Next, the drawing apparatus 100 returns to step S20.
The coordinate values on the image data of the corresponding points of all the edges extracted in 2 are acquired (step S204). Next, the drawing apparatus 100, based on the triangulation principle (stereo ranging principle), uses the coordinate values of the corresponding points of the image data and the camera movement amount (Rc, Tc) to correspond to all edges. The coordinates (xc, yc, zc) of the point in the camera coordinate system are detected (step S205). The coordinates (xc, y) in the camera coordinate system of the corresponding point of the edge in step S205.
The method of detecting c, zc) will be described in detail later.

Next, the drawing apparatus 100 returns to step S20.
The coordinates (x in the camera coordinate system) of the corresponding points of the edges obtained in 5
(c, yc, zc) is inversely transformed into coordinates (xv, yv, zv) in the vehicle coordinate system using the equation (1), and the corresponding point of the edge is referred to on the road surface by referring to this zv value. Whether or not it exists (specifically, | zv | <ε (ε = 50 mm)
Then, all edges on the road surface are removed (step S206). After that, the drawing apparatus 100
The remaining edge is determined to be the edge of the three-dimensional obstacle (step S207), and the process ends.

In the process of step S206,
The drawing apparatus 100 may remove edge data far from the vehicle in order to reduce the amount of calculation. Considering parallel parking and the like, there is no problem even if an edge beyond 8 m is removed.

FIG. 9 is a diagram for explaining the process of step S203 in FIG. 8 (camera movement amount (Rc, Tc) measurement process) in more detail. The process of step S203 will be described below with reference to FIG. Since the parking operation is performed at a low speed, it can be approximated that the tire does not skid. Here, the “Ackerman model (two-wheel model)” that does not consider the skidding of the tire is used. The assumptions of the Ackerman model are: (Assumption 1): The sideslip of the tire does not occur. (Assumption 2): The steering angle of the steering wheel and the turning angle of the front wheels are in a proportional relationship. Is.

Based on (Assumption 1), when there is no skidding on the tires, when the steering is turned off, the vehicle turns around a point O on the extension line of the rear wheel axle. The turning radius Rs at the center of the rear wheel is expressed as Rs = 1 / tan β (2) using the cutting angle β of the front wheel and the wheel base 1.

Assuming that the center of the rear wheel moves from Ct to Ct + 1, the moving amount h is calculated by using the left and right wheel speeds Vl, Vr or the turning radius Rs of the center of the rear wheel and the moving rotation angle α as follows. It is represented by a mathematical formula.

[Equation 1]

From the expressions (2) and (3), the moving rotation angle α is expressed as follows.

[Equation 2]

The movement vector Tc from Ct to Ct + 1 is expressed as follows when the X axis is in the vehicle traveling direction and the Y axis is in the direction perpendicular to the vehicle traveling direction.

[Equation 3]

Based on (Assumption 2), assuming that the steering angle of the steering wheel is proportional to the turning angle of the front wheels, the turning angle β of the front wheels is β.
Is expressed as follows using the steering angle θ of the steering wheel. β = kθ (k is a proportional constant) (6)

Finally, according to the equations (4), (5) and (6), the two-dimensional rotation amount α of the vehicle and the movement vector T are obtained by determining the left and right wheel speeds Vl and Vr of the rear wheels and the steering wheel steering. It is expressed as follows using the angle θ.

[Equation 4]

Therefore, when the position coordinate of the camera in the vehicle coordinate system at time t is Xv (t), the relational expression in the vehicle coordinate system when moving from time t to time (t + 1) is:
Using the rotation amount α and the movement vector Tc, Xv (t + 1) = Rv (t + 1) · Xv (t) + Tv
It is expressed as (t + 1). here,

[Equation 5] Is.

Since the relationship between the camera coordinates and the vehicle coordinates is fixed, if the position coordinates of the camera in the camera coordinate system at time t are Xc (t), then Xc (t) = R.Xv (t) + T Xc (t + 1) = R · Xv (t + 1) + T. Here, R and T are a rotation matrix and a translation vector for converting the vehicle coordinates into the camera coordinates (see FIG. 6).

From the above, the camera movement amount (Rc, Tc)
Is calculated by the following equation.

[Equation 6] As described above, in the process of step S203, the camera movement amount (Rc, Tc) can be measured from the wheel speed and the steering wheel steering angle.

FIG. 10 shows step S205 in FIG.
FIG. 6 is a diagram for explaining in more detail (processing of detecting coordinates of edge corresponding points in camera coordinates) of FIG. Below, Figure 1
The process of step S205 will be described with reference to 0. In FIG. 10A, the edge corresponding point is P. The coordinates of the point P in the absolute coordinate system are Pw (xw, y
w, zw). The absolute coordinate system is a coordinate system fixed on the road surface. The coordinates of the point P in the camera coordinate system at time t are Ps (xs, ys, zs). Time t of point P
The coordinate in the camera coordinate system at +1 is Pr = (xr, y
r, zr).

Now, between the absolute coordinate system and the camera coordinate system, 3 × 3 rotation matrices Rws, Rwr and three-dimensional translation vectors Tws, Twr representing the translation amount of the origin coordinates.
It can be expressed as follows using and. Pr = Rwr · Pw + Twr (7) Ps = Rws · Pw + Tws (8)

FIG. 10B is a diagram showing the relationship between the edge corresponding point P on the obstacle derived from the focal length of the lens of the camera 10 and the image data. In FIG. 10B, it is assumed that the focal length is fr and the coordinate of the corresponding point P on the image data at time t is (xr ′, yr ′).
Similarly, the coordinates of the corresponding point P on the image data at time (t + 1) are (xs ', ys'), and the focal length is fs.
The following relational expression holds from the similarity of triangles. xr '/ fr = xr / zr, yr' / fr = yr / zr (9) xs '/ fs = xs / zs, ys' / fs = ys / zs (10)

From the equations (7) and (8), the relative relational expressions of the cameras at the times t and t + 1 can be obtained as follows. Ps = Rs · Pr + Ts (11) where R = Rws · inv (Rwr) and Ts = Tws
−M · Twr, M = Rws (inv means inverse matrix). In addition, Rs, Ts

[Equation 7] To represent.

From equations (9), (10) and (11), x
By deleting r, yr, zs, and ys, the following relationship is obtained.

[Equation 8]

By solving the simultaneous equations for zr and zs from the equation (12), zr can be obtained,

[Equation 9] Becomes

From the above zr and equation (9), the corresponding point P
It is possible to obtain the coordinate Pr in the camera coordinate system at the time (t + 1) of

[Equation 10] Becomes

Similarly, zs can be obtained and the coordinate Ps of the corresponding point p at the camera coordinate at time t can be obtained. In this way, the coordinates (xc, yc, zc) of the corresponding point P in the camera coordinate system can be obtained from the camera movement amount (Rc, Tc).

FIG. 11 is a flowchart for explaining the surface area extraction processing (step S103) in FIG. 7 in more detail. The surface area extraction processing will be described below with reference to FIG. First, the drawing apparatus 100 removes an edge shorter than a predetermined length from the edges of the extracted image data (see step S201 in FIG. 8) (step S301).

Next, the drawing apparatus 100 calculates the coordinate values when the remaining edge corresponding points are projected on the road surface based on the coordinates in the camera coordinate system regarding the corresponding points of the remaining edges, and the Hough conversion is performed. An equation of a straight line that approximates the edge corresponding point projected on the road surface by is calculated (step S
302). The Hough transform will be described in detail later. Next, the drawing apparatus 100 obtains a plane that passes through the edge and is perpendicular to the road surface based on the equation of the straight line obtained by the Hough transform (step S30).
3). The operation of step S303 will be described in detail later.

Next, the drawing apparatus 100 determines the area of the plane perpendicular to the road surface obtained previously so as to completely cover the obstacle (step S304). Step S304
The operation of will be described later in detail. Thereafter, the drawing apparatus 100 removes unnecessary planes to easily represent the planes, integrates the plane areas, determines a final plane area (step S305), and ends the operation. Step S
The operation of 305 will be described in detail later.

Next, the Hough conversion will be described in detail. The Hough transform is a method of converting a binary image represented by the presence or absence of coordinate points into an equation such as a straight line that can represent the figure drawn in the binary image with parameters. is there. For example, in the case of obtaining a straight line equation from a figure drawn on a binary image, first, all straight line equations Y = mX + c passing through one point of the binary image on the XY plane are converted into straight lines on the mc plane. Do this for all binary image points. Then, on the mc plane, there appear points at which the straight lines intersect many times.
The mc plane coordinate (m, c) of the point is the inclination m and the intercept c when the figure drawn in the binary image is represented by the equation of the straight line on the XY plane. Although the slope m and the intercept c are used as the parameters of the straight line equation, the parameters (ρ, θ) used in the straight line equation Xcos θ + Y sin θ = ρ may be used separately.

FIG. 12 shows step S30 in FIG.
6 is a flowchart illustrating in more detail the processing by Hough transformation of No. 2. Hereinafter, the processing by the Hough conversion in step S302 will be described with reference to FIG. First, the drawing apparatus 100 calculates the coordinates on the road surface when the edge corresponding points are projected on the road surface based on the coordinate values of the edge corresponding points in the camera coordinate system (step S).
401). Next, the drawing apparatus 100 performs Hough transform on the edge corresponding points projected on the road surface (step S40).
2). After that, the drawing apparatus 100 uses the parameters obtained by the Hough transform in the vehicle coordinate system to
A straight line equation that approximates an edge is derived (step S4
02), the processing is ended.

FIG. 13 is a diagram for specifically explaining the process of step S302. FIG. 13A is a diagram in which the edge corresponding points projected on the road surface are plotted. In FIG. 13A, the origin is the vehicle coordinate system (Xv, Y
v, Zv) is the origin on the Xv-Yv plane. The horizontal axis is
The Xv-Yv plane represents the Xv coordinate value, the vertical axis represents the Yv coordinate value, and the unit is mm.

FIG. 13 (b) is a plot of the result of Hough transform applied to the data shown in FIG. 13 (a). A straight line equation Xcos θ + Y sin θ = ρ is used for the Hough transform here. Figure 1
3 (b), the horizontal axis represents ρ (mm) and the vertical axis is θ.
Indicates (degree). As shown in FIG. 13B, (ρ,
The plot results of θ) are concentrated at two points A and B. That is, the coordinate values (ρ, θ) of the points A and B become the parameters of the straight line obtained by the Hough transform.

FIG. 13C is a diagram in which the equation of the straight line obtained by the above Hough transformation is drawn on the graph of FIG. 13A. As shown in FIG. 13C, it can be seen that the obtained straight line approximates the set of edge data projected on the road surface with a straight line.

FIG. 14 shows the step S30 in FIG.
It is a flow chart which explained processing of 3 in more detail.
The process of step S303 will be described below with reference to FIG. First, the drawing apparatus 100 converts the direction vector of the straight line in the vehicle coordinate system derived in the process of step S302 into the direction vector of the straight line Lc in the camera coordinate system (step S501). Specifically, the straight line Lv in the vehicle coordinate system derived in the process of step S302
X = Xi + t · av (Xi = (xv, yv, zv) is one point on the straight line Lv. Av is the direction vector of the straight line Lv. T is
Parameter), the direction vector ac of the straight line Lc in the camera coordinate system is calculated as ac = R · av + T.

Next, the drawing device 100 calculates the normal vector of the road surface in the camera coordinate system (step S5).
02). Specifically, the normal vector Hv of the road surface in the vehicle coordinate system is Hv = (0,0,1), so if Hv is converted using the rotation vector R, the camera coordinate system The normal vector Hc of the road surface at is calculated. That is, the road surface normal vector Hc in the camera coordinate system is Hc = R (0,0,1). Hereinafter, Hc will be expressed as Hc = (λc, μc, ηc).

Next, the drawing apparatus 100 calculates the outer product Hk of the direction vector of the straight line in the camera coordinate system and the normal vector of the road surface (step S503). Outer product Hk = H
Let c × ac be expressed as Hk = (λk, μk, ηk).

Next, the drawing apparatus 100 returns to step S50.
The inner product between the normal vectors obtained in 3 is calculated (step S504). Specifically, the normal vector obtained in step S503 is Hk1 = (λk1, μk1, ηk
1) and Hk2 = (λk2, μk2, ηk2),
The inner product is (Hk1 · Hk2) = cos θ = λk1 · λk2 + μk
It is calculated by 1 · μk2 + ηk1 · ηk2.

Next, the drawing apparatus 100 returns to step S50.
The distance between the straight lines in the camera coordinate system corresponding to the normal vector whose inner product is calculated in 5 is calculated (step S505).

Next, the drawing apparatus 100 returns to step S50.
The angle θ corresponding to the inner product obtained in 5 is a predetermined threshold value ε
(For example, 5 deg) and step S5
It is determined whether or not the distance between the straight lines calculated in 06 is less than a predetermined threshold value D (for example, 10 cm) (step S506).

If the determination in step S506 is affirmative, the drawing apparatus 100 proceeds to step S507, determines that the plane corresponding to the normal vector is the same plane, and proceeds to step S508. Step S
In the operation of 508, the drawing apparatus 100 searches for the closest corresponding point from the camera position among the corresponding points of the edges on the same plane (step S508). Next, the drawing apparatus derives a plane that passes through a straight line approximating the edge including the corresponding point searched in step S508 and is perpendicular to the road surface to determine the plane (step S509), and ends the processing. On the other hand, if the determination in step S506 is negative, the rendering apparatus 100 proceeds to operation in step S510, derives a plane that passes through a straight line that approximates the edge and is perpendicular to the road surface, and determines the plane. To finish.

FIG. 15 shows step S30 in FIG.
4 is a flowchart illustrating in more detail the plane area determination processing of No. 4; Hereinafter, referring to FIG. 15, step S
The plane area determination process of 304 will be described. First, the drawing apparatus 100 calculates the direction vector of the line of intersection between the plane passing through the straight line that approximates the edge calculated in step S303 and the road surface (step S601). Specifically, the normal vector of the plane in the camera coordinate system is Na = (nc1, nc
2, nc3) and the normal vector of the road surface is Nc = (nc
1, nc2, nc3), the direction vector of the line of intersection can be obtained by the outer product Aac = Na × Nc of Na and Nc.

Next, the drawing apparatus 100 calculates the coordinates of points existing on the intersecting line (step S602). Specifically, the equation of the plane perpendicular to the road surface in the camera coordinate system is as follows: na1 · x + na2 · y + na3 · z = −pa (13) (where, −pa is a straight line that approximates the edge. It is calculated from one point and the normal vector Na), and the equation of the road surface in the camera coordinate system is λc (x-oxc) + μc (y-oyc) + ηc (z-ozc) = 0. (14) (where (oxc, oyc, ozc) = R (0, 0,
0) + T = T. λc, μc, and ηc are calculated from the normal vector (nc1, nc2, nc3). R and T are a rotation matrix and a translation vector for conversion from the vehicle coordinate system to the camera coordinate system. ), Δ
When 12 = na1 · nc2-na2 · nc1 ≠ 0, z = 0 in Equations (13) and (14) is set, and the simultaneous equations are solved to obtain the intersection with the road surface as one point on the intersection. it can. Also, when Δ12≈0, Δ23 = na
When 2 · nc3−na3 · nc2 ≠ 0, x = 0 and solve similarly. Further, when Δ12≈0, Δ31 = na3 · nc
When 1-na1 · nc3 ≠ 0, y = 0 and solve similarly. In this way, the coordinates of the points existing on the intersecting line in the camera coordinate system can be obtained.

Next, the drawing apparatus 100 returns to step S60.
An equation of the line of intersection is derived based on the direction vector of the line of intersection calculated in 1 and one point on the line of intersection calculated in step S602 (step S603). Next, the drawing apparatus 100 searches for the two corresponding points with the longest distance from the corresponding points of the edges corresponding to the straight line that approximates the edge included in the plane perpendicular to the road surface (step S604). Next, the drawing apparatus 100 projects the two corresponding points searched for on the road surface to obtain the coordinates of the projected point in the camera coordinate system (step S605).

Next, the drawing apparatus 100 returns to step S60.
A perpendicular line from the two projected points is drawn on the intersection line derived in 3 to obtain the intersection points with the perpendicular line (step S60).
6). Next, the drawing apparatus 100 finds the section of the line of intersection by regarding the two points of intersection obtained above as two end points representing the section of the line of intersection between the plane and the road surface (step S607). The processing of steps S601 to S607 is performed on the plane perpendicular to all road surfaces.

Next, the drawing apparatus 100 obtains the intersections between the respective intersection lines derived above (step S608). Next, the drawing apparatus calculates the distance between the intersection of the two intersections and the section endpoints of these intersections (the endpoints of the section calculated in step S607) (step S609). Next, the drawing apparatus 100 determines whether or not the distance calculated in step S609 is less than a predetermined value (step S610). If it is less than the predetermined value in the determination of step S610, the drawing apparatus 100 updates the section of the intersection line so that the intersection point of the two intersection lines is one end (step S611), and proceeds to the processing of step S612. On the other hand, when it is equal to or larger than the predetermined value, the drawing apparatus 100 proceeds to the process of step S612 without updating the section of the intersection line. Step S612
In the processing of 1, the drawing apparatus 100 determines the final section of the intersection line and ends the processing. The final section of the intersection line obtained in step S612 will be referred to as [gmin, gmax]. gim and gmax are (x, y, z) coordinates in the camera coordinate system.

FIG. 16 shows step S30 in FIG.
5 is a flowchart illustrating the plane integration process of No. 5 in more detail. Hereinafter, with reference to FIG. 16, step S30
The plane integration processing of No. 5 will be described. To make planes that obscure obstacles simpler, unnecessary planes should be removed and planes should be integrated. When the obstacle is considered to be a vehicle, the surfaces on the same side are separated by at least 1.5 m or more when the side surfaces of the vehicles are planes in parallel parking. Therefore, for example, when the two surfaces are separated by 1.5 m or more, they are recognized as different surfaces. Other than that, if it is almost parallel, it may be integrated into a plane closer to the vehicle. In step S305, a process for realizing the above concept is performed.

First, the drawing apparatus 100 acquires the line of intersection between each plane and the road surface (step S701). Next, the drawing apparatus 100 determines whether or not the two intersection lines are parallel from the direction vector of any two intersection lines (step S).
702). If the two intersection lines are not parallel to each other, the drawing apparatus 100 performs step S707 without performing plane integration.
Proceed to operation.

On the other hand, when the two intersecting lines are parallel, the drawing apparatus 100 calculates the distance between the two intersecting lines (step S703). Next, the drawing apparatus 100 determines whether the calculated distance is less than or equal to a predetermined value (for example, 1.5 m) (step S704). If it is not less than the specified value,
The drawing apparatus 100 proceeds to the operation of step S707 without integrating the planes. On the other hand, when it is less than or equal to the predetermined value,
The drawing apparatus 100 determines that the plane including the two intersecting lines is the same plane (step S705), and the step S706.
Proceed to operation. In the operation of step S706, the drawing apparatus 100 selects one of the two planes that are the same,
The plane on the side closer to the camera is integrated as a representative plane of the two planes, and the plane is assumed to be a plane that covers obstacles, and the process proceeds to step S707.

In the operation of step S707, the drawing apparatus 100 finally determines the plane area of the plane which approximates the existing area of the obstacle and covers the obstacle, and ends the processing. The plane area of a plane is a coordinate section [gmi at both ends of the line of intersection between the plane and the road surface
n, gmax] and the plane equation. Here, the both-end coordinate section [gmin, gmax] of the line of intersection between the plane and the road surface can be said to be the base of the plane.

FIG. 17 shows step S104 in FIG.
6 is a flowchart for explaining the mapping table correction process of FIG. FIG. 18 is a diagram for intuitively understanding the mapping table correction process shown in FIG. Hereinafter, the mapping table correction process will be described in detail with reference to FIGS. 17 and 18. First,
The drawing apparatus 100 refers to the standard mapping table 1081 and extracts the surface area (step S103 in FIG. 7).
The coordinates on the composite image corresponding to the points on the bottom section [gmin, gmax] of the plane obtained in step S3 are extracted (step S).
801). A point on the image coordinates corresponding to the section as shown in FIG.

Next, the drawing apparatus 100 is a point on the straight line Lc connecting the point A and the camera position, and the section [gmin,
gmax], and an arbitrary point existing on the opposite side of the point A across the line segment on the image is extracted (step S80).
2). As shown in FIG. 18A, an arbitrary point existing on the opposite side of A is B.

Next, the drawing apparatus 100 performs compression for projecting the point B onto the stereoscopic projection plane Tm from the road surface to a certain height Lmax, which is a plane surface area corresponding to the section [gmin, gmax]. Calculate the rate (step S80)
3). Specifically, as shown in FIG. 18B, hc
Is the height of the camera 10 from the road surface (known), h0 is the height of the virtual camera Kca from the road surface (known), dc is the distance from the camera 10 to the obstacle Ss, d0 is the camera 1
When 0 is the distance (known) between the road surface projection points at the virtual camera Kca position and L is the height of the obstacle Ss, the compression ratio γ
Is a compression ratio γ = {(dc + d0) / dc} × {(hc-L) due to the similarity of the triangle formed by the stereoscopic projection plane Tm and the imaging plane Sm.
It can be calculated by / (h0-L)}. As a result, FIG.
As shown in (a), the point C on the straight line Lc where AC = γ · AB can be determined.

Next, the drawing apparatus 100 refers to the standard mapping table 1801 and corresponds to the point B (i,
The X and Y coordinates on the image data of j) are extracted (step S804). Next, the drawing apparatus 100 obtains the image of the point B obtained in step S804 as the X coordinate and the Y coordinate of (i, j ^) on the modified mapping table 1802 corresponding to the point C obtained by the conversion with the compression rate γ. X coordinate on the data,
The Y coordinate is set (step S805).

Next, the drawing apparatus 100 is on the straight line connecting the camera position and the point A on the section, is on the opposite side of the camera position across the section, and is not represented as the point C. Modified mapping table 18 for points on straight line Lc
It is assumed that (i ^, j ^) on 02 has no data (step S806).

Next, the drawing apparatus 100 returns to step S80.
5 and (i ^, j ^) obtained in S806 (i
^, J ^) corresponding standard mapping table 1801
The modified mapping table 1802 is created by copying (i, j) of (step S807), and the process ends.

As described above, the drawing apparatus 100 according to the first embodiment defines the plane on which the image of the detected obstacle is projected,
Since the projection surface of the image of the obstacle can be a three-dimensional projection surface that matches the actual obstacle by making the surface areas that approximate the existence area of the obstacle, the distortion of the composite image can be reduced. It becomes possible to reduce.

Further, since the drawing apparatus 100 creates the modified mapping table 1802 and converts the image of the obstacle into the three-dimensional projection plane, it is not necessary to perform enormous calculation and the distortion of the composite image can be reduced. Is possible.

The drawing apparatus 10 according to the first embodiment.
In the surface model extraction unit 105 of 0, in addition to the surface area of the plane perpendicular to the road surface that approximates the area where the three-dimensional obstacle exists, the area of the plane that approximates the shape of the three-dimensional obstacle (vertical relationship with the road surface Planes that are not necessarily present in (1) may be extracted. For example, the steric obstacle may be approximated as a very simple polyhedron. In this case, the drawing device can create a more realistic composite image by projecting the captured image on each surface of the polyhedron.

The drawing apparatus 10 according to the first embodiment.
At 0, the amount of movement of the vehicle (may be the amount of movement of the camera)
A predicting unit that predicts future coordinate values of the coordinate data of the obstacle detected by the obstacle detecting unit 104 may be provided based on the above. By providing such a prediction unit, it was possible to measure the edge corresponding point of the obstacle last time and obtain the coordinate data of the obstacle, but this time, due to the convenience of the image data, the edge corresponding point is firmly measured. In the case where the coordinate data of the obstacle cannot be sufficiently obtained because the distance could not be obtained, the coordinate data of the current obstacle can be predicted from the coordinate data of the previous obstacle. As a result, even if sufficient coordinate data cannot be obtained, the surface model can be created. The predicted value of the coordinate data represents the amount of movement of the vehicle by a rotation matrix and translation vector,
It can be obtained by performing coordinate transformation.

(Second Embodiment) FIG. 19 is a block diagram showing the functional arrangement of a drawing apparatus 200 according to the second embodiment. In FIG. 19, parts having the same functions as those of the drawing apparatus 100 according to the first embodiment are designated by the same reference numerals, and description thereof will be omitted. In FIG. 19, the drawing device 200 according to the second embodiment includes an imaging unit 101, a vehicle speed sensor 102, and a steering angle sensor 103.
An obstacle detection unit 104, a surface model extraction unit 105,
Mapping table correction unit 108 and image composition unit 206
A display unit 207, a CG data creation unit 208, and a vehicle model data unit 209.

The CG data creating section 208 refers to the modified mapping table 1082, extracts the coordinates (i, j) of the composite image treated as having no image data, and uses the coordinates (i, j) as the coordinates. Create a CG image to paste in the represented area. The CG data creation unit 208
As the CG image, an image of the vehicle viewed from above, an image of asphalt or concrete representing a road surface, or the like is created.

The host vehicle model data section 209 stores data of an image of the host vehicle viewed from directly above (hereinafter referred to as host vehicle model diagram).

The image synthesizing unit 206 receives the image data from the image pickup unit 101, and then the correction mapping table 10
Based on 82, the image data is processed by associating the pixels of the image data with the composite image. The image synthesizing unit 206 pastes the CG image created by the CG data creating unit 208 into an area treated as having no data in the modified mapping table 1082 to create a composite image. The image combining unit 206 sends the created combined image to the display unit 207.

When the composite image is sent from the image composition unit 206, the display unit 207 displays the own vehicle model data unit 209.
Paste the image of the vehicle viewed from directly above stored in the composite image, create and display the final composite image,
Present to the user.

FIG. 20 is a diagram showing an example of a composite image finally created by the drawing apparatus 200 according to the second embodiment. As shown in FIG. 20, a CG image of the road surface is displayed in a region treated as a pure black in the first embodiment (see FIG. 5C, a region treated as having no data in the correction mapping table 1082). A CG image Ucg of the Rod and the vehicle viewed from above is attached.

FIG. 21 is a flow chart showing the operation of the drawing apparatus 200 according to the second embodiment. Below, FIG.
The operation of the drawing apparatus 200 will be described with reference to FIG. In FIG. 21, steps similar to those of the drawing apparatus 100 according to the first embodiment are designated by the same step numbers to simplify the description.

First, the drawing apparatus 200 takes an image of the surroundings of the own vehicle by the image pickup unit 101 and acquires image data (step S101). Next, the drawing apparatus 200 detects the coordinate data in the camera coordinate system of the points forming the obstacle imaged in the acquired image data (step S10).
2). Next, the drawing apparatus 200 approximates the existing area of the obstacle from the acquired coordinate data of the obstacle, and extracts a planar surface area that completely covers the obstacle (step S103). Next, the drawing apparatus 200 creates a modified mapping table 1082 based on the extracted surface area,
Proceed to the operation of step S906.

In the operation of step S906, the drawing apparatus 200 creates CG data corresponding to the area treated as having no data in the modified mapping table. Next, the drawing apparatus 200 and the image data acquired in step S101 and the C created in step S906.
A composite image is created by superimposing it on the G data (step S907). Then, the drawing apparatus 200 further superimposes the own vehicle model diagram on the composite image to create and display a final composite image (step S908), and ends the process.

As described above, the drawing apparatus 200 according to the second embodiment provides a neat and easy-to-see image by pasting the CG data in the area treated by the modified mapping table 1802 as the absence of image pickup data. It becomes possible to do.

(Third Embodiment) FIG. 22 is a block diagram showing the functional arrangement of a drawing apparatus 300 according to the third embodiment. In FIG. 22, the drawing apparatus 100 according to the first embodiment and the drawing apparatus 20 according to the second embodiment.
Portions having the same function as 0 are designated by the same reference numerals, and description thereof will be omitted. 22, the drawing device 300 includes an image pickup unit 101 and a vehicle speed sensor 102.
A steering angle sensor 103, an obstacle detection unit 104, an image composition unit 306, a display unit 307, and an area extraction unit 308.
, Own vehicle model data unit 209, and CG image creation unit 3
10 and a standard mapping table unit 3081.

The area extracting section 308 is provided with the obstacle detecting section 104.
The coordinate data of the points that form the obstacle detected by is projected on the road surface, and the projection area on the road surface of the obstacle when the obstacle is viewed from directly above is extracted.

The CG image creating section 310 includes the area extracting section 30.
A CG image of an auxiliary line indicating the projection area on the road surface of the obstacle extracted by 8 is created. The CG image creating unit 310 transmits the created CG image to the image synthesizing unit 306 according to the instruction from the image synthesizing unit 306. The CG image creating unit 31
For 0, a CG image of hiding (masking) the projection area of the obstacle may be created in addition to the CG image of the auxiliary line indicating the projection area.

The standard mapping table section 3081 stores a standard mapping table. The standard mapping table is the same as that of the first embodiment, and therefore its explanation is omitted.

The image synthesizing unit 306 receives the image data from the image pickup unit 101, and then the standard mapping table unit 3
With reference to the standard mapping table 081, the image data is processed into an image viewed from above the virtual vehicle position (virtual camera position). The image composition unit 306 attaches the CG image created by the G image creation unit 310 to the image processed based on the C standard mapping table to create a composite image. The image composition unit 306 sends the composite image to which the CG image is attached to the display unit 307.

The display unit 307 affixes the own vehicle model diagram stored in the own vehicle model data unit 209 to the combined image sent from the image combining unit 306, creates a final combined image, and displays it to the user. Present.

FIG. 23 is a diagram for explaining a composite image displayed by the drawing device 300 according to the third embodiment.
FIG. 23A shows image data processed by the standard mapping table. As explained in the prior art,
In FIG. 23A, the upper part of the vehicle is distorted so much that it is difficult to recognize the obstacle. FIG. 23
(B) is an image displayed as a final composite image by the drawing device 300. In FIG. 23B, the auxiliary line Hli indicating the area where the obstacle exists is attached.
This allows the user to clearly recognize the area where the obstacle exists.

FIG. 24 is a view showing another example of the composite image displayed by the drawing device 300. In FIG.
In addition to the auxiliary line Hli shown in (b), a CG image H that covers an image of an obstacle that is greatly distorted in FIG. 23 (a).
su is attached. This allows the user to further clearly recognize the area where the obstacle exists.

FIG. 25 is a flow chart showing the operation of the drawing apparatus 300. The operation of the drawing apparatus 300 will be described below with reference to FIG. In FIG. 25, steps of the same operations as those of the drawing apparatus 100 according to the first embodiment are denoted by the same step numbers, and the description will be simplified.

First, the drawing apparatus 300 uses the image pickup unit 101 to pick up an image of the surrounding area of the vehicle and acquire image data (step S101). Next, the drawing apparatus 300 combines the acquired image data with an image viewed from above the own vehicle based on the standard mapping table (step S100).
1). Next, the drawing apparatus 300 detects the coordinate data in the camera coordinate system of the points forming the obstacle imaged in the acquired image data (step S102).

Next, the drawing apparatus 300 extracts the existing area when the obstacle is projected on the road surface based on the coordinate data forming the obstacle (step S1003). Step S
The processing of 1003 will be described in detail later. Next, the drawing apparatus 300 creates a CG image (a CG image of an auxiliary line or a CG image for masking) for more clearly showing the existing area extracted previously (step S10).
04).

Thereafter, the drawing apparatus 300 proceeds to step S1.
The synthesized image of the periphery created in 001 and step S100
The CG image created in 4 and the model diagram of the vehicle are overlaid to create and display a final composite image (step S1005).

FIG. 26 is a flow chart for explaining the obstacle existing area extraction processing (step S1003) of FIG. 25 in more detail. Hereinafter, the obstacle existing area extraction processing will be described with reference to FIG. First, the drawing device 3
00 projects the coordinate data of the points forming the obstacle on the road surface, and calculates the coordinate values projected on the road surface (step S1201). Next, the drawing apparatus 300 extracts the shortest projection point in each radial direction from the position where the camera position is projected on the road surface (step S1202).

Next, when the density of the extracted shortest projection point neighborhood is a certain value or more, the drawing apparatus 300 extracts the projection point as an area extraction point forming an obstacle area (step). S1203). Drawing device 300
Performs the operations of steps S1202 and S1203 for all projection points.

Thereafter, the drawing apparatus 300 proceeds to step S1.
When the distance between the area extraction points extracted as the points forming the area in 203 is equal to or less than a certain value, the area extraction points are interpolated with a straight line to determine the existence area of the obstacle on the projection surface ( Step S1204).

As described above, in the drawing apparatus 300 according to the third embodiment, the standard mapping table is used to display an image showing the position of a significantly distorted obstacle and to perform processing such as hiding the greatly distorted obstacle. It is possible to provide an image that is easy for the user to see.

The drawing device 3 according to the third embodiment.
In 00 as well, as in the first embodiment, a prediction unit that predicts coordinate data of an obstacle from the moving amount of the vehicle may be provided. In this way, the area of the obstacle projected by the area extraction unit 308 on the road surface can be predicted from the prediction of the coordinate data of the obstacle. As a result, the drawing apparatus 300 can extract the area of the obstacle even when the coordinate data of the obstacle cannot be sufficiently obtained from the image data.

(Fourth Embodiment) FIG. 27 is a block diagram showing the functional arrangement of a drawing apparatus 400 according to the fourth embodiment. In FIG. 27, the drawing apparatus 100 according to the first embodiment and the drawing apparatus 20 according to the second embodiment.
The components having the same functions as those of the drawing device 300 according to the third and third embodiments are denoted by the same reference numerals, and the description thereof will be omitted.

In FIG. 27, the drawing device 400 includes an image pickup section 101, a vehicle speed sensor 102, and a steering angle sensor 103.
An obstacle detecting unit 104, a region extracting unit 308, a vehicle region extracting unit 401, a vehicle type determining unit 402, a model size changing unit 403, and a vehicle model database unit 404.
An image combining unit 406, a display unit 407, a vehicle model data unit 209, and a standard mapping table unit 308.
1 and 1.

The image synthesizing unit 406 receives the image data from the image pickup unit 101, and then the standard mapping table unit 3
By referring to the standard mapping table stored in 081, the image data is processed into an image viewed from above the own vehicle (virtual camera position). The image combining unit 406 gives the created combined image to the display unit 407.

The vehicle area extraction unit 401 determines whether or not a vehicle exists in the image data given from the image pickup unit 101, and extracts an area showing the appearance of the vehicle. The vehicle area extraction unit 401 performs edge extraction using a Sobel filter or the like, and extracts the appearance area of the obstacle in the height direction and the lateral direction based on the peak value of the frequency distribution of the horizontal edges and the vertical edges to determine the appearance of the vehicle. The indicated area is extracted. Also,
The vehicle area extraction unit 401 may extract the license plate, tires, bumpers, and the like to obtain the appearance area of the vehicle. Many other technologies have been published at present regarding the method of extracting the exterior region of a vehicle. The vehicle area extraction unit 401 provides the extracted vehicle appearance area to the vehicle type determination unit 402.

The vehicle model database section 404 stores image data (vehicle model diagram) obtained by photographing various types of vehicles from several patterns of distances and directions. The vehicle model diagram stored in the vehicle model database unit 404 has a color and a texture that does not cause a feeling of discomfort with the own vehicle model diagram stored in the own vehicle model data unit 209.

The vehicle type determination unit 402 determines whether or not there is a vehicle model that matches the appearance region of the vehicle given from the vehicle region extraction unit 401 by referring to the vehicle model database unit 404, and the vehicle type of the vehicle. To judge. The vehicle type determination unit 402 uses the determined vehicle type as the model size change unit 40.
Tell 3.

The model size changing unit 403 matches the size of the obstacle area given from the area extracting unit 308 with the vehicle model diagram of the vehicle type given by the vehicle type determining unit 402 (when viewed from above). Change the size of the vehicle model diagram). The model size changing unit 403 creates a CG image that covers (masks) the area of the obstacle given from the area extracting unit 308, and creates a CG image in which the resized vehicle model diagram is attached to the CG image. To do. When the obstacle area given from the area extracting unit 308 is not the vehicle existing region, the model size changing unit 403
Creates only a CG image that masks the area of the obstacle. The model size changing unit 403 creates the C
The G image is given to the display unit 407.

The display unit 407 attaches the own vehicle model diagram of the own vehicle model data unit 209 and the CG image provided from the model size changing unit 403 to the combined image provided from the image combining unit 406, and finally combines the images. Create and display an image.

FIG. 28 is a diagram showing an example of a composite image created by the drawing apparatus 400. As illustrated in FIG. 28, the drawing device 400 includes a CG image Su and a vehicle model diagram Smd indicating the vehicle area created by the model size changing unit 403.
And the own vehicle model diagram Jsh are pasted on an image combined based on the standard mapping table 1801 to create a final combined image.

FIG. 29 is a flow chart showing the operation of the drawing apparatus 400. The operation of the drawing apparatus 400 will be described below with reference to FIG. In FIG. 29, the drawing apparatus 100 and the third device according to the first embodiment are described.
The same step numbers are attached to the steps of the same operation as that of the drawing apparatus 300 in the above embodiment to simplify the description. First, the drawing device 400 is in the drawing mode ON.
It is determined whether or not (step S1101). Here, the drawing apparatus 400 turns on the drawing mode when the driver operates the switch or when the back gear is entered.
Suppose

Until the drawing mode is turned on, the drawing device 4
00 waits for processing. On the other hand, when the drawing mode is turned on, the drawing apparatus 400 proceeds to the operation of step S101 and acquires image data by the imaging unit 101.
Next, the drawing apparatus 400 synthesizes the acquired image data with an image viewed from above the own vehicle (virtual camera position) based on the standard mapping table (step S100).
1).

Next, the drawing device 400 detects the coordinate data of the points forming the three-dimensional obstacle (step S10).
2). Next, the drawing apparatus 400 determines whether or not the detected obstacle is a vehicle (step S1105). For example, the determination is made based on whether or not the detected edge has the characteristics of the vehicle. When it is determined that the vehicle is a vehicle, the drawing device 400 determines the vehicle area by snake or the like, and refers to the vehicle model database unit 404 to obtain the data of the vehicle model viewed from various directions and positions of various vehicle types. The degree of coincidence with the vehicle data of the extracted actual image is calculated by a neural network or the like to determine the vehicle type (step S1106). Next, the drawing device 400 determines the size and orientation of the vehicle model whose vehicle type has been determined based on the positional relationship with the host vehicle, and creates a CG image in which an image for masking the vehicle region and the vehicle model diagram are superimposed. Then (step S1108), the operation proceeds to step S1109.

On the other hand, in step S1105,
When it is determined that the vehicle area is not the vehicle, the drawing apparatus 400 creates an image that masks the vehicle area (step S110).
7) and proceeds to the operation of step S1109.

In the operation of step S1109, the drawing apparatus 400 determines the composite image obtained in step S1001 and the CG image obtained by superimposing the vehicle model on the CG image masking the vehicle region obtained in step S1108, or in step S1107. The final CG image is created and displayed by superimposing the CG image for masking the obstacle and further superimposing the own vehicle model diagram.

Thereafter, the drawing apparatus 400 determines whether the drawing mode is OFF (step S1110),
If it is not OFF, the operation returns to step S101,
The above operation is repeated. On the other hand, when it is OFF, the drawing apparatus 400 ends the operation. It should be noted that, for example, when the shift enters from the back to the drive, turning back at the time of parking may be considered. Therefore, the drawing mode may be continued for a certain period of time.

As described above, when the obstacle is a vehicle, the drawing apparatus 400 according to the fifth embodiment pastes a CG image of a vehicle having the same color and texture as the own vehicle as an image viewed from above the own vehicle. Since it is attached, it is possible to create a peripheral image having no discomfort.

The vehicle model database unit 404 may store CG images other than the vehicle (for example, CG images of trees, walls, motorcycles, etc.) so that areas other than the vehicle can be determined. By doing so, it becomes possible to create a more realistic peripheral image.

[Brief description of drawings]

FIG. 1 is a block diagram showing a functional configuration of a drawing device according to a first embodiment of the present invention.

FIG. 2 is a diagram showing an example of mounting positions of two cameras 10 (# 1) and 10 (# 2) in a vehicle.

FIG. 3 is a diagram showing an example of a standard mapping table 1801.

FIG. 4 is a diagram for intuitively understanding a standard mapping table 1081 and a modified mapping table 1802.

FIG. 5 is a diagram for intuitively understanding how the captured image is processed until the rendering device 100 according to the first embodiment creates a composite image.

FIG. 6 is a diagram for explaining a camera coordinate system and a vehicle coordinate system.

FIG. 7 is a flowchart showing an operation of the drawing device 100 according to the first embodiment.

FIG. 8 is a flowchart showing in more detail the coordinate data detection process (step S102) of the obstacle in FIG.

FIG. 9 is a diagram for explaining in more detail the processing of step S203 in FIG. 8 (camera movement amount (Rc, Tc) measurement processing).

10 is a diagram for explaining in more detail the process of step S205 in FIG. 8 (the process of detecting the coordinates of edge corresponding points in camera coordinates).

11 is a diagram illustrating a surface area extraction process (step S1 in FIG. 7).
03) is a flow chart explaining in more detail.

12 is a Hough of step S302 in FIG.
It is a flowchart explaining the process by h conversion in more detail.

FIG. 13 is a diagram for specifically explaining the process of step S302.

FIG. 14 is a flowchart illustrating the process of step S303 in FIG. 11 in more detail.

15 is a flowchart illustrating in more detail the plane area determination process of step S304 in FIG.

FIG. 16 is a flowchart illustrating in more detail the plane integration process of step S305 in FIG.

FIG. 17 is a flowchart for explaining the mapping table correction processing in step S104 in FIG. 7 in more detail.

FIG. 18 is a diagram for intuitively understanding the mapping table correction process shown in FIG. 17.

FIG. 19 is a block diagram showing a functional configuration of a drawing device 200 according to a second embodiment.

FIG. 20 is a diagram showing an example of a composite image finally created by the drawing apparatus 200 according to the second embodiment.

FIG. 21 is a flowchart showing an operation of the drawing device 200 according to the second embodiment.

FIG. 22 is a block diagram showing a functional configuration of a drawing device 300 according to a third embodiment.

FIG. 23 is a diagram for explaining a composite image displayed by the drawing storage unit 300 according to the third embodiment.

FIG. 24 is a diagram showing another example of a composite image displayed by the drawing device 300.

FIG. 25 is a flowchart showing the operation of the drawing device 300.

FIG. 26 is a flowchart illustrating the obstacle existing area extraction processing (step S1003) in FIG. 25 in more detail.

FIG. 27 is a block diagram showing a functional configuration of a drawing device 400 according to a fourth embodiment.

28 is a diagram showing an example of a composite image created by the drawing device 400. FIG.

29 is a flowchart showing the operation of the drawing apparatus 400. FIG.

FIG. 30 is a diagram for explaining a problem of the conventional vehicle monitoring device.

[Explanation of symbols]

100, 200, 300, 400 drawing device 101 Imaging unit 102 vehicle speed sensor 103 Steering angle sensor 104 Obstacle detector 105 face model extraction unit 106, 206, 306, 406 Image composition section 107, 207, 307, 407 Display unit 108 Mapping table correction section 1081 Standard mapping table 1082 modified mapping table 10 cameras 208 CG data creation section 209 Own vehicle model data section 308 area extraction unit 310 CG image creation unit 3081 Standard mapping table section 401 Vehicle area extraction unit 402 Vehicle type determination unit 403 Model size change unit 404 Vehicle model database section

─────────────────────────────────────────────────── ─── Continuation of front page (51) Int.Cl. ⁷ Identification code FI theme code (reference) B60R 21/00 622 B60R 21/00 622F 622K 622L 622Q 622T 624 624C 626 626G 628 628D G06T 1/00 330 / 00 330B G08G 1/16 G08G 1/16 C H04N 7/18 H04N 7/18 JV (72) Inventor Takashi Yoshida 1006 Kadoma, Kadoma-shi, Osaka Prefecture Matsushita Electric Industrial Co., Ltd. (72) Nobuhiko Yasui 1006 Kadoma, Kadoma City, Osaka Prefecture Matsushita Electric Industrial Co., Ltd. (72) Inventor, Masamichi Nakagawa 1006 Kadoma, Kadoma City, Osaka Prefecture F Term (Reference) 5B057 AA16 CA08 CA12 CA16 CB08 CB12 CB16 CD14 CE08 DB02 DB09 DC16 5C054 CC05 EA01 EA05 FA0 0 FC01 FD03 FE09 FE12 FF07 GB16 HA30 5H180 AA01 CC04 LL01 LL02 LL04 LL08 LL17

Claims (20)

[Claims] 1. A drawing device for creating a display image for driving support of a vehicle, comprising: a captured image acquisition means for acquiring an image of the periphery of the vehicle from a camera installed in the vehicle; Based on the image acquired by the acquisition unit, a three-dimensional object detection unit that detects a three-dimensional object existing region around the vehicle, and a surface model extraction that extracts a plane that approximates the three-dimensional object existing region detected by the three-dimensional object detection unit. A drawing device comprising: a device and an image composition unit that projects an image acquired by the captured image acquisition unit onto a plane extracted by the surface model extraction unit to create a combined image to be displayed. 2. A mapping table for projecting an image acquired by the captured image acquisition means onto the plane extracted by the surface model extraction means, and a plane sequentially extracted by the three-dimensional object detection means. The drawing apparatus according to claim 1, further comprising: a mapping table correcting unit that corrects the mapping table based on the mapping table; and an image synthesizing unit that sequentially creates a synthetic image based on the mapping table. 3. The image synthesizing unit C for pasting to a region on a synthetic image in which data is treated as not existing in the mapping table.
The image synthesizing means further includes CG data creating means for creating a G image, wherein the image synthesizing means pastes the CG image created by the CG data creating means onto an area treated as having no data on the synthetic image to synthesize the CG image. The drawing apparatus according to claim 2, wherein an image is created. 4. A vehicle movement amount detection means for detecting the movement amount of the vehicle is further included, wherein the three-dimensional object detection means is based on the movement amount of the vehicle detected by the vehicle movement amount detection means, The present invention is characterized in that a distance to a three-dimensional object imaged in the image acquired by the captured image acquisition means is calculated by comparing the image after the movement, and the existence area of the three-dimensional object around the vehicle is detected. ,
The drawing device according to claim 1. 5. The vehicle movement amount detecting means detects the amount of movement of the vehicle based on measured values of a vehicle speed sensor and a steering angle sensor installed in the vehicle. Drawing device. 6. The plane model extraction further includes a prediction unit that predicts a future existence area of the three-dimensional object detected by the three-dimensional object detection unit based on the amount of movement of the vehicle detected by the vehicle movement amount detection unit. The drawing device according to claim 4, wherein the means extracts a plane based on the existence area of the three-dimensional object predicted by the prediction means. 7. The three-dimensional object detection means detects three-dimensional coordinates of points on an edge forming the contour of the three-dimensional object to detect the existence area of the three-dimensional object, and the plane model extraction means detects the three-dimensional object. The drawing apparatus according to claim 1, wherein a plane that approximates a region where a three-dimensional object exists is extracted based on the three-dimensional coordinates of the points on the edge determined by the means. 8. The image synthesizing unit creates a synthetic image by pasting a model diagram showing the own vehicle.
The drawing device according to claim 1. 9. A drawing device for creating a display image for driving support of a vehicle, comprising: a captured image acquisition means for acquiring a peripheral image of the vehicle from a camera installed in the vehicle; and the captured image acquisition. Based on the image acquired by the means, a three-dimensional object detecting means for detecting a three-dimensional object existing area around the vehicle, and an area extracting means for extracting an area when the three-dimensional object detected by the three-dimensional object detecting means is projected on a road surface. And a CG image showing the existence area of the three-dimensional object projected on the road surface based on the area extracted by the area extracting means C
An image for creating a composite image to be displayed by projecting an image acquired by the captured image acquisition means onto a road surface and superimposing the CG image created by the CG image creation means on the image obtained by projection. A drawing device comprising a combining unit. 10. A vehicle movement amount detection means for detecting a movement amount of the vehicle is further included, wherein the three-dimensional object detection means is based on the movement amount of the vehicle detected by the vehicle movement amount detection means. 10. The drawing apparatus according to claim 9, wherein a distance to a three-dimensional object captured in the image acquired by is calculated to detect a region where the three-dimensional object exists around the vehicle. 11. The vehicle movement amount detection means detects the amount of movement of the vehicle based on measured values of a vehicle speed sensor and a steering angle sensor installed in the vehicle. Drawing device. 12. A predicting means for predicting a three-dimensional object existing area detected by the three-dimensional object detecting means based on a moving amount of the vehicle detected by the vehicle moving amount detecting means, the area extracting means, 11. The drawing apparatus according to claim 10, wherein the area when the three-dimensional object is projected on the road surface is extracted based on the existing area of the three-dimensional object predicted by the prediction unit. 13. A vehicle area extracting unit that extracts an existing area of another vehicle existing around the vehicle based on the image acquired by the captured image acquiring unit, and another vehicle extracted by the vehicle area extracting unit. A vehicle type determining unit that determines the vehicle type of the other vehicle based on the existing region, wherein the CG image creating unit is a CG indicating the existing region of the three-dimensional object
The drawing apparatus according to claim 9, wherein a CG image is created by pasting a model diagram of a vehicle corresponding to the vehicle type determined by the vehicle type determination means on the image. 14. The three-dimensional object detection means detects three-dimensional coordinates of points on an edge forming the contour of the three-dimensional object to detect the existing area of the three-dimensional object, and the area extraction means includes the three-dimensional object detection means. Is extracted based on the three-dimensional coordinates of the points on the edge determined by
The drawing device according to claim 9. 15. A drawing method for creating a display image for driving support of a vehicle, comprising: a captured image acquisition step of acquiring a peripheral image of the vehicle from a camera installed in the vehicle; and the captured image acquisition. A three-dimensional object detection step of detecting a three-dimensional object existing area around the vehicle based on the image acquired in the step, and a surface model extraction step of extracting a plane that approximates the three-dimensional object existing area detected in the three-dimensional object detecting step. And an image combining step of projecting the image acquired in the captured image acquiring step onto the plane extracted in the surface model extracting step to create a combined image to be displayed. 16. A drawing method for creating a display image for driving support of a vehicle, comprising: a captured image acquisition step of acquiring a peripheral image of the vehicle from a camera installed in the vehicle; and the captured image acquisition. Based on the image acquired in the step, a three-dimensional object detection step for detecting a three-dimensional object existing area around the vehicle, and an area extraction step for extracting an area when the three-dimensional object detected in the three-dimensional object detection step is projected on the road And a CG image creating step of creating a CG image showing an existing area of a three-dimensional object projected on a road surface based on the area extracted in the area extracting step, and an image acquired in the captured image acquiring step on the road surface The CG image created in the CG image creating step is superimposed on the image obtained by projecting, and a composite image to be displayed is created. And an image synthesizing step, the drawing method. 17. A recording medium in which a program for creating a display image for driving assistance of a vehicle is recorded, the captured image obtaining step of obtaining a peripheral image of the vehicle from a camera installed in the vehicle. , A three-dimensional object detection step of detecting a three-dimensional object existing area around the vehicle based on the image acquired in the captured image acquisition step, and a plane approximating the three-dimensional object existing area detected in the three-dimensional object detection step And a surface model extracting step, and an image combining step of projecting the image acquired in the captured image acquiring step onto the plane extracted in the surface model extracting step to create a combined image to be displayed. Recording medium. 18. A recording medium in which a program for creating a display image for driving assistance of a vehicle is recorded, the captured image obtaining step of obtaining a peripheral image of the vehicle from a camera installed in the vehicle. A three-dimensional object detection step of detecting a three-dimensional object existing area around the vehicle based on the image acquired in the captured image acquisition step, and an area when the three-dimensional object detected in the three-dimensional object detection step is projected on a road surface. An area extraction step of extracting, a CG image creation step of creating a CG image showing an existing area of a three-dimensional object projected on a road surface based on the area extracted in the area extraction step, and acquired in the captured image acquisition step. The image is projected on the road surface, and the CG image created in the CG image creating step is superimposed and displayed on the image obtained by projection. And an image synthesizing step of generating a composite image to a recording medium on which a program is recorded. 19. A program for creating a display image for supporting driving of a vehicle, the captured image acquiring step of acquiring a peripheral image of the vehicle from a camera installed in the vehicle, and the captured image acquiring step. Based on the image acquired in, the three-dimensional object detection step of detecting a three-dimensional object existing area around the vehicle, a surface model extraction step of extracting a plane that approximates the three-dimensional object existing area detected in the three-dimensional object detection step, An image composition step of projecting the image acquired in the captured image acquisition step onto the plane extracted in the surface model extraction step to create a composite image to be displayed. 20. A program for creating a display image for driving support of a vehicle, a captured image acquisition step of acquiring a peripheral image of the vehicle from a camera installed in the vehicle, and a captured image acquired in the captured image acquisition step. A three-dimensional object detection step of detecting a three-dimensional object existing area around the vehicle based on the image; an area extraction step of extracting an area when the three-dimensional object detected in the three-dimensional object detection step is projected on a road surface; A CG image creating step of creating a CG image showing the existence area of the three-dimensional object projected on the road surface based on the area extracted in the extracting step; An image for creating a composite image to be displayed by superimposing the CG image created in the CG image creating step on the image obtained by And a forming step, program.