JPH06109450A - Plural-visual point three-dimensional image input device - Google Patents

Abstract

PURPOSE:To obtain an optimum image by detecting a dead angle part within a short processing time by the use of a small scale data processing circuit. CONSTITUTION:Image sensing designated areas are provided by imaging devices 40-1-40-5 in the image sensing designated area and variable density images are synthesized at the longest distances of the image sensing designated areas by display density pixel forming devices 50-1-50-5 so that distance detection accuracy (one phase difference) becomes one pixel of a display image or less. In a dead angle dealing device 60, the opposed surface is judged only on the basis of the signal of 3D cameras 30-1, 30-5 at both ends by an opposed surface judging means 61 and a dead angle is detected by a dead angle detection means to specify a dead angle part and a subject surface improper part. A dead angle correspondence selection means 64 allows the signals of intermediate 3D cameras 30--2-30-4 to correspond to the dead angle part and the subject surface improper part not only to reduce the scanning quantity of a screen but also to shorten a processing time.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a three-dimensional image information technology such as computer vision (CV) and computer graphics (CG), which inputs a three-dimensional image and synthesizes it to display the image in a three-dimensional manner. The present invention relates to a multi-viewpoint three-dimensional image input device for removing a blind spot of a subject by using a blind spot detection method.

[0002]

2. Description of the Related Art Conventionally, as a technique in such a field,
For example, there are those described in the following documents. Reference 1; Journal of Television Society, 45 [4] (1991).
P. 446-452 Document 2; Journal of the Television Society, 45 [4] (1991).
P. 453-460 Conventionally, a three-dimensional image input method includes a passive method (passive method) and an active method (active method). The active method irradiates the target with energy (light wave, radio wave, sound wave) that is skillfully controlled to obtain three-dimensional information and has some meaning with respect to its shape pattern, density, spectrum, etc. Refers to the method. On the other hand, with the passive method, even if normal lighting is performed on the target,
Measurement that does not use meaningful energy. In general, active methods are more reliable than passive methods. A typical passive method is a stereo image method, which is shown in FIG.

FIG. 2 shows the conventional 3 described in the above document 2.
It is explanatory drawing of the stereo image method which is one of the three-dimensional image input systems. In this stereo image method, two cameras 1 and 2 which are two-dimensional image input devices are arranged at a predetermined distance from each other,
The difference between the imaging positions of the subject 3 taken by the left and right cameras 1 and 2, that is, the phase difference is used, and the subject 3 is triangulated.
Is a way to measure the distance to.

FIG. 3 is an explanatory diagram of two images, a grayscale image of a signal and a distance image obtained by the stereo image method of FIG. The grayscale image is a color or monochrome image obtained by the cameras 1 and 2 in FIG. The distance image is an image regarding a three-dimensional position, and each pixel in the matrix data has information regarding the depth of the object (subject 3). From such a grayscale image and a range image, a stereoscopic image display is performed by a binocular fusion method using a polarization filter, or a stereoscopic image display is performed by using a lenticular plate. An example of a stereoscopic image display is shown in FIG.

FIG. 4 shows the conventional 3 described in the above document 1.
It is a figure which shows the principle of the multi-lens type lenticular system which is one of the three-dimensional image display systems. The multi-lens lenticular system is a lenticular plate 10 including a plurality of kamaboko-shaped lens plates.
Is used to arrange the left and right images in stripes on the focal plane of each lens plate. A, b, in one lens board
c, ..., the portion of the f, respectively _{a 1, b 1, c 1} , ...,
f ₁ striped multiview image captured from multiple directions of 1
1 is displayed. By the action of the lens plate, the striped multi-view image 11 in each direction enters the left and right eyes 12 and 13 separately, and if the viewpoint is moved, a horizontal stereoscopic image can be viewed.

[0006]

However, the apparatus having the above structure has the following problems. (1) Lenticular plate 1 as a three-dimensional image display system
When 0 is used, a two-dimensional image can be viewed three-dimensionally,
When the observer's line of sight is changed, the appearance of the object is as narrow as 5 to 10 cm in the left-right direction and ± 30 cm in the front-back direction when viewed at a distance of about 5 m as a stereoscopic viewing area. Also,
In the binocular fusion method, there is a problem that the image itself does not change even if the line of sight is changed, only with a stereoscopic representation of a two-dimensional image.

(2) Therefore, in order to solve the above-mentioned problems such as the observation field of view being narrow and the image not changing even if the line of sight is changed, the inventor of the present application previously proposed Japanese Patent Application No. 4-192272. went. In this proposal, at least two three-dimensional image input devices for inputting an image of a subject and outputting two images of a grayscale image and a distance image are arranged separately. Each three-dimensional image input device is composed of, for example, a three-dimensional camera (hereinafter referred to as a 3D camera). Then, two separated 3D cameras input an image of a subject, output signals of a grayscale image and a distance image representing the subject, and display a display image from the grayscale image and the distance image to display a three-dimensional image. It is designed to display images.
However, when two 3D cameras are used, the subjects that can be imaged are limited, and blind spots occur especially in subjects having concave portions. In order to compensate for this, the present inventor has proposed a method of increasing the number of 3D cameras as shown in FIG.

(3) FIG. 5 is a schematic block diagram of a multi-viewpoint three-dimensional image input device, which has been previously proposed by the inventor of the present application, in which the number of 3D cameras is increased from two to five.
The multi-viewpoint three-dimensional image input device includes five 3D cameras 20-1 to 20-5 and is arranged so that their optical axes intersect at an intersection point K. Each 3D camera 20-1 to 2
In 0-5, the grayscale images 21-1v to 21-5v and the distance image 22-1 representing the subject by inputting the image of the subject are input.
r to 22-5r are output. These grayscale images 21-1v to 21-5v and distance images 22-1r to 22-
Based on the data of 5r, each 3D camera 20-1 to 20-
The process of specifying the object for each of the 5 is performed, the optimum image is selected, and the display image in the line-of-sight direction desired by the observer is three-dimensionally displayed on the display device. However, with such a configuration, the grayscale image 2 for each of the 3D cameras 20-1 to 20-5
1-1v to 21-5v and range images 22-1r to 22-5
Since all the data of r are used for subject identification processing and selection of the optimum image, a large amount of data processing is required, and there is a problem that the scale of the processing circuit for processing it increases and the processing time increases. However, it has been difficult to obtain a multi-viewpoint three-dimensional image input device that is technically sufficiently satisfactory. SUMMARY OF THE INVENTION The present invention provides a multi-viewpoint three-dimensional image input device that solves the problem that the conventional technique has, that is, the circuit scale of the data processing circuit increases and the processing time increases.

[0009]

According to the first aspect of the invention, in order to solve the above-mentioned problems, an image of an illuminated subject is input and at least two gray scale images representing the subject are output. In a multi-viewpoint three-dimensional image input device having a plurality of three-dimensional image input devices having a three-dimensional image input device, an image pickup designated region is provided, and the image pickup designated region is located in all fields of view of the respective three-dimensional image input devices. Then, the respective three-dimensional image input devices are arranged so that the intersections of the center lines of the distance width of the imaged designated area are gathered at one point. According to a second aspect of the present invention, in the first aspect, an unnecessary area data discarding unit that discards all the image data outside the imaging designated area three-dimensionally is provided. In a third aspect based on the first aspect, a symbol substituting means for writing a predetermined symbol is provided in a pixel having a maximum distance value or more within a width of the designated image capturing area. According to a fourth aspect, in the first aspect, the pixel density conversion is performed on the output image of the three-dimensional image input device so that the distance detection accuracy becomes narrower than the pixel width of the desired display image at the longest distance of the designated image capturing area. Display density image creating means for synthesizing a grayscale image is provided for each three-dimensional image input device.

In the fifth invention, the display density pixel creating means of the fourth invention has a function of setting a new distance value based on the distribution of the distance value of each pixel when performing the pixel density conversion. There is. In the sixth invention, the display density pixel creation means of the fourth invention is such that, in the grayscale image value after the pixel density conversion,
It has a function of selecting and processing only the grayscale value of the pixel corresponding to the new distance value to obtain a new grayscale image value. In a seventh aspect based on the fourth aspect, the orientation of the subject plane is based on the output of the display density pixel creating means provided on both sides of the plurality of three-dimensional image input devices on the three-dimensional image input device side. Is provided on the side of the three-dimensional image input device other than the three-dimensional image input devices at the both ends, and the facing surface determination means for determining the blind area, the blind spot detection means for detecting the blind spot of the subject based on the determination result of the facing surface determination means. Based on the output of the display density pixel creation means, a blind spot correspondence selection means for selecting and fitting an optimum image for the processing area of the facing surface determination means and the blind spot detection means, and the presence or absence of residual blind spots from the fitting status of the blind spot correspondence selection means. The blind spot coping means including the blind spot presence / absence judging means for judging is connected to the output side of the plurality of display density pixel creating means.

According to an eighth invention, in the fourth invention, 2
In the phase difference detection direction of the two-dimensional image input device, the number of pixels is 2 for a predetermined optical axis interval by the number of pixels of the phase difference determined by the display pixel density synthesized by the display density image creating means.
An offset is provided at the relative position of the three-dimensional image input device. In the ninth invention, the blind spot detecting means of the seventh invention has a function of standardizing the blind spot determination value based on the distance value of the subject. In the tenth invention, the blind spot presence / absence determining means of the seventh invention has a function of changing the number of pixels serving as a criterion for determining the remaining blind spot depending on the three-dimensional position of the subject.

[0012]

According to the first aspect of the invention, as described above, the plural viewpoints 3
Since the three-dimensional image input device is configured, the imaging area (for example, width and distance) of the subject is specified by providing the imaging specification area, and the number of data of the grayscale image and the distance image to be generated is reduced, and thus less data processing is performed. Depending on the amount, the blind spot portion can be easily detected and removed by performing the subject specifying process and selecting the optimum image thereof. According to the second invention, the unnecessary area data discarding unit has a function of reducing the number of data of the grayscale image and the range image. According to the third invention, the symbol substituting means has a function of reducing the number of data of the grayscale image and the range image to be processed.

According to the fourth aspect of the present invention, the display density pixel creating means is configured so that the distance detection accuracy (for example, one phase difference) does not exceed the width of one pixel of the grayscale image at the maximum distance value of the designated image pickup area. That is, a predetermined number of original grayscale images are combined so that the distance detection accuracy (one phase difference) is equal to or less than one pixel of the display image. As a result, the size of one pixel of the grayscale image becomes one pixel of the displayed grayscale image with a size obtained by combining several pixels of one pixel of the two-dimensional image input device. According to the fifth aspect, the display density pixel creation means has a function of facilitating the conversion of the pixel density by setting a new distance value based on the distribution of the distance value of each pixel when performing the pixel density conversion. . According to the sixth aspect, the display density pixel creation means selects and processes only the grayscale value of the pixel corresponding to the new distance value, and generates the grayscale image value after the pixel density conversion as a new grayscale image value. . By selecting the grayscale image corresponding to the distance value when converting the pixel density and setting it as a new grayscale image value, the grayscale image value becomes at least the composite value of the grayscale values of the subject, and the grayscale values of other subjects are changed. It has the function of generating a high-purity gray value that does not get mixed.

According to the seventh aspect of the invention, the facing surface determining means and the blind spot detecting means detect the blind spot and the inclination of the object surface based on the distance images of the three-dimensional image input devices at both ends. The blind spot correspondence selecting means applies the optimum image in the intermediate three-dimensional image input device only to the blind spot region and the region where the inclination of the subject surface is inconvenient. The blind spot presence / absence determining means determines whether there is a residual blind spot. According to the eighth aspect, the offset provided at the relative position of the two-dimensional image input device has a function of enlarging the phase difference detection area of the two-dimensional image input device. According to the ninth aspect, the blind spot detecting means has a function of standardizing the blind spot determination value according to the distance value of the subject and facilitating the detection of the blind spot. According to the tenth invention, the blind spot presence / absence determining means is
The number of pixels serving as a criterion for determining the remaining blind spot is changed depending on the dimensional position to facilitate the determination of the presence or absence of the blind spot. Therefore,
The above problems can be solved.

[0015]

1 is a block diagram showing the configuration of a multi-viewpoint three-dimensional image input device according to an embodiment of the present invention. In this embodiment, five three-dimensional image input devices (for example, 3D camera)
Although the case of using is described, any number may be used as long as it is at least three or more. The multi-viewpoint three-dimensional image input device is
Five 3D image input devices (for example, 3D camera) 30
-1 to 30-5, and their optical axes H30-1 to H3
It is arranged that 0-5 intersect at the intersection K. Each 3D
Each of the cameras 30-1 to 30-5 has a function of inputting an image of an illuminated subject and outputting signals of at least two grayscale images representing the subject. The image creating devices 40-1 to 40-5 are connected to each other.

Image creation devices 40-1 to 40-1 in each designated image pickup area
40-5 is a predetermined grayscale image S44 and a distance image S45 in the imaging designated area based on at least two grayscale images output from each of the 3D cameras 30-1 to 30-5.
And an area-limited corresponding point search means 41. The area-limited corresponding point search means 41 scans only the necessary area (imaging designated area) for the two grayscale images output from each of the 3D cameras 30-1 to 30-5 to search for corresponding points, and then the grayscale image. And a function of outputting a distance image signal. On the output side thereof, a storage means (for example, an LSI) via an unnecessary area data discarding means 42 and a symbol substituting means 43.
A grayscale image memory 44 and a range image memory 45) which are composed of a memory or the like are connected. The unnecessary area data discarding unit 42 has a function of inputting the output image of the area-limited corresponding point searching unit 41 and discarding the data of the unnecessary area (other than the imaging designated area). The symbol substituting means 43 inputs the output image of the unnecessary area data discarding means 42, replaces the image data having a distance value outside the designated area, that is, outside the imaging designated area with a specific symbol, and the grayscale image S44 within the imaging designated area. And the distance image S45 are output and stored in the grayscale image memory 44 and the distance image memory 45, respectively.

Image forming devices 40-1 to 40-1 in the respective image pickup designated areas
At the output side of 40-5, a display density pixel creation device 50-1
.About.50-5 are respectively connected, and the blind spot coping device 60 is further connected to the output side thereof. Each of the display density pixel creation devices 50-1 to 50-5 inputs the grayscale image S44 and the distance image S45 stored in the respective grayscale image memory 44 and the distance image memory 45, converts the grayscale image S44 and the distance image S45 into the pixel density of the display image, and converts the pixels into pixels. Density-converted grayscale image S52 and range image S53
And a pixel density conversion unit 51 for exchanging the pixel density of the display image and the pixel density of the grayscale image S44.
A grayscale image memory 52 for storing the converted grayscale image S52, and a distance image memory 53 for storing the distance image S53 with pixel density conversion.

The blind spot coping device 60 removes the blind spots from the output images of the respective display density pixel creating devices 50-1 to 50-5, outputs the grayscale image and the range image from which the blind spots have been removed, and outputs the blind spots. It has a function of outputting partial data. The blind spot coping device 60 determines the facing surface determining means 61 that determines the orientation of the subject surface from the output distance images S53 of the display density pixel creating devices 50-1 and 50-5 on the 3D cameras 30-1 and 30-5 at both ends. And a blind spot detecting means 62 for detecting a blind spot portion is connected to the output side thereof. A processing area memory 63 for storing the processing areas of the facing surface determining means 61 and the blind spot detecting means 62 is connected to the output side of the blind spot detecting means 62, and the output side thereof has a blind spot corresponding to the blind spot correspondence selecting means 64. Presence determination means 65 is connected. The blind spot correspondence selection means 64 is a processing area memory 63 and each of the display density pixel creation devices 50-1 to 50-5.
The output distance image S53 of
Based on the output distance images S53 of the intermediate display density pixel creation devices 50-2 to 50-4, the minimum image of the phase difference change in the blind spot corresponding region (optimum image) for the processing region stored in
Has a function of selecting and applying. The blind spot presence / absence determining means 65 determines whether or not all the processing areas stored in the processing area memory 63 have been fitted by the blind spot corresponding selecting means 64, that is, whether or not the blind spot corresponding area has been filled, and detects the presence or absence of residual blind spots. It has a function to do.

A blind spot storage memory 70 for storing a blind spot portion S70 is connected to the output side of the blind spot presence / absence determining means 65. On the output side of the grayscale image memory 52, the facing surface determination means 61, and the blind spot detection means 62 in the display density pixel creation device 50-1, the blind spot portion of the first 3D camera 30-1 and the subject unsuitable surface are removed. A grayscale image memory 71 that stores the final grayscale image S71 and a range image memory 72 that stores the range image S72 are connected. On the output side of the blind spot countermeasure device 60, the fifth 3D camera 3 at the other arrangement end is provided.
A grayscale image memory 73 that stores the final grayscale image S73 from which the blind spot portion 0-5 and the subject unsuitable surface have been removed and a distance image memory 74 that stores the distance image S74 are connected. Further, on the output side of the blind spot correspondence selection means 64,
A pixel number S75 such as a 3D camera number (angle of view, installation angle) and an internal address of the image (X direction, Y direction) when the optimum image is stored in the blind spot portion output from the blind spot handling device 60 and the subject inadequate surface are set. A pixel number memory 75 for storing pixel numbers, a grayscale image memory 76 for storing a grayscale image S76 corresponding to the pixel number S75, and a distance image memory 77 for storing the distance image S77 are connected. .

Next, the configuration of the 3D cameras 30-1 to 30-5 in FIG. 1 will be described with reference to FIGS. 6A, 6B, 6C, 7, 8 and 9. explain. FIG. 6 (a)-
(C) is an explanatory view of the 3D camera of FIG. 1, and FIG.
Is a configuration diagram of a 3D camera, FIG. 3B is a diagram showing an example of distance-phase difference characteristics of the 3D camera, and FIG. 3C is an explanatory diagram of a relationship between the angle of view and the number of pixels of the two-dimensional image input device. is there. FIG. 7 is an explanatory diagram of an image capturing designated distance range in the apparatus of FIG. 1, and FIG. 8 is an explanatory diagram of an image capturing designated area in the apparatus of FIG.
As shown in FIG. 6A, the 3D camera shown in FIG. 1 includes two two-dimensional image input devices 31 and 32 each including an area sensor such as a charge-coupled device (hereinafter referred to as a CCD), which are arranged at regular intervals. (Optical axis interval) L _{0 is} spaced apart. When the optical axes H33 and H34 of the lenses 33 and 34 used in the two-dimensional image input devices 31 and 32 are parallel, the distance 1 and the phase difference ΔB are expressed by the following equations (1) and (2).
Are in inverse proportion. The relationship is shown in FIG. 6 (b).

[0021]

[Equation 1] On the other hand, as shown in FIG. 6C, the angle of view δ of one two-dimensional image input device (for example, 31) is determined by the width of a subject at a certain distance and the angle of view δ. The lens 33 that achieves the above is usually selected. In this case, 1
The size viewed from the pixel width w is determined by the following equations (3) and (4). w = 2 ltan (δ / 2) (3) Δw = W / N (4) where δ; angle of view Δw; field of view of 1 pixel N; number of pixels within angle of view δ Therefore, 3D camera Distance accuracy value when configured as
That is, the relationship between the value of one pixel difference in phase difference and the width of one pixel of the subject width (how many pixels are included in the distance change due to one phase difference change) is calculated as (2) and (3). , (4)
From the equation, the following equation (5) is obtained.

[0022]

[Equation 2] The expression (5) means that the distance change value of one phase difference change of a subject at a distance shorter than the distance value corresponding to a certain phase difference ΔB is equal to or smaller than the width of one pixel of the subject width of the allowable coefficient k ₂ times. Is meant to be. That is, the expression (5) means that the number of pixels of the grayscale image having the allowable coefficient k ₂ times is equal to or less than one phase difference change of a certain phase difference ΔB or more. This (5)
Based on the relationship of the equation, the allowable coefficient k _{2 is set} to, for example, 8
And the following explanation is continued. For example, the number of pixels in the horizontal direction (phase difference detection direction) of the two-dimensional image input devices 31 and 32 is 500, the angle of view δ of the lenses 33 and 34 is 50 degrees, and the optical axis spacing of the two-dimensional image input devices 31 and 32 is The distance and the phase difference characteristic when L ₀ is 20 cm are shown in FIG. 7. In this figure, L is the designated imaging range, and ΔB _mim is the desired minimum phase difference. In this example, the tolerance coefficient k _{2 is set} to 8, so
Solving the equation (5), the phase difference ΔB becomes about 67 pixels. Therefore, the distance at which the phase difference ΔB becomes 67 pixels or more (about 2
m) is the designated longest distance for imaging.

On the other hand, as shown in FIG. 1, since it is necessary to place the subject within the angle of view of the five 3D cameras 30-1 to 30-5, the designated imaging shortest distance (about 1.4 m) can be obtained. However,
Optical axes H30- of 3D cameras 30-1 and 30-5 at both ends
1, the installation angle θ of H30-5 is 65 degrees.
The installation angle θ is an angle formed by the points connecting the principal points of the 3D cameras 30-1 and 30-5 at both ends and the optical axes H30-1 and H30-5 at both ends. This pattern is shown in FIG. In FIG. 8, O ₁ , O ₂ , O ₃ , O ₄ , and O ₅ are 3D cameras 3
0-1 to 30-5 and their principal points O _{1 to} O
The optical axes H30-1 to H30-5 of ₅ are installed so as to intersect at an intersection point K. In addition, each optical axis H30-1 to H30
-5 divides the included angle at the intersection K into equal parts,
It eases the subsequent calculation process. However, it does not have to be equally divided. The hatched portion in FIG. 8 is the imaged designation area LS. The distance between the principal points O ₁ and O ₅ is set to 2 m, but the principal points O ₂ , O ₃ , and O ₄ are arranged in a slightly concave shape in order to widen the imaging area. That is, the center O of the designated imaging distance width is gathered at one point in each of the 3D cameras 30-1 to 30-5.

FIG. 9 shows the two-dimensional image input device 3 shown in FIG.
It is explanatory drawing of the offset arrangement | positioning of 1,32. As shown in FIG. 8, with the 3D cameras 30-1 to 30-5, it is only necessary to capture an image within a fixed distance range, and thus a long distance (that is, a low phase difference).
Image is not handled. Therefore, as shown in FIG.
Due to the arrangement relationship between the two-dimensional image input devices 31 and 32 in FIG. _6A , it is also possible to dispose the offset by offsetting it by the desired minimum phase difference ΔB _mim (for example, 67 pixels). If such an offset is provided, the phase difference detection areas of the two-dimensional image input devices 31 and 32 can be expanded.

Next, referring to FIG. 10 to FIG.
The operation of the device will be described. 10 (a) to 10 (c) are image creation devices 40-1 to 40-5 in the respective imaging designated areas of FIG.
It is a figure which shows the processing content of. FIG. 11 is an explanatory diagram of a limited area of the corresponding point search area in each of the image-capturing-designated-area image creation devices 40-1 to 40-5 in FIG. In each of the 3D cameras 30-1 to 30-5 in FIG. 1, the image of the illuminated subject is input and the two grayscale images S30a and S3 shown in FIG. 11 are input.
0b signals are output respectively. The signals of these two gray-scale images S30a and S30b are usually stored in an LSI memory or the like, and the image creation device 40-1 in each imaging designated area
To 40-5. The image-capturing-designated-region image creation devices 40-1 to 40-5 perform the processes shown in FIGS.

FIG. 10A shows two grayscale images S30.
a, S30b, for example, one S30a is represented. Since the imaging region is designated, the phase difference of the other grayscale image S30b exists in the range of 60 to 115 pixels in the example shown in FIG. 11, and is shifted from the region shifted by 65 pixels to one grayscale image S30a. FIG. 10A shows that the range image is detected by performing a corresponding point search from the area at the other end to an area displaced by 115 pixels. This corresponding point search is performed by the area-limited corresponding point search means 41 of FIG. That is, in the area verification corresponding point searching means 41, as shown in FIG. 10A at the time of detecting the phase difference, a necessary area (imaging designated area)
Only the LS is scanned to generate a pair of grayscale image and distance image, which are sent to the unnecessary area data discarding means 42.

In the unnecessary area data discarding means 42, FIG.
As shown in (b), the image of the unnecessary area L42 other than the image pickup designated area LS is removed and sent to the symbol substitution means 43. In addition, the image data S3 of each of the 3D cameras 30-1 to 30-5
If the memory capacity for storing 0a and S30b is set to a required capacity value in advance, the unnecessary area data discarding means 42 can be omitted. In the symbol substituting means 43, as shown in FIG. 10C, in the width range of the imaging designated area LS, as shown in FIG.
As in the hatched portion in (c), the image data of the subject L43 other than the fixed distance range (outside the designated distance) of the image pickup designated area LS is removed, and instead, the symbol outside the distance range is substituted as 0, for example. Grayscale image S44 obtained as a result
And the distance image S45 are stored in the grayscale image memory 44 and the distance image memory 45, respectively. At this stage, the grayscale image memory 44 and the distance image memory 45 store images and symbols within a predetermined distance range within the width of the designated imaging area. The grayscale image S44 and the range image S45 stored in the grayscale image memory 44 and the range image memory 45 are displayed in the respective display density pixel creation devices 50-, as shown in FIGS.
The following processes are performed by 1 to 50-5.

FIG. 12 is an explanatory diagram of the pixel density conversion example of FIG. 1, and FIG. 13 is an explanatory diagram of the pixel density conversion method. In the case of this example, since the tolerance coefficient k ₂ is 8, the pixel density conversion unit 51 first divides the range image S45 into groups of 8 in the horizontal direction H and 8 in the vertical direction V by processing 511. To go. The vertical direction V may be a number other than eight. In the process 512, the distance value indicating the maximum distribution of the distribution of the range image values (r ₁ , r ₂ , ...) In each divided group is set as the distance value r _{new of the} group, and it is sent to the process 513.
In this embodiment, the distance values r ± 1 phase differences are accumulated and counted.

Next, in process 513, the average value of only the grayscale values of the grayscale image corresponding to the pixel having the distance value r _new is taken and set as the grayscale value E _{new of the} group. As a result of the processing over the entire screen, the grayscale image S52 and the distance image S53 are displayed.
Is obtained and stored in the grayscale image memory 52 and the range image memory 53, respectively. Each 3D camera 30-1 to 3
For the output images 0-5, the image-capturing-designated-area image creation devices 40-1 to 40-5 and the display density image creation device 50 are displayed.
-1 to 50-5 are processed in almost the same manner. However, the 3D cameras 30-1 to 30-5 are set in the laterally designated image pickup area LS.
The image creation device 4 within each imaging designated area
Differences are provided in the imaging designated area LS in 0-1 to 40-5. 14 to 20 in the blind spot coping device 60, the grayscale image S52 and the distance image S53 whose pixel densities have been converted by the display density pixel creation devices 50-1 and 50-5 on the 3D cameras 30-1 and 30-5 side at both ends. The processing as shown in is performed.

FIG. 14 shows the processing contents of the facing surface determining means 61. The 3D cameras 30-1, 30 on the surface of the object.
It is explanatory drawing of the inclination of the surface with respect to -5. O _{1 in} FIG.
˜O ₅ are the main points of the 3D cameras 30-1 to 30-5.
For example, the processing content of the facing surface determination means 61 will be described using the principal point O ₅ of the rightmost 3D camera 30-5. Regarding the principal point O ₁ of the leftmost 3D camera 30-1, the basic processing contents are the same except that the positive and negative directions of the inclined surface are reversed. FIG. 15 shows the rightmost 3D camera 30-5.
(O ₅₎ is an explanatory view of the object plane orientation determination method when attention is paid to. 6A and 6B are explanatory diagrams of the angle formed by the object plane in FIG. 1 and the 3D camera installation angle. Opposing surface determination means 6
In 1, for example, pixels are scanned clockwise. in this case,
When the value obtained by dividing the distance change by 1 pixel (that is, the pixel whose pixel density has been changed, and used in the same meaning below) is a certain value or more, the surface viewed by the 3D camera 30-5 is most suitable. The other 3D cameras 30-1 to 30
At -4, the surface becomes steeper. Therefore, for these surfaces, the grayscale image S71 and the range image S72 viewed by the rightmost 3D camera 30-5 (O ₅ ) are stored in the grayscale image memory 71 and the range image memory 72, respectively. In this example, the above-mentioned fixed value means that the three 3D cameras 3 are equiangular.
Since 0-1 to 30-5 are used, the distance change dl / dB per pixel is a value expressed by 1/4 · (π / 2−θ). For example, as shown in FIG. 16, since the angle of view θ is 65 degrees, a surface having an inclination of −6.25 degrees with respect to a line forming a right angle with the optical axis H30-5 has the constant value. Then, the normalized distance change is about 1 / of the pixel width.
Since it corresponds to 9 pixels, it means that the distance change when scanning 9 pixels in the pixel width direction is less than 1 pixel.

The standardized distance change corresponds to the phase difference change having the same dimension as the pixel width at the distance value. For example, the object having a distance showing a phase difference of 67 pixels has 1/9 pixels, but the distance having a phase difference of 115 pixels has about 1/5 pixels. That is, when scanning in the width direction of 5 pixels, one phase difference change corresponds to −6.25 degrees. The judgment criteria are shown in Table 1 below.

[0032]

[Table 1] In Table 1, since both the phase difference and the pixel number are count values, the number of pixels of the calculation result is rounded off so that the quantization error becomes small. Table 1 shows
Depending on the direction of the detected surface, another 3D camera 30-1,
The data processing procedure is prioritized and the data of the 3D camera 30-5 within the criterion is applied. In the facing surface determination means 61,
Handle the subject plane up to 1 phase difference change. A phase difference change larger than one phase difference change is detected by the next blind spot detecting means 6
Processed in 2.

FIG. 17 is an explanatory diagram of a blind spot in the blind spot detecting means 62 of FIG. Even at the shortest distance, the normalized distance is 1.7 phase difference. Therefore, the blind spot detection means 62
In, the phase difference change is defined as a blind spot where there is a change of two phase differences or more between pixels. As shown in FIG. 17, there are two types of blind spots. One is a blind spot possibility area I which is a portion that becomes a shadow of the subject itself in the imaging designated area. The other one is a blind spot possibility area II which is a shadow portion cast on the far end of the imaged designated area LS. In the 3D camera 30-5 (O ₅ ), the blind spot possibility region I corresponds to a region of change equal to or more than the negative two-phase difference change, and is detected in a negligible form when the value is positive. The 3D camera 30-1 (O ₁ ) has the opposite sign. Since the width of the blind spot portion is calculated based on the magnitude of this phase difference change, the blind spot correspondence selecting means 64 applies the data of the other 3D cameras 30-1, ... To fill the blind spot portion. At this time, the 3D camera 30-1 (O
The process is started from ₁ ) and the process is performed up to the 3D camera 30-4, but the process is terminated when the pixels in the blind spots are completely filled.

FIG. 18 is a diagram showing an example of a correction coefficient memory (for example, ROM) used when the blind spot presence / absence determining means 65 of FIG. FIG. 19 is an explanatory diagram of a blind spot and an installation angle of another 3D camera that fills the blind spot. 20 is a diagram showing an example of the angle of view position and the correction coefficient value. The dimensional value of the blind spot portion has a difference in the number of pixels to be filled up depending on the installation angle of each of the 3D cameras 30-1 to 30-5. Therefore, the dimension value of this blind spot is
For example, a correction coefficient ROM as shown in FIG. 18 is created, and determination is made by multiplying the distance value 1 by a coefficient determined by the image position X.

An example of how to determine such a correction coefficient will be described with reference to FIGS. 19 and 20. The correction coefficient becomes maximum when the line of sight of the 3D cameras 30-1 to 30-5 is viewed from a direction perpendicular to the direction detected as a blind spot, and the value is set to 1. FIG. 20 shows the right-angle line of sight within the effective pixel of the 3D camera 30-5 (O ₅ ) (for example, from −25 degrees to
The coefficient value for +5 degrees is shown. The angle-of-view position corresponds to the imaging position (pixel number) of the two-dimensional image input device forming each of the 3D cameras 30-1 to 30-5. for that reason,
The correction coefficient values of other 3D cameras are obtained from the distance values and image positions of the 3D cameras 30-1 to 30-5. In addition,
The correction coefficient ROM of FIG. 18 is a 3D camera 30-5.
For (O ₅ ), the remaining 3D cameras 30-1 (O ₁ ) to
It is provided for each of 30-4 (O ₄ ). The size of the blind spot is stored in the processing area memory 63.

As described above, when the data of the other 3D cameras 30-1 to 30-4 are applied to the blind spot possibility area I shown in FIG.
Data of the D cameras 30-1 to 30-4 (surfaces with small changes in phase difference) are preferentially applied. In terms of fitting
If all the symbols including the designated far end of the image are not filled up, it is judged by the blind spot presence / absence judging means 65, and the blind spot portion S70 is stored in the blind spot storage memory 70.

Next, the blind spot possibility area II shown in FIG.
Is a grayscale image S52 of each 3D camera 30-1 to 30-5.
And the distance range from the distance value of the image having the distance value other than the portion replaced by the symbol of the distance image S53 to the far end of the designated image capturing area is the blind spot possibility area II. However, up to this blind spot possibility area II, the 3D cameras 30-1 (O ₁ ) and 30-5 are designated as blind spot possibility areas I.
(O ₅ ), and there is a possibility of blind spot I
The display for I has been replaced with an invalid symbol, so it can be said to be an area that can be ignored. That is, in the blind spot countermeasure device 60, it is sufficient to perform blind spot processing only on the blind spot possibility region I,
Moreover, since the distance of the phase difference change is smaller than the size of the display pixel, the presence or absence of the blind spot can be detected only by the 3D cameras 30-1 and 30-5 at both ends. In addition, the blind spot is the other 3
The data of the D cameras 30-2 to 30-4 are filled in, and the remaining portion can be recognized as a blind spot that cannot be detected by the apparatus of FIG. Thus, the middle 3
The blind spots of the D cameras 30-2 to 30-4 are 3D at both ends.
In the process of filling the blind spots of the cameras 30-1 and 30-5, the images with a small phase difference change are filled while being selected, so that the blind spot portion with a large phase difference is not used, and the blind spot processing is not particularly required. .

With respect to the images other than the blind spots obtained as described above, the pixel number S75 such as the camera number (angle of view, installation angle) and the pixel number (X direction, Y direction), the grayscale image S76, and the distance image S77. Are sequentially stored in the pixel number memory 75, the grayscale image memory 76, and the distance image memory 77. The present embodiment has the following advantages. (A) Image creating device 40-1 to 40-
5, the image pickup designated area LS is provided by the display density pixel creation devices 50-1 to 50-5, and the distance detection accuracy (one phase difference) is one pixel or less of the display image at the longest distance of the image pickup designated area LS. Since the grayscale image S52 is composed so as to obtain the value, the 3D cameras 30-1 and 3-3 at both ends are combined.
If only the output distance image S53 of the display density pixel creation devices 50-1 and 50-5 on the 0-5 side is determined by the facing surface determination unit 61 and the blind spot detection unit 62 performs the blind spot detection, the blind spot portion is obtained. You can easily identify the unsuitable part of the subject plane. Moreover, the output distance image S53 of the display density pixel creation devices 50-2 to 50-4 on the side of the intermediate 3D cameras 30-2 to 30-4
Since it suffices to correspond to the blind spot portion and the object surface unsuitable portion, a part of the screen may be scanned compared with the conventional full screen scanning, and the processing time can be shortened. Furthermore, the middle 3
The signals from the D cameras 30-2 to 30-4 do not require the facing surface determination means 61 and the blind spot detection means 62, and can be processed by the small blind spot correspondence selection means 64, so that the device scale can be reduced.

(B) Since the presence / absence of residual blind spots as a multi-viewpoint three-dimensional image input device is known by the blind spot presence / absence judging means 65, it is possible to know the degree of deterioration of the image quality due to the residual blind spots of the obtained image. It is possible to judge whether files can be filed at the time of filing. (C) The grayscale image S corresponding to the distance value when the pixel density conversion is performed by the display density pixel creation devices 50-1 to 50-5.
Since 44 is selected as a new grayscale image value, the grayscale image value becomes a composite value of at least the grayscale values of the subject, and the grayscale values of high purity are obtained in which the grayscale values of other subjects are not mixed. It should be noted that the present invention is not limited to the above-described embodiment, and for example, the image-capturing-designated-region image creating apparatus 40- of FIG.
1-40-5, display density pixel creation device 50-1-50-
5 and the blind spot countermeasure device 60, etc., can be modified in various ways, such as by changing to a configuration other than that shown in the drawing.

[0040]

As described in detail above, according to the first aspect of the present invention, the image pickup designation area is provided, and the image pickup designation area is located at least in all fields of view of the plurality of individual three-dimensional image input devices. However, since the individual three-dimensional image input devices are installed so that the intersections of the center lines of the distance width of the imaged designated area are gathered at one point, the subject specifying process and the selection of the optimum image thereof can be performed with a small amount of data processing. Can remove the blind spots. Therefore, it is possible to reduce the processing circuit scale and the processing time. According to the second aspect of the present invention, since the unnecessary area data discarding means for discarding all the image data outside the imaging designated area three-dimensionally is provided, it is possible to specify the blind spot portion and the object surface unsuitable portion with a smaller data processing amount, It is possible to further reduce the processing circuit scale and the processing time.

According to the third aspect of the invention, since the symbol substitution means for writing a predetermined symbol in the pixels having the maximum distance value or more within the width of the designated image pickup area is provided, the grayscale image and the distance image can be created with a small amount of data. Can be done. According to the fourth invention, since the display density pixel creating means is provided, the distance detection accuracy (for example, one phase difference) becomes a value equal to or less than one pixel of the display pixel at the longest distance of the designated image capturing area. Grayscale images are combined.
Therefore, if only the signals at both ends of the plurality of three-dimensional image input devices are used to perform the facing surface determination or the blind spot detection, the blind spot portion and the object surface unsuitable portion can be easily specified. Then, by making the signal of the three-dimensional image input device arranged in the middle correspond to the blind spot portion and the unsuitable portion of the object plane, it is sufficient to scan a part of the screen as compared with the conventional full screen scanning, and the processing time Can be shortened.

According to the fifth aspect, the display density pixel creating means has a function of setting a new distance value based on the distribution of the distance value of each pixel when performing the pixel density conversion. Easy to do. According to the sixth aspect, the display density pixel creation means selects only the grayscale value of the pixel corresponding to the new distance value in the grayscale image value after the pixel density conversion to perform a new grayscale image value. Since it has a function to
The blind spot detection using the new gray image value becomes easy.
According to the seventh aspect of the invention, since the blind spot coping means is provided, the blind spot portion and the subject surface unsuitable portion can be easily specified by performing the facing surface determination and the blind spot detection only on the signals of the three-dimensional image input devices at both ends. Then, the signal of the intermediate three-dimensional image input device corresponds to the blind spot portion and the object surface unsuitable portion,
Compared with the conventional full-screen scanning, a part of the screen may be scanned, and the processing time can be shortened. Moreover, it is not necessary to perform the processing for determining the facing surface and the blind spot detection for the signal of the intermediate three-dimensional image input device, and the processing can be performed by the blind spot correspondence selecting unit having a small circuit scale, so that the device scale can be reduced.

According to the eighth invention, since the offset is provided at the relative position of the two-dimensional image input device, the phase difference detection area of the two-dimensional image input device is expanded. According to the ninth invention,
The blind spot detecting means has a function of standardizing the blind spot discriminating value according to the distance value of the subject, so that the blind spot can be easily detected. According to the tenth invention, the blind spot presence / absence determining means is
Since it has a function of changing the number of pixels serving as a criterion for determining the remaining blind spot depending on the dimensional position, it becomes easy to determine whether or not the blind spot exists.

[Brief description of drawings]

FIG. 1 is a configuration block diagram of a multi-viewpoint three-dimensional image input device according to an embodiment of the present invention.

FIG. 2 is an explanatory diagram of a stereo image method which is one of conventional 3D image input methods.

FIG. 3 is an explanatory diagram of a grayscale image and a distance image obtained by the stereo image method of FIG.

FIG. 4 is a principle diagram of a multi-lens lenticular system, which is one of conventional 3D image display systems.

FIG. 5 is a schematic block diagram of the previously proposed multi-viewpoint three-dimensional image input device.

FIG. 6 is an explanatory diagram of the 3D camera in FIG. 1.

FIG. 7 is an explanatory diagram of an imaging designated distance range of FIG.

FIG. 8 is an explanatory diagram of an image capturing designated area in FIG.

9 is an explanatory diagram of an offset arrangement of the two-dimensional image input device shown in FIG.

10 is an explanatory diagram showing the processing contents of the image-capturing-designated-region image creating apparatus of FIG. 1. FIG.

11 is an explanatory diagram showing a limited area of the corresponding point search area in FIG. 1. FIG.

FIG. 12 is an explanatory diagram showing an example of pixel density conversion of FIG. 1.

FIG. 13 is an explanatory diagram showing a pixel density conversion method of FIG.

FIG. 14 is an explanatory diagram showing the inclination of the subject plane in FIG. 1.

FIG. 15 is an explanatory diagram showing a method of determining the object plane of FIG. 1.

16 is an explanatory diagram showing an angle formed by the object plane in FIG. 1 and a 3D camera installation angle.

17 is an explanatory diagram of a blind spot in FIG. 1. FIG.

FIG. 18 is a diagram showing a configuration example of a correction coefficient ROM used when determining the presence or absence of a blind spot in FIG. 1;

FIG. 19 is an explanatory diagram of a blind spot portion in FIG. 1 and an installation angle of another 3D camera that fills the blind spot.

20 is a diagram showing an example of the angle-of-view position and the correction coefficient value of FIG.

[Explanation of symbols]

 30-1 to 30-5 3D camera 31, 32 two-dimensional image input device 40-1 to 40-5 image creation device in image pickup designated area 41 area limited corresponding point search means 42 unnecessary area data discarding means 43 symbol substitution means 44, 52, 71, 73, 76 Gray image memory 45, 53, 72, 74, 77 Distance image memory 50-1 to 50-5 Display density pixel creation device 51 Pixel density conversion means 60 Blind spot coping device 61 Corresponding surface determination means 62 Blind spot Detecting means 63 Processing area memory 64 Blind spot correspondence selecting means 65 Blind spot presence / absence judging means 70 Blind spot storage memory 75 Pixel number memory

Claims (10)

[Claims] 1. A plurality of three-dimensional image input devices having at least two two-dimensional image input devices for inputting an image of an illuminated subject and outputting a signal of a grayscale image representing the subject. In the viewpoint three-dimensional image input device, an imaging designated area is provided, the imaging designated area is located in all the visual fields of the respective three-dimensional image input devices, and the intersections of the center lines of the distance widths of the imaging designated area gather at one point. Multiple viewpoints 3 characterized by arranging the respective three-dimensional image input devices as described above
Dimensional image input device. 2. The multi-viewpoint three-dimensional image input device according to claim 1, further comprising: unnecessary area data discarding means for three-dimensionally discarding all image data outside the imaging designated area. 3. The multi-viewpoint three-dimensional image input device according to claim 1, further comprising a symbol substituting means for writing a predetermined symbol in a pixel having a maximum distance value or more within the width of the designated imaging region. . 4. A grayscale image is synthesized by performing pixel density conversion on the output image of the three-dimensional image input device so that the distance detection accuracy becomes narrower than the pixel width of the desired display pixel at the longest distance of the image pickup designated area. Display density pixel creation means,
The multi-viewpoint three-dimensional image input device according to claim 1, wherein the three-dimensional image input device is provided for each of the three-dimensional image input devices. 5. The display density pixel creating means has a function of setting a new distance value based on a distribution of distance values of each pixel when performing pixel density conversion. Multi-view 3D image input device. 6. The display density pixel creating means selects only the grayscale value of a pixel corresponding to a new distance value from the grayscale image values after the pixel density conversion to obtain a new grayscale image value. The multi-viewpoint three-dimensional image input device according to claim 4, which has a function. 7. A facing surface determination means for determining the orientation of the object surface based on the output of the display density pixel creation means provided on both sides of the three-dimensional image input device of the plurality of three-dimensional image input devices. And a blind spot detection unit that detects a blind spot of the subject based on the determination result of the facing surface determination unit, and the display density pixel creation unit provided on a side of a three-dimensional image input device other than the three-dimensional image input devices at both ends. A blind spot correspondence selecting means for selecting and fitting an optimum image for the processing area of the facing surface judging means and the blind spot detecting means based on the output of the blind spot presence / absence judging means for judging the presence or absence of the residual blind spot from the fitting condition of the blind spot corresponding selecting means. The multi-viewpoint three-dimensional image input device according to claim 4, wherein the blind spot coping means provided with is connected to an output side of the plurality of display density pixel creating means. 8. In the phase difference detection direction of the two-dimensional image input device, the number of pixels of the phase difference determined by the display pixel density synthesized by the display density pixel creating means is equal to the 2 The multi-viewpoint three-dimensional image input device according to claim 4, wherein an offset is provided at a relative position of the three-dimensional image input device. 9. The multi-viewpoint three-dimensional image input device according to claim 7, wherein the blind spot detecting means has a function of standardizing a blind spot discriminant value according to a distance value of the subject. 10. The multi-viewpoint three-dimensional image input according to claim 7, wherein the blind spot presence / absence determining means has a function of changing the number of pixels serving as a criterion for determining the remaining blind spot depending on the three-dimensional position of the subject. apparatus.