JP3275252B2 - Three-dimensional information input method and three-dimensional information input device using the same - Google Patents

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to measurement of the position, shape and movement of a three-dimensional moving object, and more particularly to inputting, recognizing and expressing (CG) three-dimensional information of a moving object. The present invention relates to a method and a three-dimensional information input device using the same.

[0002]

2. Description of the Related Art Conventional three-dimensional information input technologies can be classified into three types. The first technique is stereo vision.
This is to associate the points and lines on the two images taken by the two cameras with the left and right images, so that the camera position and the corresponding points and lines of the left and right images can be used to determine the scene corresponding to the pair. This is a technique for estimating the positions of points and lines in space. Second
Is a method of restoring from motion. This is a technique for estimating a position of a point in a scene space corresponding to each feature point by tracking each feature point on the image with respect to a continuous image captured while moving a camera. The third technique is a back projection method. This is a technique for restoring the three-dimensional structure of the scene space by back-projecting the image onto the scene space.

[0003] The first technique is (for example, Kanade T. and Okutom).
i M. and Nakahara T .: “A multiplebaseline stereo me
thod ”, Proc. Image Understanding Workshop, pp. 409-
426,1992 or US Patent No.4,654,872), because many irregularities of the object tend to obstruct the line of sight of either camera, it is very difficult to accurately locate the feature points between the left and right images. It is difficult to obtain good three-dimensional information. The second method (eg Bolles RC,
Baker HH and Marimont DH: “Epipolar-plane im
age analysis: an apprproach to determining structu
re from motion ”, IJCV, Vol.1, No.1, pp.7-55,1987),
It cannot be applied to moving objects because the object must not move while taking images while moving the camera. Recently, a technique has been announced that can simultaneously extract the three-dimensional shape and motion of an object from a series of images taken by a single camera (Tomasi C. and Ka
nade T .: “Shape and motion from image streams unde
rorthography: a factorization method ”, IJCV, Vol.9,
No.2, pp. 137-154, 1992), since this method basically obtains three-dimensional information of feature points by tracking feature points between images, the method on the object caused by the movement of the object Very vulnerable to partial shielding. Because the feature point of interest on the image disappears or appears due to such occlusion, the locus of the feature point between the images is interrupted and tracking becomes difficult. Therefore, this method is difficult to adapt to the animal body. A third method is silhouette projection (eg, Ahuja N. an
d Veenstra J .: “Generation octree from object silh
ouettes in orthographic views ”, IEEE Trans.PAMI, V
ol. 11, No. 2, pp. 137-149, 1989), since it is extremely difficult and unstable to generate a silhouette image, it is difficult to obtain accurate three-dimensional information.

[0004]

On the other hand, as another example of the third method, an edge on an image is extracted, and the edge of the three-dimensional object is restored by voting in the scene space using the extracted edge. There is a method (Hamano T. and Yasuno
T. and Ishii K .: “Direct estimation of structure f
rom non-linear motion by voting algorithm with-out
tracking and matching ”, Proc. of ICPR, Vol.1, pp.
505-508,1992, and S. Kawato: “3D ShapeRecovering
by Octree Voting Technique ”, PROCEEDINGS of SPLE
-The International Society for Optical Engineerin
g, 15-16 Nov. 1992), such a method attempts to extract multiple feature points at the same time, and the processing for each feature point interferes with each other, which leads to the problem that false feature points are extracted. is there. To solve the problem, a large number of images are required. In the case of a moving object, it takes time to take a large number of images with one camera, and it is very expensive to construct a simultaneous imaging system with many cameras.

SUMMARY OF THE INVENTION An object of the present invention is to solve the above-mentioned problem and to provide a three-dimensional object information input method capable of acquiring three-dimensional information of an object from an object image taken by a small number of cameras, and a method using the method.
It is to provide a dimension information input device.

[0006]

According to a first aspect of the present invention, there is provided a three-dimensional information input method comprising the steps of: (a) picking up an object with n cameras, n being an integer of 3 or more; And output those images as image data,
(b) receiving the image data, extracting feature points of the object on the n-frame image, respectively, and (c) n
One of the cameras is selected as a reference camera, an image generated by the reference camera is used as a reference image, the other cameras are used as reference cameras, images generated by the cameras are used as reference images, The center of the lens,
A backprojection line passing through a feature point in the reference image selected corresponding to the three-dimensional feature point of interest on the object is defined as a reference backprojection line, and (d) the backprojection line is defined on the reference image by the reference camera. Projecting a reference backprojection line to define an epipolar line, respectively, and (e) a backprojection line passing through the feature point on the epipolar line on the reference image by the reference camera and the lens center of the reference camera. Is defined as the reference backprojection line, and (f) the coordinate position of the intersection of the reference backprojection line and the reference backprojection line is obtained, and the intersection of the reference backprojection line and the reference backprojection line at each intersection is obtained. Count the number, determine the maximum intersection of the counted value as the position of the noted three-dimensional feature point, and (g) add a series of the above steps (c) to (f) to each feature point on the reference image. By repeating the Obtaining a position of the three-dimensional feature point, respectively as three-dimensional information of the object.

[0007] The three-dimensional information input method according to the second aspect of the present invention comprises the steps of: (a) capturing images of an object by n cameras, n being an integer of 3 or more, obtaining images of n frames, Outputting the image as image data, (b) receiving the image data, extracting the feature points of the object on the n-frame image, and (c) extracting one of the n cameras
One as a reference camera, an image generated by the reference camera as a reference image, another camera as a reference camera, an image generated by them as a reference image, a lens center of the reference camera, and the object A back projection line passing through the feature point in the reference image selected corresponding to the above noted three-dimensional feature point is defined as a reference back projection line, and (d)
The reference backprojection line is projected onto the reference image by the reference camera to define an epipolar line, respectively, and (e) the feature points on the epipolar line on the reference image by the reference camera, A backprojection line passing through the lens center of the camera is defined as a reference backprojection line, and (f) a coordinate position of an intersection of the reference backprojection line and the reference backprojection line is obtained, and the coordinate position along the reference backprojection line is determined. The distribution of the intersection is subjected to a filtering process to emphasize the density of the distribution of the intersection by a convolution process, and the position where the distribution of the filtered intersection becomes the maximum is determined as the position of the noted three-dimensional feature point, (g) A series of the above steps (c) to (f)
Is repeated for each feature point on the reference image, thereby obtaining the position of each three-dimensional feature point on the object as three-dimensional information of the object.

A three-dimensional information input device according to a third aspect of the present invention is arranged at different positions from each other, and outputs images obtained by imaging an object as image data.
Where n is an integer of 3 or more, and image data from the n camera units, and image information processing unit that obtains three-dimensional information of the object from feature points in each image; Wherein the image information processing means includes: a feature extraction means for extracting a feature point in the image; one of the n cameras as a reference camera; an image generated by the reference camera as a reference image; Another camera is used as a reference camera, an image generated by the camera is used as a reference image, and a lens center of the reference camera and a feature point in the reference image selected corresponding to the noted three-dimensional feature point on the object are Reference backprojection line generating means for obtaining a backprojection line passing through as a reference backprojection line, and epipolar lines for projecting the reference backprojection line onto the reference image by the reference camera to obtain epipolar lines, respectively. Generating means, reference backprojection line generating means for obtaining a backprojection line passing through the feature point on the epipolar line on the reference image by the reference camera and the lens center of the reference camera as a reference backprojection line, ,
Determine the coordinate position of the intersection of the reference backprojection line and the reference backprojection line, count the number of intersections of the reference backprojection line and the reference backprojection line at each intersection, and determine the maximum intersection of the count value. Three-dimensional feature extracting means for determining the position of the three-dimensional feature point of interest, and for each feature point on the reference image, determining the position of each corresponding three-dimensional feature point on the object. Obtained as three-dimensional information of the object.

A three-dimensional information input device according to a fourth aspect of the present invention is arranged at different positions from each other, and outputs images obtained by imaging an object as image data.
Where n is an integer of 3 or more, and image data from the n camera units, and image information processing unit that obtains three-dimensional information of the object from feature points in each image; Wherein the image information processing means includes: a feature extraction means for extracting a feature point in the image; one of the n cameras as a reference camera; an image generated by the reference camera as a reference image; Another camera is used as a reference camera, an image generated by the camera is used as a reference image, and a lens center of the reference camera and a feature point in the reference image selected corresponding to the noted three-dimensional feature point on the object are Reference backprojection line generating means for obtaining a backprojection line passing through as a reference backprojection line, and epipolar lines for projecting the reference backprojection line onto the reference image by the reference camera to obtain epipolar lines, respectively. Generating means, reference backprojection line generating means for obtaining a backprojection line passing through the feature point on the epipolar line on the reference image by the reference camera and the lens center of the reference camera as a reference backprojection line, ,
A filter process for determining the coordinate position of the intersection of the reference backprojection line and the reference backprojection line, and emphasizing the density of the distribution of the intersection by convolution processing on the distribution of the intersection along the reference backprojection line. Filter processing means, and three-dimensional feature extraction means for determining a position where the distribution of the filtered intersection becomes the maximum as a position of the three-dimensional feature point of interest, and each feature point on the reference image. , The positions of the corresponding three-dimensional feature points on the object are obtained as three-dimensional information of the object.

[0010]

In the method and apparatus for inputting three-dimensional moving object information according to the present invention, the backprojection area is greatly narrowed by adopting the projection geometric constraint, so that the interference of backprojection is reduced to a minimum, and a small number of images are used. Can obtain three-dimensional information by back projection. Therefore, when obtaining three-dimensional information of a moving object every moment by simultaneous photographing with a plurality of cameras, it can be realized with a smaller number of cameras as compared with the related art.

[0011]

Embodiments of the present invention will be described below in detail with reference to the drawings. FIG. 1 is a block diagram showing the configuration of an embodiment of the three-dimensional information input device of the present invention. The three-dimensional information input device includes an image input unit 10 and an image information processing unit 20. N image input units 10 (n is an integer of 3 or more)
(For example, a TV camera, hereinafter simply referred to as a camera)
111 to 11n, A / D converters 121 to 12n, frame memories 131 to 13n, a camera control unit 14,
It comprises a movable base 15A, a drive device 15B, and a drive control unit 16. The cameras 111 to 11n are mounted on a common movable base 15A so as to have different heights, and their relative positions are maintained. The movable base 15A is a drive control unit 1.
Under the control of No. 6, three-dimensional movement is possible by a driving device 15B constituted by a robot arm.

The cameras 111 to 11n simultaneously image the object 19 at a fixed period (frame period) and output analog image signals. If the object is not moving, it is not always necessary to capture images at the same time. Analog image signal is A / D
The digital signals are converted into digital signals by the converters 121 to 12n, and are converted into image data for each frame.
3n. The operations of the camera, the A / D converter, and the frame memory are controlled by the camera control unit 14. The image information processing unit 20 includes frame memories 131 to 13
The image data read from n is given.
By performing the processing described with reference to FIG.
Find dimensional information.

FIG. 2 is a functional block diagram showing a processing procedure of the image information processing section 20 constituted by a computer. Camera 1
The position and orientation information of the camera at the time of image input of the object 19 by 11 to 11n is transmitted from the drive control unit 16 to the image information processing unit 20.
Is input to the camera information input unit 21 of the camera. On the other hand, the image data read from the frame memories 131 to 13n is supplied to the feature extracting unit 22. The feature extraction unit 22 extracts feature points (for example, iso-density points / regions of a normal image, iso-thermal points / regions of a thermal image, etc.) from n-frame images input from the image input unit 10. The reference back-projection line generation unit 23 selects any one of the n-frame images from which the feature points have been extracted as a reference image, and selects the feature points on the reference image and the reference points input by the camera information input unit 21. A reference backprojection line passing through the center of the lens of the camera that generated the image (referred to as a reference camera) is obtained. The epipolar line generation unit 24 projects the reference backprojection line generated by the reference backprojection line generation unit 23 onto an image (referred to as a reference image) surface generated by a camera other than the reference camera (referred to as a reference camera). Obtain the obtained epipolar lines. The reference back-projection line generation unit 25 uses the camera information input unit 21 with each feature point existing on the epipolar line on each reference image generated by the epipolar line generation unit 24 or within a certain distance range on both sides from the epipolar line. All the reference backprojection lines passing through the center of the reference camera corresponding to those input epipolar lines are obtained. The intersection extraction unit 26 calculates the coordinates of all the intersections between the reference backprojection line generated by the reference backprojection line generation unit 23 and the reference backprojection line generated by the reference backprojection line generation unit 25. The intersection counting unit 27 counts the number of intersections at the intersection or intersection area between each reference backprojection line extracted by the intersection extraction unit 26 and the reference backprojection line along the reference backprojection line. Filtering unit 2
8 filters the distribution of the intersection count with the reference backprojection line along the reference backprojection line counted by the intersection counting unit 27 along the reference backprojection line. The three-dimensional feature extraction unit 29 determines an intersection closest to the peak position where the filtered intersection distribution exceeds a certain threshold value as the position of the three-dimensional feature point. The three-dimensional motion extraction unit 2A obtains the motion of each three-dimensional feature point / region based on the three-dimensional feature point / region information at each time extracted by the three-dimensional feature extraction unit 29.

Next, the operation of the thus constructed apparatus of the present invention will be described with reference to FIGS. FIG. 3 shows the arrangement of the respective cameras 111 to 11n and the inverted image formed on each focal plane by the respective cameras as a set image 32 by backprojecting the inverted image to the front of the lens for convenience. First, the camera control unit 14 simultaneously operates the n cameras 111 to 11n from the input time t, and converts the obtained analog image signals into digital image signals by the n A / D converters 121 to 12n. Frame memories 1
The image data is taken into the memories 31 to 13n, and one frame of image data is obtained in each memory. The camera information input unit 21 obtains camera information at the time t by calibration (for example, J. Weng and P. Cohen and M. Herniou: “Calibration o
f stereocameras using a non-linear distortion mode
l ", Proc. ICCV, pp. 246-253, 1990).

The feature extraction unit 22 extracts feature points (for example, iso-density points / areas of an image, contour lines / areas of a differential image, isothermal points / areas of a thermal image, etc.) on each image 32 shown in FIG. . Here, for example, Canny's filter (Canny JF:
“A Computational approach to edge detection”, IEE
E Trans. PAMI, Vol. 8, No. 6, pp. 679-698, 1986) to extract image edges and use them as feature points. Next, an arbitrary one-frame image 32 is selected as a reference image 32s by the reference backprojection line generation unit 23, and one focused feature point 33s on the reference image 32s and the image input by the camera information input unit 21 are displayed. A back projection line 34s passing through the center 31s of the lens of the corresponding camera (reference camera) is determined.

The coordinates (x _b ,
y _b , z _b ) is the following equation

[0017]

(Equation 1) To be satisfied. Here, (x _p , y _p , z _p ) and (x _c , y _c , z _c ) are the coordinates of the feature point of interest on the image 32s and the lens center 31s of the reference camera, respectively. Such a back projection line 34s is set as a reference back projection line. Any one point E = (x on the epipolar line 35 obtained by projecting the reference back projection line onto the image of the reference camera
_e , y _e )

[0018]

(Equation 2) There is a relational expression as follows. Here, R ^{T ′} and T ′ are two-dimensional representations of a transpose matrix of a matrix defining rotation of the camera optical axis with respect to the coordinate system and a vector defining translation, respectively. B ′ = (x _b , y _b ) ^T is a two-dimensional representation of an arbitrary point on the back projection line. As is apparent from FIG.
s lies in the plane containing the center 31 of the lens of each reference camera and the epipolar line 35 from that reference camera. This plane is called the epipolar plane by the camera. All three-dimensional feature points on the object 19 existing in the extension plane of the epipolar plane of each camera are projected onto the epipolar line 35 of the camera. Next, the reference backprojection line generation unit 25 calculates all feature points 33e on the obtained epipolar line 35.
Find out. These feature points 33 on the epipolar line 35
One of e corresponds to the noted feature point 33s. Actually, since an image quantization error exists, the epipolar line 3
In some cases, a feature point that should be on 5 deviates from the epipolar line. Therefore, the feature points existing within a certain distance from the epipolar line are replaced with the feature points 33e of the epipolar line 35.
Consider Next, a back projection line 34 passing through each feature point 33e on the epipolar line 35 of the image 32 by each reference camera and the center 31 of the lens of the reference camera is obtained, and such a back projection line 34 is referred to as a back projection. A line. Next, the intersection extraction unit 26 calculates the coordinates of the intersection 36 between the reference backprojection line 34 s generated by the reference backprojection line generation unit 23 and the reference backprojection line 34 generated by the reference backprojection line generation unit 25. Ask. At this time, since the feature extraction error and the camera lens center error on the image 32 are present, the reference backprojection line 34s and the reference backprojection line 34 often do not intersect in the three-dimensional space. In this embodiment, the reference back projection line 3
4s and each reference backprojection line 34, and if this shortest distance is smaller than a certain threshold value, the reference backprojection line 3
4s and the reference backprojection line 34 are considered to intersect.

That is, if any two points B _b1 = (x _b1 , y _b1 , z _b1 ) ^T and B _b2 = (x _b2 , y _b2 , z _b2 ) ^T on the reference back projection line 34 s, Any two points on the projection line: a matrix composed of B _r1 = (x _r1 , y _r1 , z _r1 ) ^T and B _r2 = (x _r2 , y _r2 , z _r2 ) ^T

[0020]

(Equation 3) If the rank of is 1, the two lines are not parallel in space. In this case, two points (x _bs , y _bs , z _bs ) ^T and (x _rs , y _rs , z) on the reference back projection line and the reference back projection line, and giving the shortest distance between the two lines, respectively. _rs ) ^T

[0021]

(Equation 4) Given by _{Here, K b = (G r H} -F r G b) / (F b F r -H 2) K r = (F b G r -G b H) / (F b F r -H 2) F _b = (x _b2 -x _b1 ) ² + (y _b2 -y _b1 ) ² + (z _b2 -z _b1 ) ² F _r = (x _r2 -x _r1 ) ² + (y _r2 -y _r1 ) ² + ( z _r2 -z _r1 ) ² G _b = (x _b2 -x _b1 ) (x _b1 -x _r1 ) + (y _b2 -y _b1 ) (y _b1 -y _r1 ) + (z _b2 -z
_b1 ) (z _b1 -z _r1 ) G _r = (x _r2 -x _r1 ) (x _r1 -x _b1 ) + (y _r2 -y _r1 ) (y _r1 -y _b1 ) + (z _r2 -z
_r1 ) (z _r1 -z _b1 ) H = (x _b2 -x _b1 ) (x _r1 -x _r2 ) + (y _b2 -y _b1 ) (y _r1 -y _r2 ) + (z _b2 -z
_b1) (z _r1 -z _r2) If, as shown the distance d between these two points then the threshold λ
If the condition d = [(x _bs -x _rs ) ² + (y _bs -y _rs ) ² + (z _bs -z _rs ) ² ] ^1/2 <λ (6) is satisfied, these 2 considers each reference backprojection line 34 and the reference backprojection line 34s in the presence of the point cross, the midpoint _{x = (x bs + x rs} ) of the distance between the coordinate position of the intersection 36 of these two points / 2 y = ( y _bs + y _rs ) / 2 z = (z _bs + z _rs ) / 2 (7) Next, the intersection counting unit 27 counts the number of intersections with the reference backprojection lines 34 from all the reference cameras at each position on the reference backprojection line 34s.

When images taken by two (n = 2) cameras are used, a plurality of reference backprojection lines 34 are provided at a plurality of different positions with respect to one reference backprojection line 34s. Intersecting with the projection line 34s can occur. In this case, it is not possible to uniquely determine which intersection the three-dimensional feature point corresponding to the feature point 33s through which the reference backprojection line 34s passes corresponds, but which does not. Therefore, the following is considered.

The number of feature points on the epipolar line of each of the other cameras defined by the straight line 34s back-projected from one camera to the feature point of interest is assumed to be equal to M for simplicity. Only one of the M feature points on the epipolar line on the image from each camera corresponds to the feature point of interest, or none of them. Here, it is assumed that one of the M feature points on each epipolar line corresponds to the feature point of interest. Therefore, M reference backprojection lines are obtained by one reference camera, and the probability β that any one of the M intersections of the M reference backprojection lines and the reference backprojection line is the feature point of interest is β = 1 / M. The same applies to other reference cameras. Therefore, the reference back projection line 34
The probability α that one reference backprojection line from each reference camera intersects at an arbitrary intersection on s is α = β ^n-1
Becomes When there is only one feature point on the epipolar line of the reference camera (M = 1), the feature point corresponds to the feature point of interest, and the reference back-projection line passing through the feature point and the center of the camera lens is of interest. There is no problem because it intersects with the feature point. A problem arises when the number M of reference backprojection lines from each reference camera is 2 or more, so the value of β = 1 / M may be considered from 0 to 0.5.

FIG. 4 shows that for various values of β in the range 0 <β ≦ 0.5, (n−1) reference backprojection lines from (n−1) reference cameras are on the reference backprojection line. Probability of intersection at one point α
Is shown in a graph. As is clear from this graph, even when β = 0.5, the number of all cameras (the number of viewpoints) n is 6
If this is the case, the probability α that five reference backprojection lines from the reference camera intersect at one point on the reference backprojection line is (0.5) ⁵ = 0.03
And become negligibly small. Even when n = 5 (the number of reference cameras is 4), α is as small as 0.06. That is, the possibility that a plurality of reference backprojection lines from each reference camera accidentally intersect at the feature point of interest sharply decreases with the number n of cameras. Conversely, if the number of reference backprojection lines that actually intersect at a certain point on the reference backprojection line is plural (for example, four or more), it is very likely that the intersection is the feature point of interest. Means. Therefore, as described above, the number of reference backprojection lines (simply called the number of intersections) intersecting at each point on the reference backprojection line 34 s is counted by the intersection counting unit 27, and the maximum If the position given the number of intersections is obtained, the position of the focused three-dimensional feature point can be uniquely determined. In this way, an arbitrary three-dimensional feature point on the object 19 can be made to correspond to one intersection on the reference back projection line.

From the above, according to the principle of the present invention, the value of n is selected to be 3 or more. In a preferred embodiment, six different viewpoints, that is, six cameras are used. With such a configuration, the object 1 can be viewed from a certain camera.
There is a high possibility that the three-dimensional feature point of the part which is not observed because of being covered by the camera 9 itself will be more likely to be observed by other cameras, and therefore the position of such three-dimensional feature point can be determined. In the actual measurement, as shown in FIGS. 5A and 5B, the feature point 33 on the image 32 and the camera lens center 3
The back projection line may deviate from the true direction due to the position error of 1. For example, since the image 32 composed of the image data obtained by each camera has been subjected to the quantization processing by the pixels of the image, the position of each feature point on the image includes the quantization error by the pixels. Therefore, as shown in FIG. 5A, the true feature points 33 on the image 32 by the reference camera are displayed.
Is shifted to P 'due to a quantization error, the reference backprojection line 34 is shifted by being rotationally shifted about the lens center 31, so that the true feature point B on the reference backprojection line 34s is B' ( (Not necessarily on the reference backprojection line). Alternatively, as shown in FIG. 5B, if the lens center determined by measuring with respect to the true camera lens center L is shifted to L ′, the true three-dimensional feature point B on the reference back projection line 34 s becomes B ″. It is determined that there is.

Reference back projection line 3 due to the influence of these two errors
Because of the shift of 4, the reference back projection line 34 that passes through the camera lens center 31 and the corresponding feature point 33 that is projected on each image from the same three-dimensional feature point on the object is such that the three-dimensional feature point truly exists. There is a possibility of crossing out of where you do.
As a result, the displaced point B 'or B "
If the intersection number is within the threshold distance from s, there is a possibility that when the number of intersections is counted by the intersection counting unit 27, the intersection number may be counted at an intersection position different from the true focused three-dimensional feature point B on the reference back projection line 34s. is there. In such a case, if the position of the intersection where the maximum count value exceeding the predetermined count value is given as the position of the three-dimensional feature point is directly extracted by the threshold value processing in the three-dimensional feature point extraction unit 29, the three-dimensional feature point of interest is extracted. There is a possibility that three-dimensional feature points other than points may be extracted as pseudo three-dimensional feature points. However, since the direction and magnitude of the error of these back projection lines are random, in the actually obtained distribution of discrete and widened counts, the central part where the intersections are concentrated is a true three-dimensional target. It can be estimated that the feature points are most likely to be present. Therefore, filtering is performed along the reference backprojection line 34 s at which the intersection is found by the filtering unit 28, and the distribution of the intersection count is emphasized where the intersection is concentrated, thereby reducing false extraction of pseudo three-dimensional feature points. Can be done. Examples of filters used include a square filter, a triangular filter, a Gaussian filter, and a Laplacian filter.
Gaussian filters, etc. are possible.

To filter the discrete distribution P (s) of the cross counts 51 along the reference back projection line 34s as shown in FIG. 6A by the function f (t), the following equation P '(s ) = ∫P (st) · f (t) dt (8) The convolution operation may be performed. Here, P (s) and P ′ (s) are distributions of the backprojection line intersection count before and after convolution of the filter function f (t), respectively, and s represents a position along the reference backprojection line 34s. . Various filter functions as described above can be used as the filter function f (t). In this embodiment, a Laplacian-Gaussian filter (referred to as a ▽ ² G filter) 52 having a characteristic as shown in FIG. 6B is convolved with the distribution P (s) of the intersection count 51 along the reference backprojection line 34 s. Do. ▽ Since ² G filter has a characteristic emphasizing peaks of the waveform are continuous while suppressing the high-frequency discrete noise, the convolution discrete intersection counts is suppressed by the processing, concentrate on the intersection of the count are Peaks are highlighted as shown in FIG. 6C. Therefore, the three-dimensional feature extraction unit 29 next detects the intersection count peak 51p on the reference back projection line 34s, and determines that the intersection closest to the peak position is the position of the focused three-dimensional feature point. It is possible to prevent false extraction of pseudo three-dimensional feature points.

As described above, the intersection with the reference backprojection line 34 from each reference camera is defined on the reference backprojection line 34s, and the number of intersections at each intersection is counted. Because of the lens center error or the pixel quantization error of the feature point as shown in the following, the intersection of the reference backprojection line 34 from each reference camera corresponding to the true focused three-dimensional feature point with the reference backprojection line 34 s is strictly Not one point. Therefore, in order to count the number of intersections at each intersection, the reference backprojected line 34s is divided into unit lengths in the length direction using a predetermined small length Δs as a unit length, and each unit length Δs What is necessary is just to count the number of intersections inside. In this case, the smaller the unit length Δs is selected, the higher the resolution with respect to the intersection becomes. Therefore, the unit length Δ at the same position on the reference back projection line 34 s
The number of intersections contained in s will be reduced. If the unit length Δs is reduced to such a degree that all the intersections are separated, the maximum number of intersections at each intersection is 1, and it is not possible to uniquely identify the focused three-dimensional feature point. However, even in this case, for example, as shown in FIG. 6A, the intersections are dense near the focused three-dimensional feature point.
If the convolution processing by the above-described filter is performed on the discrete distribution of intersections, a peak is obtained at the center of the dense area of the intersection, and the intersection closest to the peak position is set to true 3
The position of the dimensional feature point can be determined.

All other feature points 33 on the reference image 32s
By repeating the above process for each of s
The coordinates of all three-dimensional feature points 36t corresponding to all feature points 33s on the reference image 32s can be determined.
At this time, since the back projection lines that intersect each other at the position determined to correspond to the target three-dimensional feature point are related only to the target three-dimensional feature point, when the other three-dimensional feature points are obtained, By removing feature points 33 and 33s on the images 32 and 32s corresponding to the three-dimensional feature points whose coordinates have been determined from the respective images, it is possible to reduce the interference of backprojection and the amount of calculation. Further, by selecting another image as a reference image and repeating the above processing until all images are selected, it is possible to obtain the positions of the three-dimensional feature points corresponding to the feature points on the images obtained from all viewpoints. it can.

Finally, the three-dimensional motion extracting unit 2A extracts three-dimensional motion information of the moving object by obtaining a temporal change of the three-dimensional feature point information obtained from the three-dimensional feature extracting unit 29. As a specific method, a two-dimensional motion information extraction method (for example, R. Agarwal and J. Sklansky: Optical flo
w estimation by the clustering of constraint sets
in velocity space, Technical Report TP-91-2, Departm
ent of Electrical and Computer Engineering, Patter
n Recognition and Image Modeling Project, Univers
ity of California, Irvine, CA, USA) to a three-dimensionally extended method.

The motion of the three-dimensional feature points extracted by the three-dimensional motion extraction unit 2A is given to the drive control unit 16 of the image input unit 10, the next viewpoint is planned, the drive unit 15 is operated, and a plurality of At the same time as the imaging units 111 to 11n are moved, the amount of movement of the driving unit 15 at the input time t + w is measured and given to the camera information input unit. By repeating such an operation, it is possible to track the moving object.

[0032]

As described above, the present invention greatly reduces the backprojection area by adopting the projection geometric constraint, so that the interference of the backprojection is reduced to a minimum and the number of images in a small number of images is reduced. Three-dimensional information can be obtained by back projection. Therefore, the three-dimensional shape of the moving object can be restored every moment by simultaneous photographing with a small number of cameras.

[Brief description of the drawings]

FIG. 1 is a block diagram showing a configuration of an embodiment of a three-dimensional information input device according to the present invention.

FIG. 2 is a processing block diagram for explaining details of an image processing unit 20 in FIG. 1;

FIG. 3 is a back projection diagram for explaining the principle of the present invention.

FIG. 4 is a stereograph for explaining the probability that reference backprojection lines intersect at the same point.

5A is a perspective view illustrating a pixel quantization error of a feature point in an image, and FIG. 5B is a perspective view illustrating an error of a lens center.

6A is a diagram illustrating an intersection distribution on a reference backprojection line, FIG. 6B is a diagram illustrating an example of a characteristic of a filter, and FIG. 6C is a diagram illustrating an example of an intersection distribution subjected to a filtering process.

[Explanation of symbols]

 DESCRIPTION OF SYMBOLS 10 Image input part 111-11n Camera 15A Movable base 15B Drive 19 Object 31 Lens center 32 Image 33 Feature point 34 Back projection line 35 Epipolar line 36 Intersection

──────────────────────────────────────────────────の Continuation of the front page (51) Int.Cl. ⁷ Identification code FI G06T 7/00 G06F 15/68 400A (56) References JP-A-63-10280 (JP, A) JP-A-61-125686 ( JP, A) (58) Fields investigated (Int. Cl. ⁷ , DB name) G01B 11/00-11/30 102 G06T 1/00 G06T 5/20 G06T 7/00 G06T 7/60

Claims (22)

(57) [Claims] An image is obtained by capturing an object with a plurality of cameras, and three-dimensional feature points on the object are made to correspond to feature points in each of the images to obtain three-dimensional information on the object. A method for inputting dimensional information, including the following steps: (a) Obtaining n-frame images by imaging objects with n cameras, where n is an integer of 3 or more, and using those images as image data (B) receiving the image data, extracting feature points of the object on the n-frame image, respectively, and (c) selecting one of the n cameras as a reference camera. The image generated by the camera is used as a reference image, the other cameras are used as reference cameras, and the images generated by the cameras are used as reference images, corresponding to the lens center of the reference camera and the noted three-dimensional feature point on the object. Select A backprojection line passing through the selected feature point in the reference image is defined as a reference backprojection line, and (d) the reference backprojection line is projected onto the reference image by the reference camera to define an epipolar line. (E) defining a backprojection line passing through the feature point on the epipolar line on the reference image by the reference camera and the lens center of the reference camera as a reference backprojection line; Determine the coordinate position of the intersection of the reference backprojection line and the reference backprojection line, count the number of intersections of the reference backprojection line and the reference backprojection line at each intersection,
By determining the maximum intersection of the count value as the position of the three-dimensional feature point of interest, (g) repeating a series of the steps (c) to (f) for each feature point on the reference image, The position of each three-dimensional feature point on the object is obtained as three-dimensional information of the object. 2. Obtaining an image by imaging an object with a plurality of cameras and obtaining three-dimensional information on the object by associating three-dimensional feature points on the object with feature points in each of the images. A method for inputting dimensional information, including the following steps: (a) Obtaining n-frame images by imaging objects with n cameras, where n is an integer of 3 or more, and using those images as image data (B) receiving the image data, extracting feature points of the object on the n-frame image, respectively, and (c) selecting one of the n cameras as a reference camera. The image generated by the camera is used as a reference image, the other cameras are used as reference cameras, and the images generated by the cameras are used as reference images, corresponding to the lens center of the reference camera and the noted three-dimensional feature point on the object. Select A backprojection line passing through the selected feature point in the reference image is defined as a reference backprojection line, and (d) the reference backprojection line is projected onto the reference image by the reference camera to define an epipolar line. (E) defining a backprojection line passing through the feature point on the epipolar line on the reference image by the reference camera and the lens center of the reference camera as a reference backprojection line; The coordinate position of the intersection of the reference backprojection line and the reference backprojection line is obtained, and the distribution of the intersection along the reference backprojection line is subjected to convolution processing to perform filter processing to emphasize the density of the distribution of the intersection. Determining the position where the distribution of the filtered intersection becomes the maximum as the position of the noted three-dimensional feature point, and (g) performing a series of the steps (c) to (f) on the reference image. Repetition By returning, the position of each three-dimensional feature point on the object is obtained as three-dimensional information of the object. 3. The three-dimensional information input method according to claim 1, wherein the position on the reference backprojection line is defined in units of a predetermined unit length in the length direction, and the intersection is defined. Is the number of intersections within the unit length at the position on the reference backprojection line through which the reference backprojection line passes. 4. The three-dimensional information input method according to claim 3, wherein any reference backprojection line intersecting with the reference backprojection line has a distance from the reference backprojection line within a predetermined range. Reference backprojection line. 5. The three-dimensional information input method according to claim 1, wherein the step of counting the number of intersections is performed by convoluting the distribution of the intersections along the reference backprojection line. This is a step of performing a filter process for emphasizing, and determining the position where the distribution of the filtered intersection becomes the maximum as the position of the noted three-dimensional feature point. 6. The three-dimensional information input method according to claim 2, wherein the filter processing is processing by a Laplacian-Gaussian filter. 7. The three-dimensional information input method according to claim 2, wherein the filter processing is processing using a square filter. 8. The three-dimensional information input method according to claim 2, wherein the filtering is performed by a triangular filter. 9. The three-dimensional information input method according to claim 2, wherein the filter processing is processing using a Gaussian filter. 10. The three-dimensional information input method according to claim 1, wherein the step (g) of repeating the series of steps includes executing the series of steps to determine a feature point corresponding to the focused three-dimensional feature point. For each identification, the method includes a step of removing the identified feature point from a processing target in a repetition of the following series of steps. 11. The three-dimensional information input method according to claim 1, wherein the number n of the cameras is 5 or more and 8 or less. 12. The three-dimensional information input method according to claim 1, wherein one of the reference cameras is set as a new reference camera each time the step (g) is completed, and the reference camera is set as a new reference camera. New step (g) as a camera
Is repeated for all the cameras. 13. The three-dimensional information input method according to claim 1, wherein the imaging of the object by the n cameras is simultaneously performed at regular intervals, and the three-dimensional information of the object is determined according to the series of steps. Obtaining dimensional information. 14. An n-number of camera units which are arranged at mutually different positions and output images obtained by imaging an object as image data, n is an integer of 3 or more, and n camera units Image information processing means for obtaining three-dimensional information of the object from characteristic points in each image, which is provided with image data, wherein the image information processing means includes: feature extracting means for extracting characteristic points in the image And one of the n cameras as a reference camera, an image generated by the reference camera as a reference image, another camera as a reference camera, and an image generated by the cameras as a reference image. A back projection line passing through the center of the lens and a feature point in the reference image selected corresponding to the noted three-dimensional feature point on the object as a reference back projection line. And epipolar line generation means for projecting the reference backprojection line onto the reference image by the reference camera to obtain an epipolar line, respectively, and the feature points on the epipolar line on the reference image by the reference camera. A reference backprojection line generating means for obtaining a backprojection line passing through the lens center of the reference camera as a reference backprojection line; and obtaining coordinate positions of intersections of the reference backprojection line and the reference backprojection line, respectively. 3D feature extraction means for counting the number of intersections between the reference backprojection line and the reference backprojection line at an intersection, and determining the maximum intersection of the counted value as the position of the 3D feature point of interest. For each feature point on the reference image,
A three-dimensional information input device for obtaining a position of each corresponding three-dimensional feature point on the object as three-dimensional information of the object. 15. An n-number of camera units arranged at mutually different positions and outputting images obtained by imaging an object as image data, where n is an integer of 3 or more, and n number of the camera units Image information processing means for obtaining three-dimensional information of the object from characteristic points in each image, which is provided with image data, wherein the image information processing means includes: feature extracting means for extracting characteristic points in the image And one of the n cameras as a reference camera, an image generated by the reference camera as a reference image, another camera as a reference camera, and an image generated by the cameras as a reference image. A back projection line passing through the center of the lens and a feature point in the reference image selected corresponding to the noted three-dimensional feature point on the object as a reference back projection line. And epipolar line generation means for projecting the reference backprojection line onto the reference image by the reference camera to obtain an epipolar line, respectively, and the feature points on the epipolar line on the reference image by the reference camera. A reference backprojection line generating means for obtaining a backprojection line passing through the lens center of the reference camera as a reference backprojection line; and obtaining a coordinate position of an intersection of the reference backprojection line and the reference backprojection line, respectively. Filter processing means for performing filter processing to emphasize the density of the distribution of the intersection by convolution processing with respect to the distribution of the intersection along the back projection line, and focusing on the position where the distribution of the filtered intersection becomes maximum. Three-dimensional feature extraction means for determining the position of the three-dimensional feature point, and for each feature point on the reference image, The position of each of the three-dimensional feature point to respond inputting device for obtaining a three-dimensional information of the object. 16. The three-dimensional information input device according to claim 14, wherein the three-dimensional feature extracting means determines the density of the distribution of the intersection by convolution processing on the distribution of the intersection along the reference backprojection line. It includes filter processing means for enhancing, and determines the position where the distribution of the filtered intersection becomes the maximum as the position of the noted three-dimensional feature point. 17. The three-dimensional information input device according to claim 15, wherein said filter processing means is a processing means using a Laplacian-Gaussian filter. 18. The three-dimensional information input device according to claim 15, wherein the filter processing means is a processing means using a square filter. 19. The three-dimensional information input device according to claim 15, wherein said filter processing means is a processing means using a triangular filter. 20. The three-dimensional information input device according to claim 15, wherein the filter processing means is a processing means using a Gaussian filter. 21. The three-dimensional information input device according to claim 14, wherein the number n of the cameras is 5 or more and 8 or less. 22. The three-dimensional information input device according to claim 14, wherein the imaging of the object by the n cameras is simultaneously repeated at a constant cycle, and the image information processing is performed every time image data is obtained. Camera control means for providing the means is provided.