JP2002032742A - System and method for three-dimensional image generation and program providing medium - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a three-dimensional image generation system which automates the feature point setting and batch setting processing of a photographed image and can generate a three-dimensional image of high picture quality. SOLUTION: Object characteristics are detected and a three-dimensional model generation area based upon a shape model stored in a database is automatically extracted. For example, the area of face, etc., is automatically detected. A feature point is automatically set in the area of an image according to feature point position setting data of an object model stored in the database with respect to the extracted object area. This method is more accurate and securer than a general way of determining a feature window with feature variable evaluated values. Further, a triangular patch by the model is automatically generated for an extracted feature point and then wire frame rendering matching the actual object shape becomes possible, so that respective processes of the three-dimensional model generating process are automated.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a method for tracking a feature point of an object of interest in a plurality of two-dimensional images taken by a single-lens camera.
The present invention relates to a three-dimensional image generation system and a three-dimensional image generation method for realizing a technique for automatically extracting, tracking and correcting feature points and automatically creating a triangular patch based on a knowledge base or a target model in 3D modeling for generating a three-dimensional shape.

[0002]

2. Description of the Related Art In a plurality of images (or time-series images) such as landscapes, buildings, and portraits of people, which are taken and recorded by a digital camera or a video camera while walking outdoors, the photographing viewpoint and the object are determined. If there is a relative movement between them, it is possible in principle to restore the three-dimensional shape of the photographing target from the plurality of images and the time-series images. Actually, research on a method for simultaneously obtaining the relative motion of the camera and the target shape using a plurality of images is an important theme expected in various application fields. One typical technique is to extract the features of the target (points and lines such as corners and edges of the object) in the image, track the features of the same target in the time-series image, and perform linear camera projection. A method of reconstructing the 3D shape of an object by a factorization method (Factorization) using a model was proposed by Kanade [for example, (1) Carlo Tomasi: Shape and Motion from Im
age Streams: a Factorization Method. CMU-CS-91-17
2, Sep. 1991, (2) Conrad J. Poelman: A Paraperspecti
ve and Projective Factorization Methods for Recove
ring Shape and Motion. CMU-CS-95-173, (1995), (3) J
oao Costeira and Takeo Kanade: A Multi-body Factor
ization Method for Motion Analysis. CMU-CS-TR-94-2
20, (1994), (4) Toshihiko Morita and Takeo Kanade:
A Sequential Factorization Method for Recovering S
hape and Motion from Image Streams. CMU-CS-94-158,
(1994), (5) Naoki Chiba and Takeo Kanade: A Tracke
r for Broken and Closely-Spaced Lines. CMU-CS-97-1
82, Oct. (1997)]. FIG. 1 shows a conceptual diagram of the three-dimensional shape and camera motion restoration by this method.

First, an object is photographed while changing the viewpoint of the camera. F ft (x, y) | t
= 1,. . . . , F} and the first image f1
P feature points are extracted from (x, y) by the variance evaluation value in the window and the like, and are traced over F images. The feature point coordinates (Xfp, Yfp) on each frame obtained by the feature point tracking can be represented by a matrix W (herein referred to as a measurement matrix). Then, by applying a linear projection model, the measurement matrix W can be decomposed into two orthogonal matrices U and V ′ and a diagonal matrix に よ っ て by singular value decomposition (SVD). Therefore, the orthogonal matrices U and V ′ include the camera motion information M and the target three-dimensional information S, respectively. Further, by using the constraint condition of the unit vector (i, j, k) representing the posture of the camera, a matrix A for uniquely determining M and S is obtained. Therefore, the camera movement M and the three-dimensional shape S of the object are uniquely determined.

In general, this method restores a three-dimensional shape of an object at a high speed and simultaneously obtains three-dimensional information of a photographing camera.
(Position, posture, angle of view, etc.) can also be restored.

[0005]

In this method, it is most important to reliably extract and track required feature points. However, in practice, the criterion as to what feature points are required to create a target three-dimensional shape is not clear. In addition, it is not always easy to reliably associate feature points (feature point tracking) in a plurality of images with respect to those required feature points.
Furthermore, in order to represent the target three-dimensional shape, it is necessary to create triangular patches between feature points and map texture images to those patches. Automatic creation of triangular patches is not easy.

The present invention examines these problems, shows a feature point determination method based on a priori knowledge of an object, and proposes a feature point tracking method and an erroneous tracking correction method. It is another object of the present invention to provide a method capable of automatically creating a triangular patch between feature points based on the same knowledge model.

[0007]

SUMMARY OF THE INVENTION The present invention has been made in consideration of the above problems, and a first aspect of the present invention is a three-dimensional image generating method for generating a three-dimensional image by performing feature point tracking processing on a plurality of input images. In the image generation system, a three-dimensional model generation region extraction processing means for setting a three-dimensional model generation region on an input image based on a shape model stored in a database, and feature point position data stored in the database A three-dimensional model feature point selection processing means for setting feature points in the three-dimensional model generation area of the input image; and a patch area in the three-dimensional model generation area of the input image based on the patch area setting data stored in the database. And a three-dimensional model patch area setting processing means for setting.

Further, in one embodiment of the three-dimensional image generation system according to the present invention, the three-dimensional model generation region extraction processing means determines characteristics such as hue on the input image, and determines a region shape obtained by the determination; It is characterized in that a three-dimensional model generation area is set on an input image by comparison with a shape model stored in the database.

Further, in one embodiment of the three-dimensional image generation system of the present invention, the three-dimensional model feature point selection processing means applies a window setting template applicable to a specific model in the feature point selection processing. In addition, the present invention is characterized in that a feature point tracking window setting process is performed on a plurality of frame images.

Further, in one embodiment of the three-dimensional image generation system of the present invention, the coordinate values of the feature point tracking result obtained in the feature point tracking processing by template matching and the initial tracking result coordinate values using the feature point map are used. The difference is compared with a predetermined threshold value. If the difference is larger than the threshold value, the feature point tracking result by the template matching is determined as an error, and the feature point determined as the error is determined as a surrounding feature. It is characterized by having a configuration for executing a correction process for correcting based on points.

Further, in one embodiment of the three-dimensional image generation system of the present invention, the three-dimensional model patch area setting processing means generates a three-dimensional model of an input image based on patch area setting data stored in a database. It is characterized in that it has a configuration for setting a triangular patch area having the feature point set in the area as a vertex.

Further, in one embodiment of the three-dimensional image generation system of the present invention, the three-dimensional image generation system selects an image having the highest resolution for each patch plane constituting the three-dimensional image, and converts the selected image. It is characterized by having a texture mapping means for executing texture mapping as a texture image applied to each patch surface.

Further, a second aspect of the present invention is a three-dimensional image generating method for generating a three-dimensional image by feature point tracking processing on a plurality of input images, the input image being based on a shape model stored in a database. A three-dimensional model generation region extraction processing step of setting a three-dimensional model generation region thereon; and a three-dimensional model for setting a feature point in a three-dimensional model generation region of an input image based on feature point position data stored in a database. A feature point selection processing step; and a three-dimensional model patch area setting processing step of setting a patch area in a three-dimensional model generation area of the input image based on the patch area setting data stored in the database. 3D image generation method.

Further, in one embodiment of the three-dimensional image generation method of the present invention, the three-dimensional model generation region extraction processing step includes the steps of: determining characteristics such as hue on the input image; A three-dimensional model generation area is set on the input image by comparison with a shape model stored in the database.

Further, in one embodiment of the three-dimensional image generating method according to the present invention, in the three-dimensional model feature point selecting step, a window setting template applicable to a specific model is applied in the feature point selecting process. Then, a feature point tracking window setting process is performed on a plurality of frame images.

Further, in one embodiment of the three-dimensional image generation method of the present invention, the coordinate values of the feature point tracking result obtained in the feature point tracking processing by template matching and the initial tracking result coordinate values using the feature point map are used. Difference and
By comparing with a predetermined threshold value, if the difference is larger than the threshold value, the feature point tracking result by template matching is determined as an error, and the feature point determined as the error is determined based on surrounding feature points. And performing a correction process for performing correction.

Further, in one embodiment of the three-dimensional image generating method according to the present invention, the step of setting a three-dimensional model patch area includes generating a three-dimensional model of an input image based on patch area setting data stored in a database. It is characterized in that a triangular patch area having a vertex at a feature point set in the area is set.

Further, in one embodiment of the three-dimensional image generation method according to the present invention, the three-dimensional image generation method selects an image having the highest resolution for each patch plane constituting the three-dimensional image, and selects the selected image. A texture mapping step of executing texture mapping as a texture image to be applied to each patch surface.

Further, a third aspect of the present invention tangibly embodies a computer program for causing a computer system to execute a three-dimensional image generation process for generating a three-dimensional image by a feature point tracking process for a plurality of input images. A program providing medium to be provided, wherein the computer
The program includes a three-dimensional model generation region extraction processing step of setting a three-dimensional model generation region on an input image based on the shape model stored in the database, and an input based on feature point position data stored in the database. A three-dimensional model feature point selection processing step of setting feature points in the three-dimensional model generation area of the image, and setting a patch area in the three-dimensional model generation area of the input image based on the patch area setting data stored in the database And a three-dimensional model patch area setting processing step.

A program providing medium according to a third aspect of the present invention is a medium for providing a computer program in a computer-readable format to a general-purpose computer system capable of executing various program codes. . The form of the medium is not particularly limited, such as a storage medium such as a CD, an FD, and an MO, and a transmission medium such as a network.

Such a program providing medium defines a structural or functional cooperative relationship between the computer program and the providing medium in order to realize a predetermined computer program function on a computer system. Things. In other words, by installing the computer program into the computer system via the providing medium, a cooperative operation is exerted on the computer system, and the same operation and effect as the other aspects of the present invention can be obtained. You can do it.

Still other objects, features and advantages of the present invention are:
This will become apparent from the following detailed description based on the embodiments of the present invention and the accompanying drawings.

[0023]

DESCRIPTION OF THE PREFERRED EMBODIMENTS A three-dimensional image generation system and a three-dimensional image generation method according to the present invention will be described in detail below. First, FIG. 2 shows the configuration of the three-dimensional image generation system of the present invention. The three-dimensional image generation system according to the present invention includes an image photographing unit 2 for photographing an object of interest using a commercially available digital camera, DV cam, digital camera on a personal computer with a camera, camera with a 3D shape photographing (processing) mode, or the like.
01, an image storage unit 202 for storing a plurality of captured images in an internal memory of a personal computer or a recording medium (commercially available memory card, magnetic tape, laser recording disk, etc.) attached to the camera, By means, an image transfer unit 203 capable of transferring an image stored in a recording medium on the camera side to a personal computer equipped with a standard interface such as USB, i.Link, a memory card adapter, or a dedicated device is prepared in advance. The internal parameters of the camera are determined using the image pattern (for example, a checker pattern, etc.), and the image correction unit 204 that removes the geometric distortion of the input image using the parameters, Estimating the three-dimensional shape of the target of interest from the input image after the above-described image correction processing 3D processing unit 20
5. Paste several three-dimensional shapes estimated from image sequences taken from different viewpoints to the same target to create a three-dimensional shape around the target, A three-dimensional shape bonding unit 206 for bonding a three-dimensional shape to create a three-dimensional panoramic shape, a texture mapping unit 2 for mapping a high-quality texture image to estimated three-dimensional shape data (triangular patches)
07, a 3D image display unit 208 that stores 3D shape data with a texture image in a standard VRML file format and executes processing for displaying the data locally or through a network. Hereinafter, details of each unit will be described.

The image capturing unit 201 captures an object with a sensing device such as a commercially available digital camera, a DV cam, a digital camera on a personal computer with a camera, and a camera with a 3D shape capturing (processing) mode.

The image storage unit 202 stores images captured by a camera on a recording medium such as a memory card, a magnetic tape, or a laser disk (registered trademark).
For a personal computer with a camera, it is also possible to directly store the captured image in a recording medium such as an internal memory or a hard disk.

The image transfer unit 203 transfers an image stored in a recording medium on the camera side to a personal computer or a dedicated three-dimensional processing device through a standard interface such as USB, i.Link, or a memory card adapter. The image correction unit 204 captures a known image pattern (for example, a checker pattern or the like) prepared in advance with the same camera, and obtains internal parameters of the camera. And a plurality of captured images (input images)
, An image correction process is performed using the internal parameters of the camera, and a process of removing geometric distortion and the like of the input image is executed.

A method of calibrating the internal parameters of the camera executed by the image correction unit 204 will be described with reference to the flowchart of FIG.

First, in step S301, a characteristic pattern in which a specific shape pattern is set (for example, a checker pattern as shown in the lower right of FIG. 3) is photographed, and is photographed.
(X, y). Next, in step S302, an ideal composite image Im2 (u, u) is obtained from the captured image Im (x, y).
Create v). The relationship between the captured image and the composite image:
An image obtained by transforming the composite image using the projective transformation matrix H and transforming the image by a distortion parameter inside the camera is a captured image. Further, in step S303, an error between the captured image and the composite image is evaluated, and a distortion parameter inside the camera is estimated so that the error is minimized. Note that the error between the captured image and the composite image is obtained by the following equation.

[0029]

E = {Im1 (x, y) −Im2 (u, v)} ²

Here, the camera internal parameters to be estimated include distortion centers cp, cq (distortion center), aspect ratio sv (aspect ratio), and kappa κ (distortion c).
oefficient). The image correction unit 204 performs correction of a captured image using these camera internal parameters.

The three-dimensional processing unit 205 mainly executes feature point extraction, tracking, and patch setting as processing for generating a three-dimensional shape of a target from a plurality of images. The details will be described with reference to FIG.

As shown in FIG. 4, the three-dimensional processing unit 2
05 is a three-dimensional model generation area extraction processing means 2051,
It has three-dimensional model feature point selection and tracking processing means 2052, three-dimensional model patch area setting processing means 2053, and a shape model (knowledge model) database 2054.

Three-dimensional model generation area extraction processing means 205
1 receives a plurality of images from the image transfer unit, and extracts a generation region of a three-dimensional model from the reference image, for example, using the input image of the first frame from the plurality of images as the reference image. The region extraction processing is executed based on data in the shape model (knowledge model) database 2054. The three-dimensional model generation region extraction processing means 2051 discriminates the hue characteristics on the input image, and compares the region shape obtained by the discrimination with the shape model stored in the shape model (knowledge model) database 2054 to determine the input image. A three-dimensional model generation area is set above.

Three-dimensional model generation area extraction processing means 205
When the three-dimensional model generation region is selected by step 1, the three-dimensional model feature point selection and tracking processing means 2052 sets feature points in the selected region, and furthermore, the correspondence of the feature points in images other than the reference image. Search for points, that is, perform feature point tracking. This processing is also executed based on the data in the shape model (knowledge model) database 2054.

3D model generation area extraction processing means 205
A specific example of the three-dimensional model generation region selection processing and the three-dimensional model feature point selection and tracking processing means 2052 by the tracking point processing unit 2052 will be described with reference to FIGS.

FIG. 5 is a diagram showing a practical example of a process of extracting a face portion from a first frame input image as a three-dimensional model generation region and extracting feature points. FIG. 5A shows an input image, and FIG. 5B shows a shape model (knowledge model).
A face shape model stored in the database 2054 is shown. In the system of the present invention, a shape model (knowledge model)
Using the face shape model and the face hue characteristics stored in the database 2054, the three-dimensional model generation area extraction processing means 2051 compares the input image with the model, and as shown in FIG. A region of a natural face is detected as a three-dimensional model generation region.

Further, the three-dimensional model feature point selection / tracking processing means 2052 includes a face feature point position model (FIG. 5) stored in the shape model (knowledge model) database 2054.
Using (e)), a matching process with the face position in the actual input image is executed to determine the position of the feature point of the face (FIG. 5 (f)).

In the system according to the present invention, a face area, which is a three-dimensional model generation area, is detected from a substantially elliptical face model (FIG. 5B), and the face area is set corresponding to the elliptical shape. The feature points are automatically set in the three-dimensional model generation area set in the input image by the feature point position model in which the one feature point is set.

FIG. 6 shows another method of extracting a face area. The three-dimensional model generation region extraction processing means 2051 analyzes the color distribution of the input image of FIG. 6A and detects the region of the face (including the neck) based on the hue characteristics (FIG. 6).
(b)). Therefore, since the hue characteristics of the eyes are different from the hue characteristics of the face portion, the positions of the eyes (centers of the eyes) and the center position between the two surfaces are simultaneously detected. Then, a face region in the image is detected using a face geometric shape model (for example, x0 / y0 in FIG. 6D) prepared in the shape model (knowledge model) database 2054 in advance (FIG. 6D). 6 (e)).

As described above, in the system according to the present invention, the photographed subject is analyzed based on, for example, the hue characteristics of the input image, and the shape model (knowledge model) database 2054 is obtained.
The three-dimensional model generation area is extracted by applying a suitable model to the correlation with the shape model stored in the reference image. Automatically set the feature points.

FIG. 7 is a flowchart showing the above processing. First, in step S701, an image is input, and in step S702, a target model, that is, a shape model corresponding to a target for generating a three-dimensional model is selected from the models stored in the database. Further, in step S703, a region for generating a three-dimensional model is extracted while comparing the hue characteristics and the like of the input image with the selected model. Further, in step S704, feature points are set in the input image based on a feature point position model in which feature point positions have been set corresponding to models stored in the database.

After extracting the feature points, selecting the three-dimensional model feature points,
The tracking processing unit 2052 performs a feature point tracking process on the set feature points.

FIG. 8 shows one method of tracking feature points. As shown in step S801, tracking processing of n feature point positions extracted by the above-described method is performed on the F input images. For example, the feature point Pj in the i-th image fi (x, y)
In order to find a corresponding point on the next (i + 1) -th image fi + 1 (x, y), first, a search area on the image fi + 1 (x, y) is set (step S804). Then, in step S805, the most similar to the feature point Pj in the search area is detected based on a certain evaluation value (similarity evaluation value). The similarity evaluation value here is obtained by a normalized correlation, a least squares method, or the like. Further, the pattern in the area to be searched may have been subjected to a rotation process. The one with the highest evaluation value is selected (step S806) and set as a tracked feature point.

With such processing, tracking is performed across all images, and the results are stored and the tracking processing is completed (S810). The stored result is checked for a feature point tracking error (S811).
In the feature point error correction mode (S812), the coordinate values of those error tracking points are corrected (S815). When all the erroneous tracking points are corrected, the feature point coordinates are stored in the measurement matrix, and the feature point tracking is completed (S817).

Based on the correspondence between a plurality of images obtained as a result of tracking these feature points, the shape of the target and the camera motion are uniquely obtained using the camera projection model by the factor decomposition method shown in FIG.

The feature point selection and tracking processing when a face is used as a three-dimensional image generation model will be described with reference to FIGS. As shown in the upper part (a) of FIG. 9, frames 1 to 12 are input as a plurality of images, and one of the reference images, for example, frame 1 is selected from the frames and a shape model is selected in the image on the reference frame. (Knowledge model) Correlate with the shape model stored in the database 2054, apply an appropriate model, extract a three-dimensional model generation region, and further set a feature point position corresponding to the model. A feature point is automatically set in the reference image according to the position model.

Further, a similar frame is set in another frame image area where an image corresponding to the template set as the reference image is captured (FIG. 9B). Further, a feature point map constituted by a rectangular area window is set in the template of the reference image, a similarity pattern with respect to the feature point of the reference image is searched in another frame image, and a similarity evaluation value is calculated. Compared with the determined threshold, only those satisfying the criteria are identified and stored as feature points. As described above, the initial tracking of the feature points is performed by associating a plurality of images with the feature point map, and an approximate feature point position between the images is determined.

FIG. 10 shows a detailed processing flow of the feature point tracking including a feature point tracking error process which is erroneous due to occlusion or the like. Step S1 in FIG.
In step 001, F images are input, and step S1002
Then, one image is selected as a reference image and, for example, P
Create feature point maps. In step S <b> 1003, a template is set so that the region of the feature point can be identified.

Next, a feature point tracking process (S1006) is performed for each feature point and each frame image using the template set as the reference image. Evaluation of tracking results
As shown in step S1007, an error of the coordinate value is calculated from E = | tracking result (coordinate value) −initial tracking result (coordinate value) | and compared with a predetermined threshold value (S100).
8) Then, only those satisfying the criteria are stored as valid feature point tracking results (S1009).

Specifically, a point matched by, for example, normalized correlation using template matching is set as a tracking result (coordinate value), a tracking result based on the above-described feature point map is set as an initial tracking result (coordinate value), and the difference is calculated. Is calculated, and the calculated difference is compared with a threshold. If the comparison does not satisfy the criterion, in step S1013, a label indicating a feature point identified as an error is set for the tracking result. When these processes are performed on all feature points and image frames, tracking points to which an error label has been added are extracted. In step S1011, the error tracking points are associated with the feature points around the error tracking points. Correct the added tracking result. As shown in FIG. 10 (c), there are feature points that are correctly tracked around the tracking result, and correction of the tracking points with error labels is performed using these.

3D model feature point selection and tracking processing means 2
When the feature point selection processing and the tracking processing according to 052 are completed,
The patch area setting is executed by the three-dimensional model patch area setting means 2053 shown in FIG.

FIG. 11 shows that, when the face is targeted, the feature points (number and position) of the face model prepared in advance are associated with the feature points of the actually detected face, and the face model is defined. An example is shown in which a triangular patch of a face is created based on the relationship between the obtained feature points. 3D model patch area setting means 2
The patch area setting according to 053 is executed based on the patch area data corresponding to the model stored in the shape model (knowledge model) database 2054 in advance. If the model is a face, for example, the patch area data 11 is stored in the shape model (knowledge model) database 2054.
01 is extracted, and a patch area is set in the image based on the patch area data. A patch is set by a triangular area having feature points as vertices. For example, in the patch area data 1101, a triangular area having three vertices of feature points 1, 2, and 5 and a triangular area having three vertices of 1, 5, and 3 are set as patch areas.

FIG. 12 shows an embodiment in which a texture image is mapped on the set patch surface. Using the camera motion and position information obtained by the above-described method, an image (the image with the highest resolution) photographed at the position of the camera most directly opposed to the normal vector of each triangular patch surface is selected, and the image density value Mapping is performed using one texture image obtained by the normalization process and the smoothing process between each patch surface.

FIG. 13 is a flowchart showing the overall processing procedure of the three-dimensional image generation process in the three-dimensional image generation system of the present invention.

In step S1301, the image photographing unit 201, the image storage unit 202, and the image transfer unit 2
03 is an image shooting and transfer process, which is performed in step S13.
02 is an image correction processing step executed by the image correction unit 204 described above.

As described above, the feature point automatic selection of the knowledge base in step S1303 is performed by the three-dimensional model feature point selection and tracking processing means 2052 shown in FIG. 4 on the data stored in the shape model (knowledge model) database. This is a feature point automatic selection process executed based on the above. The tracking processing of the feature window and the saving of the result in step S1304 are also executed by the three-dimensional model feature point selection and tracking processing means 2052 shown in FIG. 4, and a plurality of images are added to the selected feature window. Over to
The same feature window (correspondence of feature windows) in each image is searched. That is, the processing described with reference to FIGS.

Further, in step S1305, it is determined whether or not there is an error in the tracking of the feature point. If there is an error, in step S1306, the tracking error is corrected. This process is the process described above with reference to FIG.

If it is difficult to obtain a completely accurate feature point position even by using the above-described tracking processing, the position (coordinates) of the error tracking point is manually set in the feature point position correction mode. Value) may be modified. In this method, a window corresponding to a template is displayed on a display for each of a reference image and a feature point setting target image by a visual interface, and a user can set an accurate feature point position. By confirming the screen by the user, it is possible to input and correct accurate feature points.

The feature points on the input image are checked, and if all the feature point positions are correctly corrected, a three-dimensional shape estimation process (S1307) is performed, and the three-dimensional shape of the object of interest and the camera movement (position, Posture) and outputs the results (S1308). Here, the three-dimensional shape data is a feature point position (three-dimensional coordinate value) when the feature point of the target of interest on the two-dimensional image is projected back to the three-dimensional space. Next, in steps S1309 and S1310,
A three-dimensional image is generated by executing a triangular patch generation, a texture mapping, and a three-dimensional shape bonding process.

FIG. 14 is a diagram showing a specific example of the target model-based feature point selection and triangular patch creation processing by the above-described three-dimensional image generation system of the present invention. FIG.
(a) shows the input image of the first frame. FIG. 14 (b)
The area of the face detected based on the hue characteristic of the face and the geometric shape model, and the feature point position. Here, the number and position of the feature points are created in advance by a face model. Figure 1
4 (c) shows a triangular patch created using the extracted feature points based on the face shape model.
FIG. 4 is a diagram showing a result of performing texture mapping using the image with the highest resolution on the triangular patch surface and the triangular patch (wire frame).

The present invention has been described in detail with reference to the specific embodiments. However, it is obvious that those skilled in the art can modify or substitute the embodiment without departing from the scope of the present invention. That is, the present invention has been disclosed by way of example, and should not be construed as limiting. In order to determine the gist of the present invention, the claims described at the beginning should be considered.

[0062]

As described above, according to the three-dimensional image generation system and the three-dimensional image generation method of the present invention,
Various effects are provided as follows. (1) By the region extraction method based on the characteristics and the shape model of the target, for example,
It is possible to automatically detect a face area and the like, and the present invention can be applied to face detection and face recognition. In addition, (2) the feature points can be automatically determined for the extracted target region by the target knowledge model, which is more accurate than the general method of determining the feature window based on the feature value evaluation value. It is sure. (3) Correspondence (feature point tracking) between a plurality of images and correction of erroneous correspondence are performed on the extracted feature points, thereby enabling more accurate three-dimensional shape restoration. (4) By automatically creating a triangular patch based on the knowledge model for the extracted feature points of the target, it is possible to represent a wire frame that matches the actual target shape. (5) For each created triangular patch,
The texture image most directly facing the normal direction of these triangular patches is automatically selected, and high-resolution texture mapping including gray-scale value normalization processing and the like can be performed.

[Brief description of the drawings]

FIG. 1 is a diagram illustrating a factor decomposition method in three-dimensional image processing.

FIG. 2 is a diagram showing a system configuration of a three-dimensional image generation system according to the present invention.

FIG. 3 is a diagram illustrating camera parameter calibration processing in the three-dimensional image generation system of the present invention.

FIG. 4 is a block diagram showing a detailed configuration of a three-dimensional processing unit of the three-dimensional image generation system of the present invention.

FIG. 5 is a diagram showing a specific example of a three-dimensional model generation region extraction process of the three-dimensional image generation system of the present invention.

FIG. 6 is a diagram showing a specific example of a three-dimensional model generation region extraction process of the three-dimensional image generation system of the present invention.

FIG. 7 is a diagram showing a processing flow of a three-dimensional processing unit of the three-dimensional image generation system of the present invention.

FIG. 8 is a flowchart illustrating a feature point tracking process in the three-dimensional image generation system of the present invention.

FIG. 9 is a diagram illustrating a feature point tracking process in the three-dimensional image generation system of the present invention.

FIG. 10 is a flowchart illustrating a feature point tracking process in the three-dimensional image generation system of the present invention.

FIG. 11 is a diagram showing a specific example of a three-dimensional model patch area setting process of the three-dimensional image generation system of the present invention.

FIG. 12 is a diagram illustrating a texture mapping process in the three-dimensional image generation system of the present invention.

FIG. 13 shows a diagram of a 3D image generation system according to the present invention;
It is a flowchart explaining a dimension model generation process.

FIG. 14 is a diagram illustrating a specific example of a feature point selection and a triangular patch generation process in the three-dimensional image generation system of the present invention.

[Explanation of symbols]

 Reference Signs List 201 Image photographing unit 202 Image storage unit 203 Image transfer unit 204 Image correction unit 205 3D processing unit 206 3D shape bonding unit 207 Texture mapping unit 208 3D image display unit 2051 3D model generation area extraction processing means 2052 3D model Feature point selection and tracking processing means 2053 3D model patch use area setting processing means 2054 3D model (knowledge model) database

 ──────────────────────────────────────────────────続 き Continued on the front page F term (reference) 5B057 CA01 CA13 CB01 CB13 CD20 DA07 DB03 DB06 DC05 DC09 DC25 DC32 5B080 AA14 AA18 BA07 FA02 GA22 5C061 AA21 AB03 AB08

Claims (13)

[Claims] 1. A three-dimensional image generation system for generating a three-dimensional image by feature point tracking processing for a plurality of input images, wherein a three-dimensional model generation area is set on the input image based on a shape model stored in a database. Three-dimensional model generation region extraction processing means for performing the following processing; three-dimensional model characteristic point selection processing means for setting a characteristic point in the three-dimensional model generation area of the input image based on the characteristic point position data stored in the database; A three-dimensional model patch area setting processing means for setting a patch area in a three-dimensional model generation area of an input image based on stored patch area setting data. 2. The three-dimensional model generation region extraction processing means determines characteristics such as hue on an input image, and compares an area shape obtained by the determination with a shape model stored in the database. The three-dimensional image generation system according to claim 1, wherein a three-dimensional model generation area is set on the image. 3. The feature point selection processing means according to claim 2, wherein said feature point selection processing includes applying a window setting template applicable to a specific model to a feature point tracking process for a plurality of frame images. The three-dimensional image generation system according to claim 1, further comprising a configuration for executing a window setting process. 4. The three-dimensional image generating system according to claim 1, wherein a difference between a coordinate value of a feature point tracking result obtained in a feature point tracking process by template matching and an initial tracking result coordinate value using a feature point map is determined in advance. If the difference is larger than the threshold value, the feature point tracking result by template matching is determined to be an error, and the feature point determined to be the error is corrected based on surrounding feature points. The three-dimensional image generation system according to claim 1, further comprising a configuration for executing a correction process. 5. A three-dimensional model patch area setting processing means, based on patch area setting data stored in a database, a triangular patch area having feature points set in a three-dimensional model generation area of an input image as vertices. The three-dimensional image generation system according to claim 1, further comprising: 6. The three-dimensional image generating system selects an image having the highest resolution for each patch plane constituting the three-dimensional image, and executes texture mapping as a texture image to be applied to each patch plane. 2. The apparatus according to claim 1, further comprising texture mapping means.
3. The three-dimensional image generation system according to 1. 7. A three-dimensional image generation method for generating a three-dimensional image by feature point tracking processing for a plurality of input images, wherein a three-dimensional model generation area is set on the input image based on a shape model stored in a database. A three-dimensional model generation area extraction processing step of setting a feature point in a three-dimensional model generation area of an input image based on feature point position data stored in a database; A three-dimensional model patch area setting processing step of setting a patch area in a three-dimensional model generation area of the input image based on the stored patch area setting data. 8. The three-dimensional model generation region extraction processing step includes determining characteristics such as hue on an input image, and comparing the region shape obtained by the determination with a shape model stored in the database. The three-dimensional image generation method according to claim 7, wherein a three-dimensional model generation area is set on the image. 9. The three-dimensional model feature point selecting step includes: in the feature point selecting process, applying a window setting template applicable to a specific model to a feature point tracking process for a plurality of frame images. The three-dimensional image generation method according to claim 7, wherein a window setting process is performed. 10. The three-dimensional image generation method according to claim 1, wherein a difference between a coordinate value of a feature point tracking result obtained in a feature point tracking process by template matching and an initial tracking result coordinate value using a feature point map is determined in advance. If the difference is larger than the threshold value, the feature point tracking result by template matching is determined to be an error, and the feature point determined to be the error is corrected based on surrounding feature points. The three-dimensional image generation method according to claim 7, wherein a correction process is performed. 11. The three-dimensional model patch area setting processing step comprises the steps of: based on patch area setting data stored in a database, a triangular patch area having a feature point set in a three-dimensional model generation area of an input image as a vertex The three-dimensional image generation method according to claim 7, wherein 12. The three-dimensional image generating method according to claim 1, wherein an image having the highest resolution is selected for each patch surface forming the three-dimensional image, and texture mapping is executed as a texture image applied to each patch surface. The method according to claim 7, further comprising a texture mapping step. 13. A program providing medium tangibly providing a computer program for causing a computer system to execute a three-dimensional image generation process for generating a three-dimensional image by a feature point tracking process for a plurality of input images, The computer program includes: a three-dimensional model generation region extraction processing step of setting a three-dimensional model generation region on an input image based on a shape model stored in a database; and a feature point position data stored in the database. A three-dimensional model feature point selecting step of setting feature points in the three-dimensional model generation area of the input image; and a patch area in the three-dimensional model generation area of the input image based on the patch area setting data stored in the database. 3D model patch area setting processing step of setting Program providing medium characterized.