JP2000348213A - Three-dimensional image generating device, three- dimensional image generating and display device, and method thereof and recording medium - Google Patents

Abstract

PROBLEM TO BE SOLVED: To improve the appearance of a display image by generating data showing how color data of a three-dimensional model are seen in multiple directions by using three-dimensional shape model data of an object and pieces of image data as to the object. SOLUTION: Image data is inputted from an image input part 10 and a solid shape input part 12 performs three-dimensional modeling based upon pieces of image data as to the object photographed from multiple positions to obtain three-dimensional shape data of the object. The image data is supplied to a color information conversion part 14 as well. This color information conversion part 14 generates color information related to respective positions of the three- dimensional shape data. The three-dimensional shape data and color information related thereto are stored in a data storage part 16. The image data read out of the data storage part 16 is supplied to a video generation part, where video data of the object at various viewpoints specified by a user are generated. On the basis of the video data, a video display part displays video of the object.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a three-dimensional image generating technique for generating three-dimensional data of an object from a plurality of pieces of image data of the object, and more particularly to a three-dimensional image generating apparatus, a three-dimensional image generating and displaying apparatus, and a method thereof. And a recording medium.

[0002]

2. Description of the Related Art Conventionally, various methods for generating three-dimensional images have been known as (1) to (3). (1) Using multiple viewpoint images as they are, such as QuickTime VR SEChen: Quick Time VR-An Imaged-based Approach
to Virtual Environment Navigation, Proc.SIGGRAPH
In the method called QuickTimeVR introduced in 95, pp. 29-38, one object is photographed from a very large number of viewpoints, and during playback, according to the user's viewpoint specification , The corresponding image of the viewpoint is displayed.
This method has the advantage that images from various viewpoints can be redisplayed without obtaining the shape of the object, but has the disadvantage that images from viewpoints other than the imaging viewpoint cannot be redisplayed.

Therefore, when it is necessary to redisplay an image at a very fine viewpoint step, it is necessary to shoot from a very large number of viewpoints, and as a result, the amount of data becomes extremely large. (2) Transform multi-view images into ray space Fujii, Harashima: 3D image coding based on structure extraction, Image coding symposium (PCSJ94), pp.23-24, M.Levoy, PH
anrahan: Light Field Rendering, Proc.SIGGRAPH 96,
pp. 31-42, and SJGortler et al .: The Lumigraph,
The ray space method introduced by Proc. SIGGRAPH 96, pp. 43-54, etc. captures an object from many viewpoints. Then, for the target, a reference plane at an arbitrary arbitrary position is set, and for each pixel on the reference plane, a θφ space of the viewing angle is set, and this space is filled with a color corresponding to the θφ of each viewpoint. . At the time of reproduction, for the viewpoint designated by the user, the θφ space data is retrieved from the reference plane according to the viewpoint, and an image is generated and displayed. This method requires that the viewpoint information be extremely accurate.

[0004] Further, similarly to QuickTime VR, when it is necessary to redisplay an image at a very fine viewpoint step, there is a problem that the amount of image data to be held becomes very large. The image data can be compressed to a high degree so that the size of the image data when recorded on a medium such as a hard disk can be reduced to some extent. It is necessary to develop in a state, and it is inevitable that the data amount at this time becomes very large. Further, when the compressed data is stored in the memory of the computer, it is necessary to perform a decompression process at the time of display, which makes high-speed display difficult. (3) A method in which a three-dimensional model of an object is once generated from a multi-view image and color information of each part is held as a texture image. have. Then, the texture (color) information is extracted by, for example, projecting this on the current image. At the time of reproduction, an image is generated and presented by creating a projection image of each polygon for a viewpoint designated by the user and mapping texture information of each polygon to the projection image. According to the three-dimensional modeling, there is an advantage that the data amount can be considerably small and an image can be reproduced from an arbitrary viewpoint.

[0005] Therefore, various techniques have been proposed for obtaining a plurality of images from around the object with a camera and performing three-dimensional modeling from the images. For example, a shape estimation by a stereo method or a silhouette method is typical. In addition, a method of irradiating a target with a laser beam or the like, observing the target with a camera, and obtaining the shape of the target by the principle of triangulation is also widely used.

[0006]

However, in the three-dimensional modeling by the stereo method, when the surface of the object has no feature, the three-dimensional shape model cannot always be generated accurately and the shape accuracy cannot be sufficiently high. There is. Further, the silhouette method has a problem that a concave shape that does not appear in the contour cannot be modeled. Further, even with the method of irradiating a laser beam, there are cases where the shape accuracy is insufficient due to the problem of self-concealment and the like.

Further, since the texture information is obtained by projecting a polygon representing the shape of the object onto the photographed image, if the shape of the object is incorrect in the first place, the texture information becomes extremely inaccurate. .

Further, since the texture information is obtained from an image obtained by imaging the object, the color information of each portion may be inaccurate depending on the illumination. For these reasons, even in the method (3), when the object is displayed from various viewpoints, there is often a problem that a suitable shape / color cannot be reproduced.

SUMMARY OF THE INVENTION Accordingly, an object of the present invention is to provide an image generation, display, and data holding method capable of improving the appearance of a display image in a method of performing three-dimensional modeling.

It is another object of the present invention to perform high-quality display at high speed while suppressing the amount of memory used in a display device during display.

[0011]

According to the present invention, there is provided a three-dimensional image generating apparatus for generating three-dimensional image data of an object from a plurality of pieces of image data of the object. Model data reading means for reading the three-dimensional shape model data, using a plurality of image data for the three-dimensional shape model data and the object,
A color data generating unit configured to generate color data on the surface of the three-dimensional model as data viewed from a plurality of directions.

In the apparatus, the color data generating means divides a surface of the three-dimensional model into a plurality of elements,
Color information about at least one of these elements
By acquiring from a plurality of pieces of image data of the object that is the original image, it is obtained as data viewed from a plurality of directions, and further, by re-arranging these color information in a region different from the original image, It is characterized by generating three-dimensional image data.

In the above-mentioned apparatus, the plurality of elements are triangular polygons, and when rearranging the color information in the another area, the elements are converted into right triangles and the color information is rearranged.

In the above apparatus, the plurality of elements are triangular polygons, and when rearranged in the another area, the elements are converted into right-angled triangles whose two short side lengths are powers of two. The color information is rearranged.

In the above-mentioned apparatus, the plurality of elements are quadrangular polygons, and when rearranging in the another area, the plurality of elements are converted into rectangles and the color information is rearranged.

In the above apparatus, when the color information is rearranged in the another area, the color information is rearranged by rearranging the elements based on the length of a side of the element.

In the above-mentioned device, the size of the plurality of elements may have a certain limit.

In the above apparatus, the size of the plurality of elements may be given a certain limit, and elements exceeding the limit may be subdivided so as to satisfy the limit.

In the above-mentioned apparatus, the size of the plurality of elements is given a certain limit, and an element exceeding the limit is reduced to satisfy the limit.

In the apparatus, the color data generating means divides a surface of the three-dimensional model into a plurality of elements,
For all the elements, the color data on the surface of the three-dimensional model is generated as data viewed from a single direction, and a part of these elements is further selected from a plurality of divided elements, and the elements of this part are selected. Is characterized in that color data on the surface of the three-dimensional model is also generated as data viewed from a plurality of directions.

In the above apparatus, the selection of the part of the element is performed based on the similarity of the color information when the element is viewed from a plurality of directions.

In the above apparatus, the similarity is characterized in that color information when the element is viewed from a plurality of directions is evaluated by moving the color information in parallel within a certain range.

In the apparatus, the similarity is evaluated by correcting the brightness of color information when the element is viewed from a plurality of directions.

In the above apparatus, the color data generated as the data viewed from the plurality of directions is associated with a viewing direction, that is, eye-gaze information.

The apparatus further comprises a similar color data extracting means for extracting color data having high similarity among color data generated as the data viewed from the plurality of directions,
A representative color data determining means for determining a representative from these similar color data, and a reference relation information providing means for giving reference relation information for referencing the representative color data to the color data other than the representative, comprising: Features.

In the above-mentioned apparatus, the color data generated in association with the line-of-sight information is generated by grouping for each same line of sight.

In the above apparatus, the color data generated by grouping the same line of sight is compressed for each group.

In the above-mentioned apparatus, the color data generated as the data viewed from the plurality of directions is hierarchically generated based on the similarity to the color data generated as the data viewed from the single direction. It is characterized by that.

Further, the three-dimensional image generating and displaying apparatus according to the present invention is a three-dimensional image generating and displaying apparatus for generating and displaying three-dimensional image data on an object, wherein: a means for obtaining a virtual viewpoint position for display; And color data selection means for selecting different color data according to

In the apparatus, the color data is generated as data obtained by viewing the surface of the target three-dimensional model from a plurality of directions.

In the above apparatus, the surface of the three-dimensional model to be displayed is divided into a plurality of elements, and for at least one of the elements, the color data is obtained by viewing the surface of the three-dimensional model from a plurality of directions. It is characterized by being generated as data.

In the above-mentioned apparatus, the plurality of divided elements are all assigned color data generated as data viewed from a single direction, and some of the elements are divided from a plurality of directions. The color data generated as the viewed data is also assigned, and the color data selection means is configured to move the virtual viewpoint for display or the display object for each of the plurality of divided elements. If the movement amount of the object is equal to or more than a certain value, select the color data generated as data viewed from a single direction for all the elements, and if the movement amount is equal to or less than a certain value, select the part. Is characterized by being appropriately selected from color data generated as data viewed from a plurality of directions.

In the above apparatus, the color data selecting means may select a plurality of pieces of color data for each element, and integrate them to generate color data for the element.

In the apparatus, the color data is generated in association with a viewing direction, that is, line-of-sight information, is further grouped and compressed for each line-of-sight, and according to a virtual viewpoint for the display, The color data of at least one group corresponding to appropriate line-of-sight information is developed and displayed.

The three-dimensional image generating method of the present invention is a three-dimensional image generating method for generating three-dimensional image data of an object from a plurality of pieces of image data of the object. Using the read model data reading step and the three-dimensional shape model data and a plurality of pieces of image data on the object, color data on the surface of the three-dimensional model is converted into color data as data viewed from a plurality of directions. And a generating step.

In the method, the step of generating the color data may include dividing the surface of the three-dimensional model into a plurality of elements, and converting the color information of at least one of these elements into an original image of the object. The three-dimensional image data is obtained by acquiring as data viewed from a plurality of directions by taking in from a plurality of pieces of image data, and further relocating these color information to an area different from the original image. It is characterized by.

[0037] In the above method, the plurality of elements are triangular polygons, and when rearranging in the another area, the color information is converted to a right triangle and the color information is rearranged.

In the above method, the plurality of elements are triangular polygons, and when rearranged in the another area, are converted into right-angled triangles whose two short side lengths are powers of two. The color information is rearranged.

In the above method, the plurality of elements are quadrangular polygons, and when relocating to the another area, the elements are converted into rectangles and the color information is relocated.

In the above method, when the color information is rearranged in the another area, the color information is rearranged by rearranging the elements based on the length of a side of the element.

In the above method, the size of the plurality of elements may have a certain limit.

The above method is characterized in that a certain limit is imposed on the size of the plurality of elements, and an element exceeding the limit is re-divided so as to satisfy the limit.

The method is characterized in that a certain limit is imposed on the size of the plurality of elements, and an element exceeding the limit is reduced to satisfy the limit.

In the above method, the step of generating the color data includes dividing the surface of the three-dimensional model into a plurality of elements, and converting the color data of the surface of the three-dimensional model from all directions into a single direction. In addition to the data generated as viewed data, a part of these elements is further selected from the plurality of divided elements, and the color data on the surface of the three-dimensional model is viewed from a plurality of directions for some of these elements. Is also generated.

[0045] In the above method, the selection of the element of the part is performed based on the similarity of the color information when the element is viewed from a plurality of directions.

In the above method, the similarity is characterized in that color information when the element is viewed from a plurality of directions is evaluated by moving the color information in parallel within a certain range.

In the above method, the similarity is evaluated by correcting the brightness of the color information when the element is viewed from a plurality of directions.

In the above method, the color data generated as the data viewed from the plurality of directions is characterized in that it is associated with the viewing direction, that is, the line-of-sight information.

The method further comprises the step of extracting similar color data from among the color data generated as the data viewed from the plurality of directions and extracting similar color data. It is characterized by comprising a step of determining representative color data and a step of giving reference relationship information to color data other than the representative to have reference relationship information for referring to the representative color data.

In the above method, the color data generated in association with the line-of-sight information is generated by grouping for each same line of sight.

[0051] In the above method, the color data generated by grouping for the same line of sight is compressed for each group.

In the above method, the color data generated as the data viewed from the plurality of directions is hierarchically generated based on the similarity to the color data generated as the data viewed from the single direction. It is characterized by that.

A three-dimensional image generation and display method according to the present invention is a three-dimensional image generation and display method for generating and displaying three-dimensional image data of an object, wherein a step of obtaining a virtual viewpoint position for display is provided. And selecting different color data.

In the display method, the color data is generated as data obtained by viewing the surface of the target three-dimensional model from a plurality of directions.

In the display method, the surface of the three-dimensional model to be displayed is divided into a plurality of elements, and for at least one of the elements, the color data is obtained by viewing the surface of the three-dimensional model from a plurality of directions. Characterized by being generated as data.

In the display method, the plurality of divided elements are all assigned color data generated as data viewed from a single direction. The color data generated as data viewed from is also assigned, and the step of selecting the color data includes, for each of the plurality of divided elements, a moving amount of the virtual viewpoint for the display, or When the moving amount of the display object is equal to or more than a certain value, color data generated as data viewed from a single direction for all elements is selected, and when the moving amount is equal to or less than a certain value, The above-mentioned some elements are appropriately selected from color data generated as data viewed from a plurality of directions.

In the display method, the step of selecting the color data is characterized in that a plurality of color data are selected for each element, and these are integrated to generate color data for the element.

In the display method, the color data is generated in association with a viewing direction, that is, line-of-sight information, is further grouped for each same line-of-sight, and is compressed, according to a virtual viewpoint for the display. The color data of at least one group corresponding to appropriate line-of-sight information is developed and displayed.

A recording medium according to the present invention is a recording medium readable by an information processing apparatus for storing a program for generating three-dimensional image data of an object from a plurality of pieces of image data of the object. For the information processing device, read the three-dimensional shape model data of the object, using the three-dimensional shape model data and a plurality of image data of the object, color data on the surface of the three-dimensional model Is generated as data viewed from a plurality of directions.

In the recording medium, the surface of the three-dimensional model is divided into a plurality of elements, and color information on at least one of these elements is fetched from a plurality of pieces of image data on the object as an original image. Thus, the three-dimensional image data is obtained by acquiring the data as data viewed from a plurality of directions, and relocating the color information to a region different from the original image.

[0061] In the above-mentioned recording medium, the plurality of elements are triangular polygons, and when rearranging in the another area, the color information is converted to a right triangle and rearranged.

In the recording medium, the plurality of elements are triangular polygons, and when rearranging in the another area, the elements are converted into a right-angled triangle whose short side length is a power of two. And rearranging the color information.

[0063] In the above-mentioned recording medium, the plurality of elements are quadrangular polygons, and when relocating to the another area, the elements are converted into rectangles and the color information is relocated.

In the recording medium, when the color information is rearranged in the another area, the color information is rearranged by rearranging the elements based on the length of a side of the element.

In the above-mentioned recording medium, it is characterized in that the size of the plurality of elements has a certain limit.

The recording medium is characterized in that the size of the plurality of elements is given a certain limit, and elements exceeding the limit are subdivided so as to satisfy the limit.

The recording medium is characterized in that the size of the plurality of elements is given a certain limit, and elements exceeding the limit are reduced to satisfy the limit.

In the recording medium, the surface of the three-dimensional model is divided into a plurality of elements, and for all the elements, color data on the surface of the three-dimensional model is generated as data viewed from a single direction. And selecting a part of these elements from the plurality of divided elements, and generating color data on the surface of the three-dimensional model as data viewed from a plurality of directions with respect to some of these elements. I do.

[0069] In the above-mentioned recording medium, the selection of the element of the part is performed based on the similarity of color information when the element is viewed from a plurality of directions.

In the above-mentioned recording medium, the similarity is characterized in that color information when the element is viewed from a plurality of directions is evaluated by moving the color information in parallel within a certain range.

In the above-mentioned recording medium, the similarity is evaluated by correcting the brightness of the color information when the element is viewed from a plurality of directions.

In the recording medium, the color data generated as the data viewed from the plurality of directions includes a viewing direction,
That is, it is characterized by being associated with the line-of-sight information.

The recording medium further extracts color data having a high similarity among the color data generated as the data viewed from the plurality of directions, determines a representative from the similar color data, and determines a representative other than the representative. The color data is characterized by having reference relation information for referring to the representative color data.

In the recording medium, the color data generated in association with the line-of-sight information is generated by grouping for each same line of sight.

In the above-mentioned recording medium, the color data generated by grouping for each line of sight is compressed for each group.

[0076] In the recording medium, the color data generated as the data viewed from the plurality of directions is hierarchized based on the similarity to the color data generated as the data viewed from the single direction. It is characterized by being performed.

Further, a recording medium of the present invention is a recording medium readable by an information processing apparatus for storing a three-dimensional image generation and display program for generating and displaying three-dimensional image data of an object. According to another aspect of the present invention, different color data is selected for the information processing apparatus in accordance with a designated virtual viewpoint position for display.

In the above-mentioned recording medium, the color data is generated as data obtained by viewing the surface of the target three-dimensional model from a plurality of directions.

In the recording medium, the surface of the three-dimensional model to be displayed is divided into a plurality of elements, and for at least one of these, the color data is obtained by viewing the surface of the three-dimensional model from a plurality of directions. Characterized by being generated as data.

In the recording medium, color data generated as data viewed from a single direction is assigned to all of the plurality of divided elements, and some of the elements are viewed from a plurality of directions. The color data generated as the viewed data is also assigned, and the selection of the color data is based on the moving amount of the virtual viewpoint for the display or the display object for each of the plurality of divided elements. If the amount of movement is equal to or more than a certain value, color data generated as data viewed from a single direction for all elements is selected. Is characterized by appropriately selecting from color data generated as data viewed from a plurality of directions.

In the recording medium, the selection of the color data is characterized in that a plurality of color data is selected for each element, and these are integrated to generate color data for the element.

In the recording medium, the color data is generated in association with a viewing direction, that is, line-of-sight information, is further grouped for each same line-of-sight, and is compressed, according to the virtual viewpoint for the display. The color data of at least one group corresponding to appropriate line-of-sight information is developed and displayed.

[0083]

Embodiments of the present invention (hereinafter referred to as "embodiments") will be described below with reference to the drawings.

FIGS. 1, 2 and 3 are block diagrams showing the configurations of an image data generating apparatus and a reproducing apparatus according to one embodiment.

First, the data generator will be described. Referring to FIG. 1, image input unit 10 captures an image of an object and generates image data. The image input unit 10 includes a plurality of color CCD cameras (hereinafter, referred to as “cameras 44”) and the like, and obtains color image data of an object from a plurality of directions.

Further, as shown in FIG. 4, the camera 44 or the object 46 is moved (for example, put on the turntable 40 and rotated), so that the camera 44 or the object
6 may be obtained.

Further, it is also preferable to have means 42 for moving the camera 44 in the vertical direction so that the object 46 can be photographed from an arbitrary bird's-eye view angle. Hereinafter, the image input in this manner is referred to as an “object image” or an “input image”.

Of course, the image input unit 10 may input an image to the apparatus by reading an object image captured by another apparatus and recorded on some medium.

The image data obtained by the image input unit 10 is input to the three-dimensional shape input unit 12.

The three-dimensional shape input unit 12 performs three-dimensional modeling based on a plurality of image data of the object photographed from a plurality of positions to obtain three-dimensional shape data of the object. For the three-dimensional modeling, for example, a method described in Japanese Patent Application Laid-Open No. H10-124704 is used.

The three-dimensional shape input unit 12 may of course input the three-dimensional shape data of the object obtained by another device, or may read the shape data recorded on some medium to read the three-dimensional shape data. The three-dimensional shape data may be input to the device. FIG. 2 is a block diagram showing the configuration of the device in such a case.

The image data is also supplied to the color information converter 14. The color information conversion unit 14 generates color information in association with each position of the three-dimensional shape data.

Then, the three-dimensional shape data and the color information associated therewith are supplied to the data storage unit 16,
It is stored here. Various memories are used for the data storage unit 16. Thus, the image input unit 10
It is possible to obtain three-dimensional shape data and color information based on a plurality of pieces of image data on the object obtained in.

Next, the data reproducing section will be described. Referring to FIG. 3, the image data read from data storage unit 16 is supplied to video generation unit 30, where video data of an object at various viewpoints according to a user's designation or the like is generated. Then, the video display unit 32 displays the video of the target based on the video data. Various display devices such as a CRT and an LCD are used for the video display unit 32.

The three-dimensional shape input unit 12 and the color information conversion unit 1
4. The data storage unit 16 and the video generation unit 30 are configured by a personal computer or the like.

Here, a specific example of the operation of the three-dimensional shape input unit 12 will be described. Here, a case in which the three-dimensional shape input unit 12 obtains the three-dimensional shape of the object will be described.

First, a plurality of image data are input from the image input unit 10. Then, a three-dimensional shape is estimated from the plurality of image data.

Hereinafter, a three-dimensional shape estimation method using the silhouette method will be described as an example.

Here, as shown in FIG.
It is assumed that the object 46 is placed on the camera 0 and the image is taken while rotating the turntable 40. The camera position with respect to the turntable 40 is obtained in advance.

Next, in FIG. 5, a process of extracting a silhouette of a target object from a plurality of object images input from the image input unit 10 is performed using the silhouette method (S50).

For the silhouette extraction, for example, a chroma key method using a blue background or a background difference method obtained by inputting background image data in advance and performing a difference process between the object image data and the background image data is used. .

Subsequently, a shape reconstruction process for reconstructing a three-dimensional shape of the object from a plurality of silhouette images is performed (S52). Here, first, a three-dimensional space where an object is predicted to be present is expressed as a voxel space. The voxel space expresses a three-dimensional space by a collection of very small solid elements (voxels).

The space in which the object exists is located on the turntable 4.
It can be predicted as an area above zero and within the field of view of the camera. Then, each voxel is projected on each silhouette image, and only those projected on all object silhouette images (or on a certain number or more of silhouette images) are extracted.

The set of voxels thus extracted represents the three-dimensional shape of the object.

If necessary, surface voxels are extracted from the set of voxels, and the voxels are converted into a polygon representation by integrating them (S54). Thereby, three-dimensional polygon data is obtained.

Although the three-dimensional shape modeling has been described assuming that the silhouette method is used among the methods performed from a plurality of images of the object, any other method, for example, the stereo method It goes without saying that a method of irradiating a laser beam, a method of irradiating a slit light, a moire method, or the like may be used.

Next, the processing performed by the color information converter 14 will be described.

Here, it is assumed that surface information of a three-dimensional shape is represented by a plurality of elements, and a method of generating color data for these will be described. Hereinafter, a case where the surface information of the three-dimensional shape is represented by a set of polygons will be described as an example. Of course, the surface information of the three-dimensional shape may be represented by a set of voxels, or may be represented by a parametric free-form surface such as NURBS.

In the following, "color information" and "texture" are used interchangeably.

In this case, each polygon is classified into a polygon having a single texture (fixed texture polygon) and a polygon having a plurality of textures (variable texture polygon). One of a plurality of textures of the variable texture polygon is regarded as a representative texture of the polygon, and this is called a “main texture”. Other textures are called "sub-textures".

On the other hand, the fixed texture polygon has only one texture, which is also the main texture.
That is, all polygons have one main texture, and some polygons also have sub-textures.

FIG. 6 is a flowchart showing the generation of color information. Referring to FIG. 6, a reference viewpoint of each polygon is determined (S60).

In this method, for example, each polygon is projected on its own object image data, and the photographing viewpoint of the object image having the largest projected area is obtained, and this is set as a reference viewpoint. FIG. 7 shows a conceptual diagram of the determination of the reference viewpoint.

In FIG. 7, images 72 from a plurality of camera viewpoints are shown.
In contrast to (a), (b), and (c), the projected area is the largest in the image of 72 (b). Therefore, it indicates that the camera viewpoint capturing 72 (b) is the reference viewpoint.

As another method, a normal line 78 of each polygon is obtained, and this is compared with a camera line-of-sight vector 79 that has taken each object image data. May be used as the reference viewpoint.

Next, main texture information of each polygon is obtained from the object image at the reference viewpoint (S61).

This can be obtained by projecting each polygon onto the object image data from each reference viewpoint, and taking in texture information of the projected portion.

Subsequently, in step S63, it is checked whether or not each polygon is visible from all viewpoints. If it is visible, in step S64, a polygon is projected on the object image data at this viewpoint, and the similarity between the texture information of the projected portion and the main texture information is obtained (S65).

Then, in step S66, for polygons determined to have insufficient similarity (determined to be smaller than a predetermined threshold), the texture information of the projection unit is fetched as sub-texture information (S6).
7). At this time, naturally, viewpoint information, that is, photographing camera viewpoint information for the polygon is also recorded. This line-of-sight information will be referred to as a “texture line-of-sight vector”.
Here, the texture line-of-sight vector is preferably described as camera line-of-sight direction information (θφ) for each polygon 160 as shown in FIG.

The above processing is performed for all polygons.

Therefore, when the above processing is completed, polygons having no sub-texture information (those having only main texture information) become fixed texture polygons, and at least one sub-texture is added to the main texture information. Those having information become variable texture polygons.

In the evaluation of the similarity, the size of the texture information is adjusted by affine transformation or projection transformation, and then the cross-correlation or the sum of squared errors is determined for all corresponding pixels. Or by calculating the sum of the absolute values of the differences. In the case of cross-correlation, the larger the value is, the higher the similarity is evaluated. On the other hand, the smaller the sum of the square errors and the sum of the absolute values of the differences, the higher the similarity is evaluated. Then, a threshold is applied to the similarity evaluation value to determine whether the similarity is sufficient or insufficient.

Here, if the obtained three-dimensional shape model is extremely accurate, and the object image is in an environment where the surface of the object is a sufficiently diffuse surface and there is no influence of illumination at all, In this case, all the polygons are classified as fixed texture polygons.

However, in general, although a considerable number of polygons are classified as fixed texture polygons, some polygons are classified as variable texture polygons.

Next, a data holding method will be described. Here, an example in which surface information of a three-dimensional shape represented by a plurality of elements is represented by a set of polygons will be described.

The three-dimensional surface model data generally includes shape data and texture data.

As shown in FIG. 8, the shape data represents three-dimensional position information of the polygon 80. On the other hand, the texture information includes correspondence information between vertices and texture image coordinates and texture image information.

More specifically, as shown in FIG.
2, the image information corresponding to the two-dimensional position in the texture image coordinates and the area 94 surrounded by the two-dimensional position.

Now, consider a case where an object image is input from 8,100 viewpoints in increments of 2 for the entire circumference in the θ direction and in increments of 2 for 1/4 circumference in the φ direction, and three-dimensionally modeled by the above-described method. . Here, it is assumed that the target model is represented by 10,000 polygons (triangles), of which 8,000 fixed texture polygons and 2,000 variable texture polygons are averaged at each viewpoint.

First, for the texture image data of the texture data, the data holding method and the data amount are estimated.

Now, it is assumed that the average texture image information amount per polygon is 500 bytes (uncompressed).

In this case, for 8,000 fixed texture polygons, the texture image information is 4 MB. On the other hand, for 2,000 variable texture polygons, 8.1
It will be huge with GB. Hereinafter, a method for further reducing the data amount of the variable texture polygon will be described. 1: Data amount reduction by cross-reference between sub-textures A variable texture polygon needs to be held when the similarity between the main texture and the sub-texture becomes insufficient.

However, the similarity between the sub-textures may be sufficiently high. In such a case, instead of having all the sub-textures, the sub-textures having a high similarity should have one representative, and only the reference relation information (link information) to the representative should be given to the rest. Thus, it is possible to significantly reduce the amount of data. FIG.
0 shows a data structure according to this method. Here, in the sub-texture information corresponding to the camera line-of-sight vector θ = 20, φ = 6, the polygon of id11 does not have the actual state of the texture information. Instead, it has link information to sub-texture information corresponding to the camera line-of-sight vector θ = 20, φ = 10.

Therefore, for the polygon of id11,
When accessing the sub-texture information corresponding to the camera line-of-sight vector θ = 20, φ = 6, the sub-texture information corresponding to the camera line-of-sight vector θ = 20, φ = 10 is actually accessed for the polygon corresponding to id11. . 2: Data amount reduction by position shift compensation When the three-dimensional information of the polygon is incorrect, the projection position of the polygon on the object image shifts with the change of the camera viewpoint. This is one of the causes of the low similarity between the main texture and the sub-texture. However, this shift can be corrected by moving the projection position on the object image in parallel.

Therefore, when the similarity between the main texture and the sub-texture is low, the projection position of the sub-texture is shifted up, down, left and right within a range of ± several pixels, and the similarity is checked again. In this recheck, if even one having a high similarity is detected, it is sufficient to have only the position shift information instead of having the sub-texture.

At the time of redisplay, the main texture image
The texture may be mapped after correcting the position shift.

FIG. 11 shows an example. Here, it is determined that the similarity between the main texture 110 and the sub-texture 112 is low, but they are slightly shifted left, right, up, and down on the object image 114. Here, the area surrounded by the solid line corresponds to the sub-texture, and the area surrounded by the broken line corresponds to the main texture.

Therefore, in such a case, the sub-texture is shifted up, down, left, and right, and the best similarity is calculated therein, whereby the similarity is determined to be high.

This position shift compensation is performed not only between the main texture and the sub-texture but also between the sub-textures. In this case, the reference destination id information may be provided together with the position shift amount.

When the position shift compensation is performed as described above, the reference destination texture image needs to be an area obtained by adding a shift amount around the polygon, and the data amount slightly increases as compared with the case where the reference is not referred. However, the merit of not requiring the texture image of the reference source is much greater, and the data amount can be greatly reduced as a whole. 3: Data amount reduction by lightness shift compensation When the photographing viewpoint changes and the lighting condition changes, the color information of the polygon changes. In this case, it is determined that the similarity between the main texture and the sub-texture is low.

In this case, instead of having the sub-texture, only the brightness shift difference from the main texture may be provided. At the time of redisplay, the texture may be mapped by writing the correction for the lightness shift in the main texture image.

As described above, if the data amount can be reduced by half by each reduction, the data amount becomes 1/8 in total. Therefore, 8.1 GB can be reduced to about 1 GB.

These images can generally be compressed to about 1/20 by a compression technique such as JPEG. Therefore, it is predicted that all the textures can be expressed in about 50 MB.

In the above description, the case where the polygon is a triangle has been described. In the case where the polygon is a quadrangle, the right triangle may be replaced with a rectangle in the above description.

Next, the data amount of the corresponding information of the texture coordinate coordinates of the shape data and the texture data is estimated. Like the texture image information, the information amount regarding the sub-texture is overwhelmingly dominant.

Also, the three-dimensional coordinate information of the polygon vertices is irrelevant to the number of photographing viewpoints, and it is only necessary to have one for each polygon vertex.

Therefore, the following description focuses on the texture image coordinates of the variable texture polygon.

Assuming that the texture coordinate information is 2 bytes per one X coordinate or Y coordinate, it is 12 bytes per polygon because it is a triangular polygon.

In order to distinguish which main texture corresponds to each of the polygons, an id is assigned to each polygon.
It is necessary to have If the upper limit of the number of polygons is 65,535, id can be expressed in 2 bytes. For example, θ = 2 in FIG.
In the sub-texture information corresponding to the camera line-of-sight vector of 0 and φ = 6, the information of the polygon with id = 10 is 14 bytes.
Becomes

Therefore, the number of viewpoints is 8,100 and 2,000
For each variable texture polygon, 226 MB is required.

As described above, when the data amount of the texture image is reduced, new information is required. In the case of data amount reduction by cross-reference between sub-textures, the id of the reference partner viewpoint is required. In the case of data amount reduction by position shift compensation, the position shift amount and the id of the reference partner viewpoint are required. In the case of data amount reduction by brightness shift compensation, a position shift amount is required.

If these are considered to be 2 bytes on average, 16 bytes are required for each polygon after all. Therefore, for 8,100 viewpoints and 2,000 variable texture polygons, 259
MB is required.

Next, a method for reducing the data amount will be described.

When expressing the sub-texture information of each polygon, the simplest method is to store the shape of each polygon projected onto each viewpoint as it is, and arrange this in a texture image.

In this case, the shapes of the respective polygons are different, and therefore, it is necessary to have vertex coordinate information in the texture image for all the polygons.

Here, as shown in FIG. 12, it is considered that all polygons are transformed into right triangles (affine transformation), sorted by long sides, and arranged in a texture image. Here, two rectangles 120 constitute one rectangle. If the lengths of the horizontal sides do not match, the shorter side may be deformed (affine transformation) so as to match the longer side.

In this case, as shown in FIG.
Side), make up one group with the same side
Consider that only the side length information 134 in the (X direction) is provided for each rectangle. Then, at the place where the length of the vertical side changes (change of group), information (separator) 130 indicating the change and new vertical length information 132 are inserted.

Since the frequency of occurrence of the separator is low, it is ignored here.

Here, assuming that the maximum length of the side is 256 pixels, the information on the coordinates per polygon is reduced from 12 bytes for the coordinate description to 0.5 bytes. Therefore, the data amount is 4.5 bytes per polygon after all,
For 00 viewpoints and 2,000 variable texture polygons, it can be expressed in 72.9 MB.

[0160] It is preferable that 1 byte of zero is used as the separator. This is because if the separator does not come to this location, it will be either the length of the vertical side or the length of the horizontal side. But these are never zero. Therefore, when this value is zero, it can be immediately determined that the separator is used.

This data amount is calculated from the coordinate data.
id information predominates. It is considered that the id of the variable texture polygon is almost the same when the camera line-of-sight vector is close.

Therefore, it is considered that further compression is possible by further applying Huffman coding to this data.

The above description has been made on the assumption that the shape of the polygon is deformed and the length of the side is variable. Instead, all the lengths of the sides may be normalized to a power of two.

In the above description, the case where the object image is input from 8,100 viewpoints at intervals of 2 degrees for the entire circumference in the θ direction and at intervals of 2 degrees for 1/4 circumference in the φ direction has been described. Obviously, other input viewpoints are also supported.

In particular, it is effective to perform efficient photographing at a position close to the pole, for example, by coarsening the angle increments over the entire circumference in the θ direction.

For compression of texture images, see JP
A method other than EG, for example, a method combining KL transformation and Huffman coding may be used.

Next, a description will be given of a process of reproducing and displaying three-dimensional model data in which fixed texture polygons and variable texture polygons coexist. This reproduction display corresponds to writing image data in an area corresponding to the screen of the virtual camera.

The reproduction and display of the three-dimensional data is performed based on a user's virtual camera viewpoint instruction using, for example, a mouse. Now, it is assumed that a virtual camera position and a posture are specified for the target.

In the case of a fixed texture polygon, the main texture may be written by projecting the polygon on the screen at the designated virtual camera position and performing affine transformation or projection transformation on the polygon.

On the other hand, in the case of a variable texture polygon, it is necessary to extract texture information optimal for the viewpoint of the virtual camera from a plurality of pieces of texture information and map this. For this purpose, first, a positional relationship (θφ information) between the polygon and the vector connecting the camera viewpoint and the polygon is obtained.

Then, the texture line-of-sight vector held in association with the sub-texture information and this θφ
The information is compared with the information, and the closest one is set as the optimal texture information. In some cases, there is no θφ information that matches the texture line-of-sight vector. In this case, the main texture is used as optimal texture information.

After that, as in the case of the fixed texture polygon, the polygon is projected on the screen at the designated virtual camera position, and affine transformation or projection transformation is performed on the polygon to write the optimum texture.

In writing texture on the screen, it is necessary to perform hidden surface processing in consideration of the context of each polygon with respect to the viewpoint. For example, use of the well-known Z buffer method or the like is preferable.

The above processing will be described again with reference to FIG.

First, the image generation viewpoint is determined (S14).
0). Subsequently, the processing described below is performed for all surface elements (for example, polygons).

First, line-of-sight information (display θφ information) formed by the polygon and the image generation viewpoint is obtained (S142).

Next, the sub-texture information having the closest camera line-of-sight vector is determined for the display θφ information (S143). Subsequently, it is checked whether or not the sub-texture information includes the sub-texture information of the surface element (S144).

If there is sub-texture information, this sub-texture information is displayed as appropriate texture information of the surface element (S145). On the other hand, if there is no texture information, the texture information of the surface element is extracted from the main texture information and displayed (S146). This is done for all surface elements.

As another method of determining the optimum texture, instead of selecting only one having the texture line-of-sight vector closest to the display θφ information from the sub-textures, a texture line-of-sight vector close to the display θφ information is selected. A plurality of selections may be made, and an integration process such as averaging them may be performed to generate optimal texture information.

Here, the three-dimensional model data generated and compressed through the process described above is usually once recorded as a file on some medium such as a hard disk, MO, CD, or DVD. At the time of reproduction display, the image is read from these media, temporarily stored in a memory of a display device such as a computer, and then generates an image of a specified viewpoint. Here, when the three-dimensional model data is temporarily stored in the memory of a display device such as a computer, if all of the above texture image data is expanded (uncompressed), 1 GB
This requires a certain amount of memory space.

Therefore, of these texture images,
It is preferable that only the main texture is expanded and the sub-texture information is stored in a memory in a compressed state.
Then, at the point in time when the virtual viewpoint is designated, only the necessary sub-textures need to be developed and texture mapped. This will increase the load on the computer CPU,
For example, in an environment where JPEG decoding hardware can be used, real-time processing becomes possible.

On the other hand, if the performance of the computer used for display is insufficient, it may be difficult to quickly follow the change of the viewpoint of the virtual camera (or the instruction to rotate / move the object) by the user at high speed.

In this case, it is preferable to perform display using only the main texture for all polygons at the time of high-speed viewpoint change, and to resume use of the sub-texture at a time when the viewpoint change is slow or stopped. By adaptively changing the reference texture in this way, it is possible to perform display processing with less stress even when the performance of the display device is low.

When the movement of the viewpoint is predicted, for example, when the object is rotated at a constant speed, a texture display with higher quality than using only the main texture can be performed by performing the following. become.

First, the reference texture update frequency is determined. This is determined according to the performance of the display device, for example,
For example, every 1/3 second.

Next, the viewpoint at the time of updating the texture is predicted. FIG. 15 shows a case where the viewpoint rotates around the object at a certain angular velocity. In this case, when the line of sight comes to the point indicated by the solid arrow, the texture is updated, and at this time, the sub-texture of the viewpoint indicated by the dashed arrow is used.

That is, during the period from time t0 to time t1, the variable texture polygon is displayed with reference to the sub-texture corresponding to the viewpoint u0. At the same time, preprocessing for using a sub-texture corresponding to the viewpoint u1 to be referred to between times t1 and t2 is performed.

Then, at time t1, the referenced sub-texture is replaced with the one corresponding to the viewpoint u1. By repeating this, rather than using only the main texture,
High quality texture display can be performed.

[0189] Recording media for storing the program of the present invention include ROM, RAM, hard disk, CD-ROM, and DV.
Various media such as D and floppy disk can be used, and the program can be provided by communication.

[0190]

As described above, according to the present invention,
A plurality of color data can be generated for one surface, so that even if there is an inaccurate part in the three-dimensional shape data, there is an effect that the display of that part can be made visually suitable.

Furthermore, high-speed reproduction and display can be performed while the storage area used for image reproduction is kept relatively small.

[Brief description of the drawings]

FIG. 1 is a block diagram showing the overall system configuration of a three-dimensional image generation device.

FIG. 2 is a block diagram illustrating an overall configuration of a system according to another embodiment of the three-dimensional image generation apparatus.

FIG. 3 is a block diagram showing an overall system configuration of the three-dimensional image generation and display device.

FIG. 4 is an explanatory diagram illustrating an example of an image input unit.

FIG. 5 is a flowchart of three-dimensional shape modeling.

FIG. 6 is a flowchart illustrating generation of color information.

FIG. 7 is an explanatory diagram of reference viewpoint determination.

FIG. 8 is an explanatory diagram of three-dimensional shape data.

FIG. 9 is an explanatory diagram of texture data.

FIG. 10 is an explanatory diagram of sub-texture data having reference information.

FIG. 11 is an explanatory diagram of relocation of texture data.

FIG. 12 is an explanatory diagram for improving the similarity of the texture by shifting the position;

FIG. 13 is an explanatory diagram of efficient arrangement of texture data.

FIG. 14 is a flowchart of an image reproduction and display process.

FIG. 15 is an explanatory diagram of viewpoint movement during image reproduction and display.

FIG. 16 is an explanatory diagram of camera line-of-sight vector information.

[Explanation of symbols]

 10 image input unit 12 three-dimensional shape input unit 14 color information conversion unit 16 data storage unit 30 image generation unit 32 image display unit 40 turntable 42 camera moving mechanism 44 camera 46 object 70 polygon 72 camera viewpoint image 78 polygon normal 79 camera Line of sight 80 3D shape data 90 Texture information area 92 Polygon vertices in texture image 94 Texture image information 110 Main texture 112 Subtexture 114 Object image 116 Position shift 120 Triangle polygon in texture image 130 Separator 132 Vertical length information 134 Horizontal length information 150 camera viewpoint

 ──────────────────────────────────────────────────続 き Continued on the front page F term (reference) 5B050 BA09 DA07 EA04 EA06 EA09 EA10 EA27 EA28 EA30 FA02 FA05 5B080 AA14 AA17 BA07 FA02 GA22

Claims (72)

[Claims] 1. A three-dimensional image generating apparatus for generating three-dimensional image data of an object from a plurality of pieces of image data of the object, comprising: a model data reading unit that reads three-dimensional shape model data of the object; Color data generation means for generating color data on the surface of the three-dimensional model as data viewed from a plurality of directions using three-dimensional shape model data and a plurality of pieces of image data on the object. A three-dimensional image generation device characterized by the above-mentioned. 2. The apparatus according to claim 1, wherein said color data generating means divides a surface of said three-dimensional model into a plurality of elements, and converts color information on at least one of these elements into an original image. The three-dimensional image is obtained by capturing data from a plurality of pieces of image data on the object, obtaining data as viewed from a plurality of directions, and further relocating these color information in a region different from the original image. A three-dimensional image generation device for generating data. 3. The apparatus according to claim 2, wherein the plurality of elements are triangular polygons, and when relocating to the another area, converting the color information into a right triangle and relocating the color information. Characteristic three-dimensional image generation device. 4. The apparatus according to claim 3, wherein the plurality of elements are triangular polygons, and when rearranged in the another area, two short side lengths are powers of two. A three-dimensional image generating apparatus that converts the color information into a right triangle and rearranges the color information. 5. The apparatus according to claim 4, wherein the plurality of elements are quadrangular polygons, and when relocating to the another area, converting the color information into a rectangle and relocating the color information. Three-dimensional image generation device. 6. The apparatus according to claim 2, wherein when the color information is rearranged in the another area, a rearrangement process is performed based on a length of a side of the element. A three-dimensional image generation apparatus, wherein the three-dimensional image generation apparatus performs rearrangement. 7. The three-dimensional image generating apparatus according to claim 2, wherein a size of the plurality of elements is limited. 8. The apparatus according to claim 2, wherein a size of the plurality of elements has a fixed limit, and an element exceeding the limit is subdivided so as to satisfy the limit. A three-dimensional image generation device characterized by the above-mentioned. 9. The apparatus according to claim 2, wherein a size of the plurality of elements is limited, and the size of the plurality of elements is reduced to satisfy the limit. Characteristic three-dimensional image generation device. 10. The apparatus according to claim 9, wherein the color data generating means divides the surface of the three-dimensional model into a plurality of elements, and converts the color data of the surface of the three-dimensional model for all the elements. In addition to generating as data viewed from a single direction, a part of these elements is further selected from a plurality of divided elements, and for this part of the elements, color data about the surface of the three-dimensional model is converted into a plurality of pieces. A three-dimensional image generating apparatus, which also generates data as viewed from a direction. 11. The three-dimensional apparatus according to claim 10, wherein the selection of the element of the part is performed based on a similarity of color information when the element is viewed from a plurality of directions. Image generation device. 12. The apparatus according to claim 11, wherein the similarity is evaluated by moving color information when the element is viewed from a plurality of directions in parallel within a certain range. 3D image generation device. 13. The three-dimensional image generation apparatus according to claim 12, wherein the similarity is evaluated by correcting the brightness of color information when the element is viewed from a plurality of directions. apparatus. 14. The apparatus according to claim 1, wherein the color data generated as the data viewed from the plurality of directions is associated with a viewing direction, that is, eye-gaze information. Three-dimensional image generation device. 15. The apparatus according to claim 1, further comprising: extracting color data having high similarity among color data generated as the data viewed from the plurality of directions. Similar color data extracting means, representative color data determining means for determining a representative from the similar color data, and reference relation information adding to the color data other than the representative to have reference relation information for referring to the representative color data Means for generating a three-dimensional image. 16. The apparatus according to claim 14, wherein the color data generated in association with the line-of-sight information is
A three-dimensional image generation apparatus characterized in that the three-dimensional image generation apparatus is generated by grouping each line of sight. 17. The three-dimensional image generating apparatus according to claim 16, wherein the color data generated by grouping the same line of sight is compressed for each group. . 18. The apparatus according to claim 10, wherein the color data generated as the data viewed from the plurality of directions is similar to the color data generated as the data viewed from the single direction. A three-dimensional image generating apparatus characterized in that the three-dimensional image generating apparatus is generated by being hierarchized on the basis of the information. 19. A three-dimensional image generation and display device for generating and displaying three-dimensional image data of an object, comprising: means for obtaining a virtual viewpoint position for display;
A three-dimensional image generation and display device, comprising: color data selection means for selecting different color data. 20. The apparatus according to claim 19, wherein the color data is generated as data obtained by viewing the surface of the target three-dimensional model from a plurality of directions. 21. The apparatus according to claim 20, wherein the surface of the three-dimensional model to be displayed is divided into a plurality of elements, and for at least one of the elements, the color data is the surface of the target three-dimensional model. A three-dimensional image generation and display device, wherein the three-dimensional image generation and display device is generated as data viewed from a plurality of directions. 22. The apparatus according to claim 21, wherein the plurality of divided elements are all assigned color data generated as data viewed from a single direction, and some of the elements are divided. Is also assigned color data generated as data viewed from a plurality of directions, and the color data selection means moves the virtual viewpoint for display for each of the plurality of divided elements. If the amount, or the amount of movement of the display object is a certain value or more,
Select color data generated as data viewed from a single direction for all elements, and if the movement amount is equal to or less than a certain value, for some of the elements, data viewed from multiple directions A three-dimensional image generation and display device, wherein the three-dimensional image generation and display device is appropriately selected from the color data generated as the image data. 23. The apparatus according to claim 19, wherein said color data selection means selects a plurality of color data for each element, and performs an integration process on these to generate color data for said element. A three-dimensional image generation and display device, characterized in that: 24. The apparatus according to claim 19, wherein the color data is generated in association with a viewing direction, that is, line-of-sight information, and is further grouped and compressed for each same line-of-sight, and the color data is displayed. A three-dimensional image generation and display device, which expands and displays at least one group of color data corresponding to appropriate line-of-sight information according to a virtual viewpoint for the display. 25. A three-dimensional image generating method for generating three-dimensional image data of the object from a plurality of pieces of image data of the object, wherein: a model data reading step of reading three-dimensional shape model data of the object; Using three-dimensional shape model data and a plurality of pieces of image data about the object, color data about the surface of the three-dimensional model, generating color data as data viewed from a plurality of directions,
A three-dimensional image generation method. 26. The method according to claim 25, wherein
The step of generating the color data, the surface of the three-dimensional model is divided into a plurality of elements, color information on at least one of these elements, from a plurality of image data of the target object is an original image The three-dimensional image is obtained by acquiring as data viewed from a plurality of directions by taking in, and by further rearranging these color information in a region different from the original image. Generation method. 27. The method according to claim 26, wherein
The three-dimensional image generation method according to claim 3, wherein the plurality of elements are triangular polygons, and when relocating to the another area, converting the color information into a right triangle and relocating the color information. 28. The method according to claim 27, wherein
The plurality of elements are triangular polygons, and when rearranging in the another area, the color information is rearranged by converting into a right-angled triangle whose two short side lengths are powers of two. A three-dimensional image generation method, characterized in that: 29. The method according to claim 28, wherein
The three-dimensional image generation method according to claim 1, wherein the plurality of elements are quadrangular polygons, and when relocating to the another area, converting the color information into a rectangle and relocating the color information. 30. The method according to claim 26, wherein when the color information is rearranged in the another area, the color information is rearranged based on the length of a side of the element. A three-dimensional image generating method, wherein the three-dimensional image is rearranged. 31. The method according to claim 26, wherein the size of the plurality of elements has a certain limit. 32. The method according to any one of claims 26 to 31, wherein a size of the plurality of elements has a fixed limit, and an element exceeding the limit is subdivided so as to satisfy the limit. A three-dimensional image generation method characterized by the following. 33. The method according to claim 26, wherein the size of the plurality of elements has a fixed limit, and the size of the plurality of elements is reduced to satisfy the limit. Characteristic three-dimensional image generation method. 34. The method according to claim 33, wherein the step of generating color data divides the surface of the three-dimensional model into a plurality of elements, and for all elements, the color of the surface of the three-dimensional model. Data is generated as data viewed from a single direction, and a part of these elements is further selected from a plurality of divided elements, and the color data about the surface of the three-dimensional model is selected for the elements of this part. A three-dimensional image generation method, wherein the three-dimensional image generation method is also generated as data viewed from a plurality of directions. 35. The method according to claim 34, wherein the selection of the element of the part is performed based on a similarity between color information when the element is viewed from a plurality of directions. Image generation method. 36. The method according to claim 35, wherein the similarity is evaluated by moving color information when the element is viewed from a plurality of directions in parallel within a certain range. 3D image generation method. 37. The method according to claim 36, wherein the similarity is evaluated by correcting brightness of color information when the element is viewed from a plurality of directions. Method. 38. The method according to any one of claims 25 to 37, wherein the color data generated as the data viewed from the plurality of directions is associated with a viewing direction, that is, gaze information. Three-dimensional image generation method. 39. The method according to any one of claims 25 to 38, further comprising extracting color data having high similarity among color data generated as the data viewed from the plurality of directions. A step of extracting similar color data, a step of determining representative color data from these similar color data,
Providing reference relationship information for giving reference relationship information for referencing the representative color data to the color data other than the representative color data. 40. The method according to claim 38, wherein the color data generated in association with the line-of-sight information is:
A three-dimensional image generation method, wherein the three-dimensional image is generated by grouping for each same line of sight. 41. A three-dimensional image generation method according to claim 40, wherein the color data generated by grouping for each same line of sight is compressed for each group. . 42. The method according to claim 34, wherein the color data generated as the data viewed from the plurality of directions is similar to the color data generated as the data viewed from the single direction. A three-dimensional image generation method characterized in that the three-dimensional image is generated by being hierarchized on the basis of the image. 43. In a three-dimensional image generation and display method for generating and displaying three-dimensional image data of an object, a step of obtaining a virtual viewpoint position for display,
A step of selecting different color data. 44. The method according to claim 43, wherein the color data is generated as data obtained by viewing the surface of the target three-dimensional model from a plurality of directions. 45. The method according to claim 44, wherein the surface of the three-dimensional model to be displayed is divided into a plurality of elements, and for at least one of these, the color data is the surface of the target three-dimensional model. A three-dimensional image generation and display method, wherein data is generated as data viewed from a plurality of directions. 46. The method according to claim 45, wherein the plurality of divided elements are all assigned color data generated as data viewed from a single direction, and some of the elements are divided. Are also assigned color data generated as data viewed from a plurality of directions, and the step of selecting the color data includes, for each of the plurality of divided elements, a virtual viewpoint for display. When the moving amount of the display object is equal to or more than a predetermined value, color data generated as data viewed from a single direction for all elements is selected, and the moving amount is constant. When the value is equal to or less than the value, the three-dimensional image generation and display method is characterized in that the partial elements are appropriately selected from color data generated as data viewed from a plurality of directions. 47. The method according to claim 43, wherein the step of selecting color data comprises selecting a plurality of color data for each element, and integrating them to obtain color data for the element. A three-dimensional image generation and display method characterized by generating. 48. The method according to claim 43, wherein the color data is generated in association with a viewing direction, that is, line-of-sight information. A three-dimensional image generation and display method, wherein color data of at least one group corresponding to appropriate line-of-sight information is developed and displayed according to a virtual viewpoint. 49. A recording medium readable by an information processing device for storing a program for generating three-dimensional image data of the object from a plurality of pieces of image data of the object, wherein the program is stored in the information processing device. On the other hand, the three-dimensional shape model data of the object is read, and using the three-dimensional shape model data and a plurality of pieces of image data about the object, color data about the surface of the three-dimensional model is read in a plurality of directions. A recording medium for generating color data as data viewed from a computer. 50. The recording medium according to claim 49, wherein the surface of the three-dimensional model is divided into a plurality of elements, and color information on at least one of these elements is converted into a color information on the object which is an original image. Acquiring as data viewed from a plurality of directions by taking in from a plurality of pieces of image data, and further generating the three-dimensional image data by relocating these color information to a region different from the original image. Characteristic recording medium. 51. The recording medium according to claim 50, wherein the plurality of elements are triangular polygons, and when relocating to the another area, converting the color information into a right triangle and relocating the color information. Recording medium characterized by the above-mentioned. 52. The recording medium according to claim 51, wherein said plurality of elements are triangular polygons, and when rearranging in said another area, two short side lengths are powers of two.
A recording medium, wherein the color information is rearranged by converting the color information into a right-angled triangle. 53. The recording medium according to claim 52, wherein the plurality of elements are quadrangular polygons, and when relocating to the another area, converting the color information into a rectangle and relocating the color information. Characteristic recording medium. 54. The recording medium according to claim 50, wherein when the color information is rearranged in the another area, a rearrangement process is performed based on a length of a side of the element. A recording medium characterized in that the recording medium is rearranged. 55. The recording medium according to claim 50, wherein the size of the plurality of elements has a certain limit. 56. The recording medium according to claim 50, wherein the size of the plurality of elements is given a certain limit, and an element exceeding the limit is re-divided so as to satisfy the limit. A recording medium characterized by the above-mentioned. 57. The recording medium according to claim 50, wherein the size of the plurality of elements is given a certain limit, and the size of the elements exceeding the limit is reduced to satisfy the limit. Recording medium characterized by the above-mentioned. 58. The recording medium according to claim 57, wherein the surface of the three-dimensional model is divided into a plurality of elements, and color data on the surface of the three-dimensional model is viewed from a single direction for all elements. In addition to generating the data as the data, a part of these elements is further selected from the plurality of divided elements, and the color data on the surface of the three-dimensional model is obtained as data viewed from a plurality of directions. A recording medium characterized in that a recording medium is also generated. 59. The recording medium according to claim 58, wherein the selection of the element of the part is performed based on the similarity of color information when the element is viewed from a plurality of directions. Medium. 60. The recording medium according to claim 59, wherein the similarity is evaluated by moving color information when the element is viewed from a plurality of directions in parallel within a certain range. Recording medium. 61. A recording medium according to claim 60, wherein said similarity is evaluated by correcting brightness of color information when said element is viewed from a plurality of directions. 62. The recording medium according to claim 49, wherein the color data generated as the data viewed from the plurality of directions is associated with a viewing direction, that is, eye-gaze information. Characteristic recording medium. 63. The recording medium according to any one of claims 49 to 62, further extracts, from among the color data generated as the data viewed from the plurality of directions, color data having a high similarity to each other. A recording medium characterized in that a representative is determined from these similar color data, and color data other than the representative has reference relation information for referring to the representative color data. 64. The recording medium according to claim 62, wherein the color data generated in association with said line-of-sight information is generated by grouping for each same line of sight. 65. The recording medium according to claim 64, wherein the color data generated by grouping for the same line of sight is compressed for each group. 66. The recording medium according to claim 58, wherein the color data generated as the data viewed from the plurality of directions is similar to the color data generated as the data viewed from the single direction. A recording medium characterized in that the recording medium is generated by being hierarchized based on characteristics. 67. A recording medium readable by an information processing device, storing a three-dimensional image generation and display program for generating and displaying three-dimensional image data on an object,
The recording medium according to claim 1, wherein the program selects different color data for the information processing apparatus according to a virtual viewpoint position instructed for display. 68. The recording medium according to claim 67, wherein said color data is generated as data obtained by viewing the surface of a target three-dimensional model from a plurality of directions. 69. The recording medium according to claim 68, wherein the surface of the three-dimensional model to be displayed is divided into a plurality of elements, and for at least one of the elements, the color data is A recording medium, wherein the recording medium is generated as data in which a surface is viewed from a plurality of directions. 70. The recording medium according to claim 69, wherein all of the plurality of divided elements are assigned color data generated as data viewed from a single direction, and some of the elements are partially divided. Is also assigned color data generated as data viewed from a plurality of directions, and the selection of the color data is based on the movement amount of the virtual viewpoint for display for each of the plurality of divided elements. Or, when the moving amount of the display object is a certain value or more,
Select color data generated as data viewed from a single direction for all elements, and when the movement amount is equal to or less than a certain value, for some of the elements, as data viewed from multiple directions. A recording medium, which is appropriately selected from generated color data. 71. The recording medium according to claim 67, wherein the selection of the color data includes selecting a plurality of color data for each element, and integrating them to generate color data for the element. A recording medium characterized by the following. 72. The recording medium according to claim 67, wherein the color data is generated in association with a viewing direction, that is, line-of-sight information, and is further grouped and compressed for each line of sight. A recording medium characterized by developing and displaying color data of at least one group corresponding to appropriate line-of-sight information according to a virtual viewpoint for display.