JPH1118091A - Multi-view image encoding / decoding apparatus and encoding / decoding method thereof - Google Patents

Abstract

(57) [Summary] [Problem] To improve the texture of an arbitrary viewpoint image reproduced by using a small amount of information while maintaining a high quality of a multi-viewpoint image, and at the same time to use the information amount in a system Within a reasonable range. A multi-viewpoint image is modeled by a modeling unit based on coordinates of points on a subject surface and luminance information. Using this model, each viewpoint image is converted into a prediction signal generation unit 1
05, and a prediction error between the prediction result and the input viewpoint image is encoded. The encoded prediction error is multiplexed with the encoded model. In this way, while encoding the error between the input multi-view image and the predicted image,
By encoding the model information as information of the selected viewpoint image, it is possible to sufficiently reduce the amount of information stored and transmitted as an encoded stream, and to have high reality regardless of the complexity of the subject. Playback can be realized.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a multi-viewpoint image encoding / decoding apparatus used in an image processing system for generating an image viewed from an arbitrary viewpoint using a multi-viewpoint image and the multi-viewpoint image. Encoding / decoding method.

[0002]

2. Description of the Related Art In recent years, with the progress of computer technology and graphics technology, walkthroughs in virtual spaces represented by virtual shopping malls and the like are becoming familiar. In the current virtual space, most of the components are generated by CG (computer graphics). On the other hand, recently, as another approach focusing on reality, a virtual three-dimensional space is generated from a real image. Attempts have been made to do so.

In the field of CG, efforts have been made to pursue reality by refinement of models and input of computational resources. Apart from this, in recent years, "image b
There is a movement to increase the reality by using live-action photography, which is referred to as “used rendering.” As an example of such a technique that has already been put into practical use, Reference 1 (SE Chen et al, SIGGRAPH)
'95 pp. 29-38).
kTime VR.

[0005] QuickTime VR is a technique for cutting out the direction of the line of sight from a panoramic image taken from one viewpoint, and is capable of interactively overlooking a 360-degree field of view around the user. However, in this method, since the viewpoint at the time of image input is fixed to a certain point, it is not possible to reconstruct an image from an arbitrary viewpoint. That is, although the gaze direction can be selected, the viewpoint is always fixed to the same viewpoint, so that it is not possible to reproduce a change in the occlusion (hidden) situation that occurs when the viewpoint changes.

[0005] Image representation methods that can realize more advanced functions such as realization of an arbitrary viewpoint including the distance direction to an object, operability as an object, and arrangability in a space include, for example, A method disclosed in Reference 2 (“A ray space projection method for efficient description of a three-dimensional real space using ray information”, IEEJ: No. 95-119, IEICE Technical Report) is known. In this method, an arbitrary viewpoint image is generated from an input multi-viewpoint image by introducing a concept of “ray space” that expresses a light ray passing through a three-dimensional space and mutually converting the multi-viewpoint image and the ray space. doing. Hereinafter, this principle will be specifically described.

The idea of description based on ray information is that information of a ray propagating in a three-dimensional space is represented by a position (X, Y, Z) and a direction (θ, φ) of the ray on a three-dimensional rectangular coordinate space. Five-dimensional information space f (X,
Y, Z, θ, φ). Here, θ indicates a horizontal inclination of a light ray passing through an arbitrary point (X, Y, Z) in the rectangular coordinate space, that is, an angle on the XZ plane, and φ indicates (X, Y, Z). ) Indicates the inclination in the vertical direction of the light beam that passes through, that is, the angle from the XZ plane to the Y axis direction.

Image information is formed by a set of light rays in the (θ, φ) direction passing through an arbitrary point (X, Y, Z) when the viewpoint is placed on the rectangular coordinate space. In consideration of this, it is understood that the image information and the above-mentioned light beam information can be mutually converted. Here, when it is assumed that the assumption is made that the light beam goes straight without attenuation, the five-dimensional information space f (X, Y, Z, θ, φ) can be reduced to four dimensions. Document 2 mentioned above describes that there are three types of recording methods, namely, flat recording, cylindrical recording, and spherical recording, depending on how the surface on which light information is recorded is taken. The necessary cylindrical recording will be briefly described.

FIG. 23 shows coordinates for cylindrical recording.

First, in order to reduce the space to four dimensions,
The coordinate conversion from (X, Y, Z, θ, φ) to (P, Q, R) is performed. Here, the R axis of the (P, Q, R) coordinate is
As shown in the drawing, it is defined so as to coincide with the traveling direction (θ, φ) of the light beam. Thus, (X, Y, Z, θ, φ)
Is converted into four dimensions of (P, Q, R), that is, (P, Q, θ, φ). All rays in the φ direction with the same θ are
It is recorded at each point on the (P, Q) plane having a normal line in the θ direction. From this, the relationship of P = Xcos θ−Z sin θ (1) Q = −X sin θ tan φ + Y−Z cos θ tan φ (2) is obtained.

In FIG. 23, a rectangular coordinate system 0-XYZ
This is an example of recording a light ray in the traveling direction (θ, φ) passing through the origin (0, 0, 0) of the image. When a leg of a perpendicular line hanging down on the XY plane from the R axis is defined as M, OM is defined as a normal line (P,
Q) A plane is defined, in which all rays in the φ direction will be recorded. In this way, all rays in the same θ direction have the same (P, Q) regardless of their φ direction.
It is recorded at the corresponding coordinate position on the plane.

Next, a description will be given of the principle of recording from a multi-viewpoint image into a light beam space using this principle and generating an arbitrary viewpoint image from the recorded light beam space.

Although the ray space is reduced to four dimensions by the above-described principle, the number of multi-viewpoint images to be handled and the amount of calculation are extremely large and impractical. Operation ”Ishikawa et al., PCSJ96 P-
In 3.13), parallax in the φ direction is ignored (p,
θ) in a two-dimensional space.

Further, since the number of light beams that can be actually handled is limited, sampling is interposed in practical use.
That is, the number of viewpoints treated as input is limited to a finite number. At this time, when the coordinates (X, Z) of a certain viewpoint on the Y = constant plane are determined, the trajectory of the ray passing through that point is given only by the equation (1). This is because the value in the Q direction does not change on the Y = constant plane. When an image is input with a finite number of viewpoints set, the ray space is filled with ray information along the trajectory of rays passing through the coordinates (X, Z) of a certain viewpoint. If a portion of the ray space where no ray information is obtained is interpolated by interpolation or the like with the obtained ray information, a ray space densely filled with ray information is obtained. At the time of reproduction, if an arbitrary viewpoint (X, Z) is specified, the trajectory of a light ray passing through that point is obtained from Expression (1). An image on one scanning line at Y = constant on the screen of the image, that is, one display line corresponding to a specific Y value is obtained.

By stacking the scanning lines thus obtained in the Y direction, an image for one screen viewed from an arbitrary viewpoint can be obtained.

As described above, in the generation of an arbitrary viewpoint image based on the principle using the ray space, the information to be handled is simplified so as to be as small as possible. Even with simplification, the amount of information and processing is still very large, real-time playback,
Reduction of the amount of information has been raised as an issue.

Next, a description will be given of a conventional technique used therefor from the viewpoint of reducing the amount of information.

Normally, encoding is performed to reduce the amount of information. When encoding a multi-viewpoint image, various methods can be considered. One is a method of encoding an image from each viewpoint as a still image by a method such as JPEG. However, this method is useless because it does not use the correlation between images from each viewpoint. The second method is a method in which a multi-view image is regarded as a moving image and is encoded by a method such as MPEG, and a correlation between adjacent viewpoint images is removed to some extent by motion compensation prediction. Especially for multi-view images 1
In encoding a stereo image as one case, a similar principle is called disparity compensation prediction.
(JP-A-2-100592).

A similar principle is also supported in the MPEG2 standard in the category of a multi-view profile. In this method, a kind of modeling is performed on an image. However, modeling is performed only for each block, and when viewed in pixel units, the deviation from the model tends to be large. In addition, the parameters of the model are given to each block in the form of a motion vector, so that the information amount of the model parameters becomes considerably large as a whole.

As a technique relating to the encoding of a multi-view image incorporating the concept of a ray space, reference 5 (“Compression of ray space data for mixed reality display”, Katayama et al., IMPS9)
6I-6.1). In this method, some of the 120 multi-viewpoint images obtained from the same subject are set as a reference image and sent as it is, and for other images, a pixel indicating which pixel value in the reference image is close is specified. Information is compressed by sending it as information. Therefore, as compared with the above-described diversion of motion compensation, the deviation from the model is small in that the modeling is performed in pixel units, but the parameter information of the model is conversely large.

In Reference 6 (Japanese Unexamined Patent Application Publication No. 8-115434), as another technique related to the compression of a multi-viewpoint image, a parameter that defines the trajectory of (small) corresponding points of the multi-viewpoint image is used. A technique for performing interpolation for an arbitrary viewpoint image on the receiving side is disclosed. At this time, if the ray space is represented only by these parameters, it is possible to represent with a very small amount of information.

The documents 5 and 6 all attempt to represent a multi-viewpoint image only by a model. However, in the latter case, in particular, in the latter case, the estimation of the model from the multi-viewpoint image is quite difficult, and it is considered that the estimation involves an error. Become. As a result, the image to be reproduced becomes unnatural for a portion where an error has occurred in the estimation of the model.

On the other hand, reference 7 (“Compression and interpolation of a multi-view three-dimensional image based on three-dimensional structure estimation”, Fujii, 3)
The dimensional image conference '94 1-4) discloses a method of generating a prediction signal of a multi-view image from a three-dimensional structure model and transmitting a prediction error. The model in this document is an approximation of a plane or a plane patch, and the accuracy of prediction by this is not very good. However, if the accuracy of the model is simply improved, the information amount of the model itself increases as the accuracy increases, and the entire information amount cannot be reduced.

[0023]

Among the above-mentioned compression methods, in the conventional methods based on diversion of coding (disparity assurance prediction, multi-view profile, etc.), the properties of multi-view images are not fully utilized. There was a problem that it was not possible to significantly compress a huge amount of information.

Also, in the method using only the model as described in References 5 and 6, it may be difficult to estimate the model, and if an error occurs in the estimation, the reproduced image becomes unnatural. Was. For example, in the above two documents, under the assumption that the subject surface is a perfect diffusion surface, the former considers that there is a pixel having the same luminance value in a near viewpoint image, and the latter considers that the same subject surface has the same luminance value. The point is that the luminance value from a point is considered to be represented by a representative value and is represented by that value. However, the actual subject may not be a perfectly diffused surface, or the corresponding points may not be visible or difficult to determine due to the relationship between occlusion and surface texture, and modeling may not always be completely performed. When a model is forcibly applied to such a pixel, there is a problem that an error becomes extremely large.

The present invention has been made in consideration of the above circumstances, and makes use of the fact that, when the pixels of each multi-viewpoint image are light rays emitted from the same point on the subject, the trajectory can be expressed by a mathematical expression. Modeling is performed on all viewpoint images using common parameters, and a large amount of information of a multi-viewpoint image is compressed and expressed with a small amount of information.
Even if the subject is not well modeled, it is possible to reproduce a natural arbitrary viewpoint image with reasonable information transmission or storage amount, and also maintain reasonable accuracy with respect to the information amount of the model itself. It is an object of the present invention to provide a multi-view image encoding / decoding device and a multi-view image encoding / decoding method that can be stored in an information amount.

[0026]

In order to solve the above-mentioned problems, the present invention according to the present invention is directed to a method for displaying a plurality of viewpoints on a subject obtained from a plurality of viewpoints whose positional relationship with the subject is known. In a multi-view image encoding apparatus for inputting an image and encoding a multi-view image including the plurality of viewpoint images, the multi-view image is modeled and configured from coordinate values and luminance values of respective points on the subject plane. A viewpoint image to be used as a model for each point on the object plane, and corresponding luminance information in the viewpoint image corresponding to the model information corresponds to the model information. Modeling means used as a luminance value of a point, and model information composed of coordinate values and luminance values of each point on the object plane, and image information of a viewpoint image used as a model of each point. Means for predicting the image of each viewpoint image from the model information, means for coding the prediction error with the input viewpoint image, and multiplexing the encoded model information and prediction error. Multiplexing means for generating an encoded stream of the multi-view image.

In this multi-viewpoint image coding apparatus, the redundancy of the multi-viewpoint image is such that when pixels of a plurality of viewpoint images are generated from light rays emitted from the same point on the subject, they have values close to each other. Modeling is performed using. In this case, the model information generated by modeling includes coordinate values and representative luminance values for each point on the object plane. Here, the term “brightness value” is a general term for a TV camera that corresponds to a brightness signal Y and color difference signals Cr and Cb, or signals R, G and B of three primary colors. As a representative luminance value for each point, a value based on corresponding luminance information in one viewpoint image selected as a model from a plurality of viewpoint images is used. The model information of each point is encoded as luminance information of the viewpoint image used as the representative luminance. Thus, the model information does not exist alone, but exists as a part of the image information of each viewpoint image.

In encoding each viewpoint image, first, the image is predicted for each viewpoint image based on the model information, and a prediction error between the predicted image and each input viewpoint image is obtained. Then, the prediction error is encoded and multiplexed together with the encoded model information in an encoded stream. In this way, by encoding the error between the input multi-view image and the predicted image, it is possible to sufficiently reduce the amount of information stored and transmitted as an encoded stream, and realize high-reality reproduction. it can.

Encoding the model information as information in one viewpoint image has the following meaning. In other words, in order to significantly reduce the amount of information for model information, it is necessary to perform compression to remove spatial correlation using correlation between neighboring points, such as by blocking both luminance information and coordinate information. There is. For this reason, creating a new model plane from a multi-view image is time-consuming, and at the same time, depending on the method, it may not be possible to obtain sufficient correlation between adjacent points on the model plane due to the influence of sampling or the like. It can happen. On the other hand, the method of encoding model information as information in one viewpoint image is a method that guarantees the maximum correlation between adjacent points among methods using information obtained from human power. . For this reason, it is possible to sufficiently compress the information of the model. Another feature is that a model can be defined for all visible points on the object plane. Since the object surface is generally a complicated curved surface, it is difficult to define a plane covering all the luminance information of the object surface. However, when actually trying to introduce block coding or the like, it is preferable that the model is defined on a plane. The method of encoding model information as information in one viewpoint image is a method having both of them.

Further, since model information of each point is encoded as luminance information of a viewpoint image used as a representative luminance of the point, prediction error information is used for an image portion used as the representative luminance. Instead, it can be reproduced only with the model information, and the information amount of that part can be saved. Therefore, the amount of information of the model can be reduced while maintaining sufficient model accuracy.

In order to improve the performance of prediction from model information, in the invention according to claim 2, a viewpoint image having a viewpoint closest to a normal direction from a point on the object plane is defined as the point image. It is characterized by being used as a model. As a result, for each point on the object plane, the corresponding luminance information of the viewpoint image having the viewpoint closest to the normal direction from that point is used as the representative luminance value, and the prediction performance is improved. Can be done. That is, since each viewpoint image obtained as an input multi-viewpoint image is an image sampled both angularly and spatially, depending on the method of selecting the viewpoint image used as the representative luminance value, the reproduced image is There is a concern that the information may become coarse or an unnecessary amount of information may be consumed. However, in claim 2, luminance information of a pixel taken from the normal direction of the subject plane is used as a representative luminance value used in the model information. Therefore, a sufficient resolution of the model can be obtained, and a necessary and sufficient predicted image can be generated. Therefore, regardless of the complexity of the subject, reproduction with high reality can be realized with a small amount of information.

Further, in order to prevent the processing on the decoding / reproduction side from becoming unnecessarily large, in the invention according to the third aspect, the multiplexing means comprises coordinate values and luminance values of each point on the object plane. The model information is collectively multiplexed at another information position different from the information position on the coded stream where the prediction error information is multiplexed. Since model information consisting of coordinates and brightness of points on the object plane is always used for reproducing an image from any viewpoint, it is used as prediction error information in which a portion used by a viewpoint differs. By dividing and multiplexing them in a specific area on the encoded stream, the decoding and reproducing side does not need to search all the information to access the model information, and does not need to perform unnecessary processing.

[0033]

Embodiments of the present invention will be described below with reference to the drawings.

In the following, a description will be given based on an example according to the cylindrical recording model shown in Reference 2 by Naemura et al.
The same applies to the case where another model is used.

First, an example of the configuration of the multi-viewpoint image input unit used in this embodiment will be described with reference to FIG. In this example, a subject 1701 is mounted on a turntable 1702 and photographed by a camera 1703. The optical axis 1706 of the camera 1703 passes through the rotation axis 1705 of the turntable 1702, and the optical axis 1706 and the rotation axis 1705 are orthogonal to each other. The positional relationship between the two is known, and the distance between them is R. The rotation of the turntable 1702 is controlled by a control device 1704 at a fixed angle (for example, 3 degrees).
An image is input by the camera 1703 each time.
In this example, this operation causes 120 different viewpoints in one rotation.
One image will be input. Next, the light beam space in this case will be described.

As shown in the above-mentioned document by Naemura et al., In the case of cylindrical recording, a ray passing through a point (X, Y, Z)
The coordinates are recorded in the coordinates represented by the following equation in the ray space.

P = Xcos θ−Z sin θ (3) Q = −X sin θ tan φ + Y−Z cos θ tan φ (4) Here, as described in the related art, Q is used so that the amount of information to be processed and the amount of calculation are not too large. It is assumed that the change of the light ray in the direction is not considered, and the light ray space is defined by a plane (for example, 1707) perpendicular to the rotation axis 1705 and is approximated by stacking in the Y direction (in this case, φ
Occlusions due to changes in direction are not reproduced.)

FIG. 21 shows the positional relationship between the cross section 1801 of the subject on the plane 1707 and the camera 1703. Here, for notation, when a polar coordinate expression (r, ψ) centered on the rotation axis 1705 of the turntable is introduced as shown in FIG. 21, X = rsinψ (5) Z = rcosψ (6) From equation (3), P = rcos θ sinψ−rsin θcosψ = rsin (ψ−θ) (7) Accordingly, the trajectory 1901 of the 120 multi-viewpoint images in the ray space is represented by P = Rsin (n * 3 ° -θ), n = 0,..., 119 (8) as shown in FIG. Curves 2001 and 200 in the locus 1901
Reference numerals 2, 2003, 2004,... Indicate viewpoint images taken from different angles by 3 °. That is, these curves 2001, 2002, 2003, 2004,.
Each represents an image for one scanning line corresponding to a specific Y position in a plurality of viewpoint images taken from different angles by 3 ° from each other.

The same property applies to a point on the object plane. When a function r (ψ) representing the object plane is introduced, the trajectory 1902 of the light beam emitted from the point (r (ψ), ψ) becomes P = r (ψ) sin (ψ−θ) (9)

Therefore, an arbitrary point (r
The light ray information of the trajectory 1902 of the light ray exiting from (ψ), ψ) is obtained by calculating each curve 200 in the trajectory 1902 and the trajectory 1901.
1, 2002, 2003, 2004,..., That is, 2005, 2006, 2007, 2008. This applies not only to points on the object plane but also to arbitrary points on the (X, Y, Z) coordinates. Therefore, when reproducing an arbitrary viewpoint image, the trajectory of a ray passing through the viewpoint is determined in the ray space of FIG.
Each curve 2001, 2002, 2003, 200 in 1
By obtaining light beam information at the intersections with 4,..., One scanning line on the screen of the arbitrary viewpoint image is reproduced. Accordingly, in this example, the same operation is performed on each plane parallel to the plane 1707, and the obtained scanning lines are stacked in the Y direction, so that the entire screen of the arbitrary viewpoint image, that is, all the scanning lines for Y can be reproduced. .

The fact that the trajectory of the light beam emerging from one point r (ψ) on the object plane in the light beam space can be expressed by the above-described equation (9) means that the value of the light beam information does not change much depending on the direction of the light beam on the object surface. If the surface has a property close to a perfect diffusion surface, a multi-viewpoint image corresponding to a point r (ψ) on the object surface can be modeled by the coordinates r of the point and a representative value of luminance. In the case of the property far from the perfect diffusion surface, it is shown that the model can be modeled by the luminance L (θ) which is a function of the coordinates r and θ. It should be noted that the term “luminance” as used herein refers to a signal corresponding to the luminance signal Y and the color difference signals Cr and Cb, or the signals R, G and B of three primary colors in a TV camera. In the referenced document, this luminance is called light beam information.

Next, an embodiment of the multi-view image encoding apparatus according to the present invention will be described with reference to FIG. In the figure, a multi-view image input unit 100 is an image input device that can capture images from a plurality of viewpoints under a condition that the positional relationship between a subject and a camera is known, for example, a specific configuration example illustrated in FIG. The multi-viewpoint image input here is temporarily stored in the storage unit 101 such as a hard disk. The control unit 102 controls the entire coding. The encoding procedure is
(1) Modeling is performed by viewing the entire multi-view image, and (2)
After the provisional encoding is performed to determine encoding parameters such as quantization from the relationship with the entire code amount, (3) actual actual encoding is performed. Therefore, the control unit 102 also controls these procedures.

First, the control unit 102 issues a model generation instruction to the modeling unit 103, and the modeling unit 103 accesses the storage unit 101 according to the instruction to read the stored multi-viewpoint image, and generates a model based on the above-described principle. The estimation phase starts (details of the model estimation method will be described later). In the present embodiment, the information representing the model is composed of polar coordinates r of each point on the subject plane and a representative value of the luminance of the point. As the representative value of the luminance, the corresponding luminance information of the viewpoint image taken from the viewpoint closest to the normal point direction from the corresponding point on the subject plane is used. Alternatively, a value for optimizing the entire prediction may be obtained in consideration of the later-described prediction. Then, the luminance value of the model information for each point is intra-coded and sent as luminance information of a viewpoint image taken from the viewpoint closest to the normal direction from that point (details will be described later). Therefore, the tasks of the modeling unit 103 are to estimate the polar coordinates r corresponding to each point on the subject plane and to specify the viewpoint image from which the representative luminance corresponding to each point is to be obtained.

When this operation is completed, the corresponding block in the viewpoint image used as the representative luminance of each point on the object plane is set in the DCT together with the block having the corresponding value of r.
The signal is sent to an intra-frame encoding circuit composed of a circuit 111, a quantization circuit (Q) 112, and a variable-length encoding circuit (VLC) 113. In this part, first, the DCT circuit 111 performs DCT on the luminance and r blocks of that part, quantizes it in a quantization circuit 112, performs variable length coding in a variable length coding circuit 113, and Sent. The signal quantized by the quantization circuit 112 is again converted into an inverse quantization circuit (Q
^-1 ) Inverse quantization and inverse DCT circuit (IDCT) 1 at 114
The inverse DCT is performed at 15 and stored in the model storage unit 104. Here, since the value of r greatly affects the performance of generating a prediction signal described later, it is desirable to quantize as finely as possible. It is also conceivable to use another encoding method such as DPCM. Further, completely lossless coding may be used. In addition, since the luminance value is also a source of a prediction value used for prediction of the entire multi-view image, it is desirable to quantize with a fine step size.

In moving picture coding such as MPEG, I,
The P picture is quantized more finely than the B picture. However, it is desirable that the P picture is differentiated more greatly than the degree of differentiation, that is, the luminance to be sent as a model is considerably finer than the quantization used for error coding described later. Control unit 1
02 determines the quantization step size of these models from the total target encoding bit number, and
And the inverse quantization circuit 114. However, for the above-described reason, it is desirable to set the model quantization step size to a small value so as not to be greatly affected by the target number of coding bits.

When the encoding of the model information is completed, the tentative encoding of the prediction error is performed, and the quantization step size for the encoding of the prediction error is determined. Here, first, the multi-viewpoint image is read from the storage unit 101 one frame at a time (assuming that the image from each viewpoint is composed of the frame referred to here). For each frame, a prediction signal is generated by the prediction signal generation unit 105 from the model information of r and luminance stored in the model storage unit 104. The principle of generating a prediction signal will be described with reference to FIG.

FIG. 2 is a diagram of the p-θ plane described with reference to FIG. As described above, this corresponds to one scanning line obtained by cutting the multi-viewpoint image at a fixed cross section of Y = Y. A frame 201 is a frame to be currently encoded (a viewpoint image obtained by photographing a subject from a specific angle), and a frame 202 is a frame including a model (a specific angle including a portion used as a representative luminance value). (Viewpoint image), and a block 206 indicates a block coded and decoded as a model. Creating the prediction signal
For each incoming encoded frame, block 20
This is performed by projecting the point to be coded from all the points on the block coded as a model as shown in FIG. Point 207 on block 206 has the decoded luminance value and the value of r. Since this point is also projected from a point on the subject, the luminance value of the point is expressed by the polar coordinate expression of the point on the subject that is the source of the luminance of this point according to the principle described above. r, ψ), it is represented as an intersection of rsin (ψ−θ) and the frame 202. Among them, the value of r is decoded as information, and the value of θ is uniquely determined by the position of the frame 202, so the value of ψ may be obtained.

There are two kinds of curves of rsin (ψ−θ) passing through this point, for example, 203 and 204 in the example of FIG.
As shown in FIG. 3, what is projected as a point visible on the p-θ plane is uniquely determined by the larger value of z = rcos (ψ−θ), which is ψ1. Accordingly, the trajectory of the light beam passing through the point 207 is determined to be 203. From this, the point on the frame 201 projected from the point 207 is represented by both curves 2
It becomes the intersection 208 of 03,201. There may be other trajectories of the ray projected onto the point 208 (in this example, 20
5). When a plurality of rays are projected on one point, the ray having the largest value of z is finally projected according to the principle of FIG. 3 (203 in the example of FIG. 3). When the projection from all points, such as 207 coded as a model, to the frame 201 is performed, the frame 201 is densely filled with the prediction signal.
The model has been selected and coded to satisfy this. In this manner, for each frame to be encoded, the image of that frame is predicted based on the model information of each point on the object plane.

As another method, a method of reducing the modeling load and allowing an incomplete model can be considered. In this case, there may be a point where no prediction signal is formed even when all points of the model have been projected. In that case, the prediction signal may be 0 (method 1) or may be interpolated from the generated neighboring prediction signals (method 2). Conversely, in this embodiment, since the minimum unit of model coding is a block, a point on the viewpoint image corresponding to a point on the object plane is encoded multiple times with respect to one point on the object plane. May be possible. In this case, when the projection is performed at one point, as a rule in the projection from the model, for example, the projection from the point coded as the model on the viewpoint image closest to the non-projection plane in terms of angle is enabled. For example, it is necessary to prevent a mismatch in prediction signal generation by projection between the encoding side and the decoding side.

The prediction signal of each frame generated in this manner is subtracted by the subtractor 106 from the corresponding input frame in block units, and a prediction error is obtained. This prediction error is sent to the DCT circuit 107. Here, the result of the DCT performed on a block basis is stored in the storage unit 108 and quantized on a block basis by the quantization circuit (Q) 109. In the tentative encoding stage for determining the step size, quantization is performed with two fixed step sizes Q ₁ and Q ₂ , and the code amount counter 117 accumulates the code amount for all frames for each. You. The control unit 102 subtracts the code amount used for the model code amount and the code amount used for additional information such as the header from the total target code amount and the remaining code amount obtained from the total target code amount. From the code amount, the step size Q _E for making the total code amount fall within the target range.
Is estimated by interpolating a curve representing the relationship between the code amount and the step size as shown in FIG. 4, and this value is used for encoding the actual prediction error in the next step. Specifically, it is assumed that log (code amount) = α log Q + β (10), α and β are estimated from the coding results of Q ₁ and Q ₂ , and Q _E is obtained inversely from the target code amount. . Here, all the prediction error information of each frame has the same property,
Since there is no difference in properties between reproduced frames as in MPEG, the step size is estimated on the assumption that the prediction error of any frame is quantized with the same step size.

In the next main encoding step, since the generation of the prediction error information and the DCT have already been completed in the temporary encoding stage, the D
The result of CT is read and quantized by the quantization circuit 109. Quantization by the quantization circuit 109 is performed by the optimum step size Q _E estimated by the control unit 102 mentioned earlier, the results are variable length coded by the variable-length coding circuit (VLC) 110 are multiplexed Sent to the conversion circuit 116. Here, it is determined that a block in which the coefficient information quantized by the quantization circuit 109 is all 0 is determined to be invalid, and no information is transmitted for that block. Since the prediction according to the present embodiment uses a model directly corresponding to a physical phenomenon, it can be expected that a prediction error hardly appears for an image that is successfully modeled. Therefore, in order to significantly reduce the amount of information, it is very important to perform the validity determination in this way so as not to send invalid blocks.

In FIG. 1, the storage devices 101 and 108 are written independently, but they may be physically the same. Also, the DCT circuits 107 and 111, the quantization circuits 109 and 112, and the variable-length coding circuits 110 and 113 are common in hardware, considering that model coding and prediction error coding are not performed simultaneously. Can be

Next, the multiplexing circuit 116 multiplexes the coding result of the model information from the variable-length coding circuit 113 and the coding result of the prediction error from the variable-length coding circuit 110 to form a multi-view image. Output the encoded stream. FIG. 6 shows a multiplexing format at this time.

The description of the multiplex format in FIG. 6 is based on the ITU-T Recommendation H. 261 multiplexing format. In the present embodiment, the format is made as similar as possible to the existing encoding such as MPEG. First, the format was a hierarchical structure. The information of each layer includes a header and a trailer including layer-specific information such as a start code and an end code. The uppermost layer is an object layer represented by one set of multi-view images, and the OSC (Ob)
project Start Code), ON (Object
t Number), a model layer including model information of each point on the subject, an error layer including prediction error information of each frame, and an OEC (Object End Code)
e). The object is divided into a model layer and an error layer in the next layer. On the playback side, the model layer is always accessed at the initial stage of decoding, regardless of the viewpoint from which the image is played back, whereas the error layer is the part accessed by the viewpoint to be played back. change. Therefore, in order to facilitate the search for necessary information, the model layer is arranged at the front of the object layer information as shown in the figure, followed by the error layer information. I have.

The model layer is an MSC (Model S)
start code), a header (QUANT) indicating a quantization step size of luminance information, a header (QUANT_r) indicating a quantization step size of coordinate information, a frame layer including information of a plurality of frames used as representative luminance, and MEC. (Model End Cod
e). The error layer is an ESC (Ero
r Start Code), a header (QUANT) indicating a quantization step size of error information, a frame layer including prediction error information of all frames, and EEC
(Error End Code).

The frame layer included in the model layer and the frame layer included in the error layer have a common structure. Each frame layer is a PSC (Picture S)
start Code), a header indicating the frame number (P
N: Picture Number), a header (R, ψ) indicating the conditions under which the frame was shot, and a plurality of slice layers belonging to the frame. R indicates the distance between the central axis of the subject and the camera, and ψ indicates the angle of the camera with respect to the subject, that is, the angular coordinates of the subject plane that intersects the optical axis of the camera. The information of R and 情報 may be included in the header of the object layer if it is limited to the case where the shooting is performed at a fixed distance and a fixed rotation angle step as described with reference to FIGS. There is no problem even if the distance from the camera and the step width of the rotation angle are different for each frame. In the present embodiment, it is assumed that the frame exists in the frame layer. Although the information of R and ψ can be logically reproduced if it is contained in either one of the frame layer of the model and the error, it should be contained in both for convenience of access. I do.

Although a macroblock (MB) layer is usually provided below the frame layer, a slice layer may be inserted between the layers as in MPEG (the slice layer is also described in the example of FIG. 6). When a slice layer is included, access to each slice becomes easy, decoding processing becomes easy, and it becomes easy to reproduce only a part of the object cut by a plane orthogonal to the rotation axis.

Each slice layer has an SSC (Slice)
Start Code), SN (Slice Num)
ber) and a plurality of macroblock layers belonging to the slice. Below the macroblock layer is a block layer, in which DCT coefficients (Tcoeff) and EOB (End of Bloc) are located.
k).

The macro block of the model is composed of blocks of r, Y (luminance), Cr, Cb (color difference) as shown in FIG. The error macro block is composed of Y (luminance), Cr, and Cb (color difference) blocks obtained by removing the r block from FIG. In both the slice and macroblock, the header includes address information in the frame (SN and M).
BA), slices and macroblocks without information are not sent (skipped). Macro block layer CBP (Co
“ded Block Pattern” is information indicating a pattern of a block including information in a macroblock, and is transmitted only for a macroblock in an error layer.

The multi-viewpoint image stream created in the multiplexed format shown in FIG. 6 is stored in a computer-readable recording medium or transmitted to a reproducing side via a transmission path.

Next, the configuration of a decoding device corresponding to the encoding device shown in FIG. 1 will be described with reference to FIG.

This decoding device is a decoding device that inputs information indicating a viewpoint and reproduces an image from the viewpoint. First, the multi-view image stream generated by the encoding device in FIG. 1 is read from a recording medium or received via a transmission path and input to the separation circuit 601. Here, the object information in the stream is separated into a model and an error. Model information is a variable length decoding circuit (VLD)
Variable length decoding is performed in 607, and DCT coefficient information among them is inversely quantized in an inverse quantization circuit (Q ⁻¹ ) 608 and inverse DCT in an inverse DCT circuit (IDCT) 609 to obtain frame numbers and values of R and ψ. It is stored in the model storage unit 610 together with the accompanying information included in such a header. Also,
The error information is stored in the error storage unit 603. You are now ready to play.

Next, as the next step of the reproduction, information representing the position of the viewpoint to be reproduced is input from the viewpoint input section 600. As an example of the viewpoint input unit 600, a virtual lever or handle is displayed on a display screen realized by a VRML browser, and the viewpoint position is input by virtually operating this with a pointing device such as a mouse. System. With this input, the generation of the image of the corresponding viewpoint is started. This will be described with reference to FIG.

First, the viewpoint input unit 600 sends parameters (viewpoint coordinates) corresponding to the viewpoint to the address calculation circuit 602. The address calculation circuit 602 calculates a curve 7 representing a screen to be reproduced on the p-θ plane from the viewpoint coordinates and the above equation (9).
01, and further, with reference to the model information from the model storage unit 610, for each coordinate of each point on the subject surface, the trajectory of the corresponding ray is obtained. FIG. 8 shows a locus 702 of a light ray corresponding to a specific point on the object plane.

Next, the address calculation circuit 602 reproduces image information corresponding to the intersection point between the curve 701 and the trajectory of the light beam corresponding to each point on the object plane such as the trajectory 702. First, a calculation is performed to determine which pixel position of which viewpoint image in a plurality of viewpoint images should be referred to. In this case, a pixel position on the viewpoint image closest to the intersection point is determined as a pixel position to be referred to. For example, regarding the point 703 on the curve 701, the pixel position 704 on the viewpoint image 705 closest to the point 703 is
Is the reference pixel position. In this embodiment, when an image of a specified viewpoint is reproduced, an image of a point on a subject plane that completely corresponds to a pixel position to be referred to (for example, 704) is decoded. This means that, when images from a certain number of viewpoints are used as the multi-viewpoint image, when an image at an arbitrary viewpoint is reproduced on the receiving side, a ray close to the corresponding line-of-sight direction is selected and used instead.

Next, the prediction signal generation unit 611 performs projection from each point of the model information to each point corresponding to 703 on the screen 701 to be reproduced based on the information of the calculated reference pixel position. , 704 are formed. In this way, a predicted image of the specified viewpoint is created from the model information. This prediction procedure is the same as the procedure performed when forming the prediction signal for each viewpoint image in the description of the generation of the prediction signal in the encoding device in FIG.

In parallel with the prediction signal generation processing, the address calculation circuit 602 generates an address for generating a specified viewpoint image in the frame memory 613 based on the calculated information on the reference pixel position. At this time, a frame number including error information, a macro block number, and a pixel number necessary for addressing the corresponding reference pixel are sent to the error accumulation unit 603. The macro block corresponding to this address is read from the error storage unit 603 and is subjected to variable length decoding by a variable length decoding circuit (VLD) 604,
Inverse quantization is performed by an inverse quantization circuit (Q ⁻¹ ) 605 and inverse DCT is performed by an inverse DCT circuit (IDCT) 606.

The created prediction signal and the inversely DCT-predicted error signal are added and decoded by an adder 612, and written to a corresponding position in the frame memory 613. Here, the prediction signal is formed for each pixel, and the prediction error is formed for each block. However, it is highly likely that a pixel to be referred to belongs to the same reference screen at a pixel which is close, and the above process is repeated by the number of pixels. If a plurality of usable pixels are included in one prediction error block decoding instead of the above, all of those pixels are decoded together. In this manner, for each pixel position to be referred to, the prediction signal generated from the model information and the decoded prediction error signal are added and written to the corresponding position in the frame memory 613, thereby specifying the designated position. An image of the viewpoint is created.

In the above description, the pixel position to be referred to is determined from the closest viewpoint image for each point on the curve 701 corresponding to the viewpoint to be reproduced, and is referred to. Each point on the surface (eg, 703)
Are obtained on the two viewpoint images (705, 706) sandwiching the pixel positions (704, 707), pixels are generated by referring to the pixel positions, and a pixel is generated from these two pixel values. The pixel value to be reproduced may be determined by insertion.

Next, the procedure of the encoding method according to the present invention will be described with reference to the flowcharts of FIGS.

This encoding method executes the same operation as the encoding process of the encoding device described in FIG. 1 by using a computer program, and the description is mostly the same as the description of the encoding device in FIG. Therefore, the detailed description is omitted and only the minimum necessary description is given.

Here, as an example, a description will be given assuming that the number of viewpoint images included in the multi-view image is n _max (for example, 120). In First step 800, sets a target total code amount of the object to be coded, the step size Q _{r (model} layer for quantizing the model information (= r and brightness) on the basis of this value QUANT encoded as r), is encoded as QUANT of _{Q L} (model layer) is determined. Next, in step 801, a multi-viewpoint image is input. The multi-view image is stored in the storage unit.
Next, in step 802, model estimation is performed. In this step, the coordinates of each point on the object plane and the luminance information are estimated in the same manner as performed by the modeling unit 103 in FIG. Next, in step 803, encoding of the model information is performed. The encoding is exactly the same as the encoding performed by the DCT circuit 111 and the quantization circuit 112 described with reference to FIG. 1, and the model information is encoded by using the hardware or only by the software.

Next, at step 804, the above-mentioned step 80
The information encoded in step 3 is decoded and stored as model information. At the same time, the above-mentioned step 803 is executed in step 805.
The variable-length information is encoded in the temporary storage unit. Next, a phase of provisional encoding of each viewpoint image is entered.

First, at step 806, n and SUM1, SU
M2 is set to 0, and the encoding loop starts. At step 808, the image n is read, and at step 809, the image n is encoded. It will be described later steps of the encoding, but quantization using the quantization step size Q ₁ and Q ₂ predetermined in step quantization in this. Since the steps up to DCT are the same in the main coding after the provisional coding, the results so far are stored in the storage unit. In this case, the previously stored image n becomes unnecessary and may be overwritten here. Next, in step 810, SUM1,
The code amounts Nob1 and Nob2 generated when encoding is performed with Q1 and Q2, respectively, are added to SUM2. After repeating this loop for the number of images, in step 812, SUM
The step size QE (encoded as QUANT of the error layer) for permanently encoding the prediction error is determined using the values of 1, SUM2. The method of determination is as described in the description of the encoding apparatus in FIG. Subsequently, the process enters a main encoding loop after step 813.

First, at step 815, D corresponding to image n
The results obtained up to the CT processing are sequentially read. Next, in step 816, the data read in step 815 is encoded. The encoding in this case is the same as the quantization in the quantization circuit 109 and the variable-length encoding in the variable-length encoding circuit 110 described with reference to FIG. 1. Encoding is performed.

[0076] In the quantization using the quantization step size _{Q E} determined in step 812. When the encoding process of the predetermined number of images is completed, finally, step 818
In step 805, the error information is multiplexed after the model information obtained in step 805, and the entire encoding processing of one object ends.

Next, the detailed procedure of the encoding process in step 809 in the flowchart of FIG. 9 will be described with reference to FIG.

First, in step 901, a prediction signal for screen n is obtained by calculation from model information. This obtaining method is exactly the same as the method already described with reference to FIG. Next, the process enters a processing loop of a macro block in the screen from step 902. First, in step 904, the data of the mb-th macroblock is read, and the difference from the prediction signal of the corresponding part is obtained. Next, steps 905, 906, 9
08, DCT, quantization and variable length coding are performed. These processes are performed by the DCT circuit 10 described with reference to FIG.
8. The processing is exactly the same as that performed by the quantization circuit 109 and the variable length coding circuit 110, and is performed by using the hardware or only the software. The data after DCT is stored in step 816.
Stored for reading. In the quantization, the step sizes Q ₁ and Q ₂ are used as described above. Step 907 is through here. When all the macroblocks of one screen have been processed (mb = m
b _last ), the processing of the screen ends. In the encoding process of step 816 in the flowchart of FIG.
Steps 901, 904, and 905 are removed from the procedure in
Step 906 is a process of reading data subjected to DCT processing and then performing quantization. The quantization step size in this case is QE as described above. Then step 9
At 07, the validity of the macroblock is determined, and only the valid macroblock is variable-length coded at step 908.

Next, the processing procedure of the decoding method according to the present invention will be described with reference to FIGS.

This decoding method is a method in which the same operation as that of the decoding device described with reference to FIG. 7 is performed using a computer program. Most of the description is the same as that of the decoding device, and therefore detailed description is omitted. Provide only minimal explanations.

First, data encoded in step 1000 is input. Since the input data has multiplexed model information and prediction error information as described above,
In step 1001, this information is separated. Next, of the separated information, first, the model information is decoded in step 1002 and stored in the storage unit. For decoding, the variable length decoding circuit 607, the inverse quantization circuit 608, and the inverse DCT circuit 609 described in FIG.
The processing is exactly the same as that performed by the software, and is performed by using the hardware or only by the software. Next, in step 1003, position information of the viewpoint is input. Hereinafter, in this loop, processing for creating an image from this viewpoint is performed.

First, a curve 701 on the p-θ plane of the reproduction screen described with reference to FIG. 8 is calculated. This curve is common regardless of the value of Y. Next, it enters a slice loop. In this example, it is assumed that one macroblock line is encoded as one slice, and decoding is performed for each slice.

In the reproduction of the screen 701, a portion corresponding to p different according to the shape of the subject is used as a reference pixel. However, since the Y coordinate refers to the same value, each macroblock line (= slice) is referred to. Is suitable for this property.

Next, at step 1007, a reference pixel position (for example, 70) for each point (for example, 703) of the reproduced image
4) is calculated. Next, in step 1008, a prediction signal is created by projecting the model to the reference pixel position. These procedures are as already described with reference to FIG. Next, in step 1009, step 1006 is performed for each pixel of the reproduced image.
The macroblock address in the image to be referred to is calculated from the information obtained in step (1). At this point, by decoding one macroblock, it is recognized how many points of the reproduced image can be reproduced simultaneously.

As a result, the number of macro blocks mb to be referred to
max is set. From here, the loop of mbmax starts. First, an error is read in step 1012, and the error is decoded in step 1013. The error is decoded by the variable length decoding circuit 604, the inverse quantization circuit 605, the inverse D
The processing is exactly the same as that performed by the CT circuit 606, and is performed by using those hardware or only by software.

Next, in step 104, the prediction signal and the prediction error are added to reproduce the image, and in step 1015, the image is written to the corresponding address of the frame memory. The above processing is performed for all mbmax macroblocks, and the processing for one slice is completed. The processing of generating a reproduced image corresponding to the input viewpoint ends after the processing of all slices ends, and the same processing is repeated again when another viewpoint is input.

Next, the encoding apparatus shown in FIG.
An example of model-related processing used in the encoding method of 0 will be described.

FIG. 14 is a flowchart showing the procedure of model estimation.

First, in step 1100, points on a subject which are characteristic as textures are extracted from images from each viewpoint. As this method, for example, horizontal edge detection is performed on a block having a predetermined size including a point (it is preferable that the size is not too large; for example, 5 * 5 pixels), and an edge having a size equal to or larger than a threshold value is detected. Are extracted as feature points. In step 1101, processing is performed on the obtained feature points in order. It is assumed that the processing is performed from a feature point near p = 0 on the p-θ plane. Next, from step 1102, a process loop for the number of feature points is entered. The estimation of the value of r corresponding to each point is performed by detecting the motion vector (MV) in the above-described block in step 1106, but the search range of the MV at that time is limited in step 1105. The principle of limiting the search range will be described with reference to FIG.

The p-θ of the ray corresponding to the point of interest
The locus on the plane is set to 1201. When the MV is to be detected for the block 1202 in FIG. 15, the point corresponding to θ = ψ ± 90 degrees on this trajectory 1201 can directly see the value of r if the point is visible as shown in FIG. 16. It is a viewpoint that is. Therefore, the value of r is always smaller than the smaller one of the envelopes at this position. Therefore, the MV obtained by converting this value into one screen interval is the maximum value of the range in which the MV is to be searched. As described above, unnecessary calculations can be avoided by using the properties obtained from the multi-viewpoint images.
Next, in step 1106, an MV search is performed between adjacent images. This is continuously performed, as shown in FIG. 17, such that the search is performed on the image adjacent to the feature point k and the search is further performed on the next image from the obtained block. In this case, the range of all the search is step 110
The value obtained in 5 is the upper limit. In addition, when comparing prediction errors in the MV search, if there is not much error difference, a criterion is set so that a point selected as a feature point is preferentially selected. In this step, a skip flag is temporarily set for another feature point k + α associated with the feature point k,
The skip flag is determined when the value of r for this point is determined in a step described later.

In the determination condition of step 1104, the subsequent processing is skipped assuming that the feature point for which the step flag is set corresponds to the point where r has already been determined. Next, in step 1107, the common r between the predetermined number of images or more is set.
It is determined whether an MV that is consistent with the value of is obtained. If “YES” here, it is checked in step 1108 whether or not the value of r fits the pixel corresponding to the edge with respect to the point, fine adjustment is performed on fine precision of r, and then the point is determined in step 1109. The value of r is determined and stored as a set with the corresponding value of ψ. When the above operation is performed by the number of feature points, the value of r is determined for the point having the characteristic texture. Therefore, in step 1111, r of the point on the surface between the points where r is determined is obtained. Since points on this plane have no texture, the values of r obtained by linearly approximating the points between which r has been determined are reconstructed as initial approximations, and the values of r are adjusted and determined by comparing with the image of each viewpoint. . Finally step 1
The entire ray space is reconstructed from the estimated values of the model obtained in 112 and compared for fine adjustment, and the estimation of the model is completed. In the present embodiment, even if the estimation of the model is incorrect to some extent, an error is sent after performing prediction using the model, so that the ray space reconstructed on the decoding / reproduction side does not greatly differ from the actual one. .

As another method for more reliably obtaining shape information as a model estimation, a shape of a subject is not estimated from an input multi-viewpoint image but directly by a 3D measuring device such as a 3D digitizer or a range finder. A method of acquiring shape information may be used. In this case, it takes time and effort in the input phase, but it can be expected that the prediction will be very good.

Next, a method of deciding which viewpoint image information to send model information from the coordinate information of each point on the object plane (= information on the shape of the object) obtained as described above will be described.

FIG. 18 is a diagram showing a section of a subject obtained by the above shape estimation. The cross section is usually represented in the form of coordinates obtained for discretely sampled points.
Consider a surface sandwiched between points A and B in the figure. When these points are converted to polar coordinates, (r ₁ , ψ ₁ ), (r ₂ , ψ
₂ ), and the angle between them is assumed to be Δψ. In this case the angle θAB averaging approximated the normal direction of the surface is obtained by solving _{r 1 cosθ AB = r 2 cos} (θ AB -Δψ). FIG. 19 shows a point A on the p-θ plane.
And the trajectory of the ray corresponding to the point B (1601,
1602). From the above calculation, point A
Surface will be the most resolution 0.00 being encoded on the closest viewpoint image 1603 at a position shifted by theta _AB from [psi ₁ between bets point B. Therefore, on this viewpoint image, an area 1604 surrounded by the locus of point A and the locus of point B is encoded as model information together with the value of r as described above. At this time, as described above, the value of r is encoded with as much precision as can be obtained for each pixel.

In the above embodiment, it has been described that a considerably large number (for example, 120) of images are handled as a multi-viewpoint image. However, this number can be reduced depending on a balance between required image quality and information amount. It is possible. In particular, since the modeling stage in the above-described embodiment is susceptible to errors, it is considered that better performance is obtained by using many images.However, the number of models and viewpoint images sent as errors is limited to modeling. Even if 120 images are used, they are reduced to, for example, 30 images, transmitted, stored, and used for reproduction.
However, as described above, the amount of information to be transmitted may be small in the case of a subject whose model is well represented even if the number of multi-viewpoint images to be transmitted is large. Therefore, reducing the number of images will be effective when the amount of information is to be considerably reduced in a complicated subject that is difficult to hit by the model.

[0096]

As described above, according to the present invention,
By encoding the error between the input multi-view image and the predicted image, the amount of information stored and transmitted as an encoded stream can be sufficiently reduced, and high realism can be achieved regardless of the complexity of the subject. Reproduction can be realized.
Furthermore, since the model information of each point used for generating the predicted image is encoded as luminance information of a viewpoint image used as a representative luminance of the point, spatial correlation such as block encoding is removed. By using such a compression method, model information can also be significantly compressed. For the image portion used as the representative luminance, it is not necessary to use the prediction error information, and that information can be saved. Thus, the amount of information can be reduced while maintaining sufficient model accuracy. Therefore, a multi-viewpoint image can be expressed with a small amount of information while maintaining high texture, and an arbitrary viewpoint image can be reproduced with a reasonable amount of information and high texture.

[Brief description of the drawings]

FIG. 1 is a block diagram showing a configuration of a multi-view image encoding device according to an embodiment of the present invention.

FIG. 2 is an exemplary view for explaining the principle of creating a prediction signal used in the encoding apparatus according to the embodiment;

FIG. 3 is an exemplary view for explaining processing of overlapping light beams in the encoding device according to the embodiment;

FIG. 4 is an exemplary view for explaining the principle of determining a quantization step size by provisional encoding in the encoding device according to the embodiment;

FIG. 5 is a diagram showing a configuration example of a macroblock in the encoding device of the embodiment.

FIG. 6 is an exemplary view showing an example of a multiplex format of an encoded stream created by the encoding device of the embodiment.

FIG. 7 is a block diagram showing a configuration of a multi-view image decoding device according to an embodiment of the present invention.

FIG. 8 is an exemplary view for explaining the principle of a reference pixel calculation operation used in the multi-view image decoding apparatus according to the embodiment.

FIG. 9 is a flowchart showing a part of a procedure of a multi-view image encoding method according to an embodiment of the present invention.

FIG. 10 is an exemplary flowchart showing the remaining part of the procedure of the multi-view image encoding method according to the embodiment.

FIG. 11 is a flowchart illustrating details of an encoding step in FIG. 9;

FIG. 12 is a flowchart showing a part of a procedure of a multi-view image decoding method according to an embodiment of the present invention.

FIG. 13 is a flowchart showing the remaining part of the procedure of the multi-view image decoding method according to an embodiment of the present invention.

FIG. 14 is a flowchart showing a modeling procedure used in the multi-view image encoding apparatus / method according to the embodiment of the present invention;

FIG. 15 is a view for explaining a limitation of a search range in the modeling processing of FIG. 14;

FIG. 16 is a view for explaining a limitation of a search range in the modeling processing of FIG. 14;

FIG. 17 is another diagram illustrating MV detection in the modeling process of FIG. 14;

FIG. 18 is a view for explaining the principle of obtaining a viewpoint image for sending a model in the multi-view image encoding apparatus / method according to one embodiment of the present invention.

FIG. 19 is another diagram for explaining the principle of obtaining a viewpoint image for transmitting a model in the multi-view image encoding apparatus / method according to the embodiment of the present invention;

FIG. 20 is a perspective view showing the configuration of an image input device used in the multi-view image encoding device / method according to one embodiment of the present invention.

21 is a plan view illustrating a relationship between a cross section of the subject and polar coordinates in FIG.

FIG. 22 is a view for explaining the positional relationship on the p-θ plane of the multi-viewpoint image input from the image input device of FIG. 20;

FIG. 23 is a view for explaining the principle of cylindrically recording a light beam space.

[Explanation of symbols]

 100 image input units 101 and 108 storage unit 102 control unit 103 modeling unit 104 model storage units 105 and 611 predicted signal generation units 106 and 612 adders 107 and 111 DCT circuits 109 and 112 quantum ... Decoding circuits 110, 113... Variable length coding circuits 114, 604, 608... Inverse quantization circuits 115, 606, 609... Inverse DCT circuits 116. ... Address calculation circuit 603 Error storage unit 604 607 Variable length decoding circuit 610 Model storage unit 613 Frame memory

Claims (17)

[Claims] 1. A multi-viewpoint image for inputting a plurality of viewpoint images of a subject obtained from a plurality of viewpoints whose positional relationship with the subject is known, and encoding a multi-viewpoint image including the plurality of viewpoint images. An encoding device, comprising: a modeling unit configured to model the multi-viewpoint image to generate model information including coordinate values and luminance values of each point on the object plane, wherein each of the points on the object plane is modeled. Modeling means for determining a viewpoint image to be used as a model, and using corresponding luminance information in the viewpoint image as a luminance value of a corresponding point of the model information, and a coordinate value of each point on the object plane Means for encoding model information composed of the image and the luminance value as image information of a viewpoint image used as a model of each point, and predicting an image of each viewpoint image from the model information Means for encoding a prediction error between the predicted image and each of the input viewpoint images; and multiplexing for multiplexing the encoded model information and the prediction error to generate an encoded stream of the multi-view image. And a multi-view image encoding apparatus. 2. The multi-viewpoint image encoding apparatus according to claim 1, wherein a viewpoint image having a viewpoint closest to a normal direction from a point on the object plane is used as a model of the point. . 3. The multiplexing means according to claim 1, wherein said multiplexing means outputs model information comprising coordinate values and luminance values of respective points on said subject plane.
The multi-view image encoding apparatus according to claim 1, wherein the prediction error information is multiplexed at another information position different from an information position on the encoded stream to be multiplexed. 4. A multi-view image decoding apparatus for decoding a coded stream of a multi-view image including a plurality of viewpoint images of the subject obtained from a plurality of viewpoints whose positional relationship with the subject is known. The encoded stream includes model information including coordinate values and luminance values of points on the subject surface, and predicts an image of each viewpoint image from the model information, and calculates a prediction result and the input The prediction error information obtained by encoding the prediction error with the viewpoint image is multiplexed, and the luminance of each point of the model information corresponds to the corresponding luminance in the viewpoint image used as the model of the point. In a multi-view image decoding device that has been encoded as luminance information, a unit that separates the model information and the prediction error information from the encoded stream; and the encoded stream. Means for decoding the prediction error information separated from the coded stream; decoding means for decoding model information separated from the encoded stream to predict a viewpoint image to be reproduced; and a prediction image obtained by the prediction. Means for combining the image obtained by decoding the prediction error information and the image to be reproduced to reproduce the viewpoint image to be reproduced. 5. A means for inputting position information for designating a viewpoint of an image to be reproduced, based on model information composed of coordinate values and luminance values of respective points on the object plane and the input position information, Means for determining a pixel position to be referred to from the multi-viewpoint image in order to reproduce an image from the specified viewpoint, the prediction image and the prediction error corresponding to each of the determined pixel positions. 5. The multi-view image decoding apparatus according to claim 4, wherein an image from the specified viewpoint is reproduced by synthesizing the decoding result of the information. 6. The encoded stream includes model information composed of coordinate values and luminance values of respective points on the object plane, and an information position on the encoded stream where the prediction error information is multiplexed. The prediction unit accesses the model information of the second information position to generate a predicted image of each of the determined pixel positions, and multiplexes the prediction information with the model information of the second information position. 6. The apparatus according to claim 5, wherein the means for decoding the error information accesses and decodes only the prediction error information necessary for decoding the obtained pixel positions.
The multi-view image decoding device according to the above. 7. The model information of the encoded stream includes:
5. The multi-view image decoding apparatus according to claim 4, wherein a viewpoint image having a viewpoint closest to a normal direction from a point on the object plane is used as a model of the point. 8. A multi-viewpoint image for inputting a plurality of viewpoint images of the subject obtained from a plurality of viewpoints whose positional relationship with the subject is known, and encoding a multi-viewpoint image including the plurality of viewpoint images. In the encoding method, a viewpoint image to be used as a model is determined for each point on the object plane, and the coordinate value and the luminance of each point on the object plane are determined by using corresponding luminance information in the viewpoint image. Generating model information composed of values, encoding the model information composed of coordinate values and luminance values of each point on the object plane as image information of a viewpoint image used as a model of each point, Predicting an image of each viewpoint image from the model information, encoding a prediction error between the predicted image and each input viewpoint image, and calculating the encoded model information and prediction error. And duplexed, multi-view image encoding method and generating a coded stream of the multi-viewpoint image. 9. The multi-view image encoding method according to claim 8, wherein a viewpoint image having a viewpoint closest to a normal direction from a point on the object plane is used as a model of the point. . 10. The model information comprising coordinate values and luminance values of respective points on the object plane is another information position different from the information position on the encoded stream where the prediction error information is multiplexed. 10. The multi-view image encoding method according to claim 8, wherein the multi-view image encoding method is multiplexed together. 11. A multi-view image decoding method for decoding a coded stream of a multi-view image including a plurality of viewpoint images of the subject obtained from a plurality of viewpoints whose positional relationship with the subject is known. The encoded stream includes model information including coordinate values and luminance values of points on the subject surface, and predicts an image of each viewpoint image from the model information, and calculates a prediction result and the input The prediction error information obtained by encoding the prediction error with the viewpoint image is multiplexed, and the luminance of each point of the model information corresponds to the corresponding luminance in the viewpoint image used as the model of the point. In a multi-view image decoding method for decoding a coded stream coded as luminance information, separating the model information and the prediction error information from the coded stream, Decoding the prediction error information separated from the encoded stream, decoding the model information separated from the encoded stream, predicting a viewpoint image to be reproduced, and a prediction image obtained by this prediction and A multi-view image decoding method, comprising: synthesizing an image obtained by decoding prediction error information to reproduce the viewpoint image to be reproduced. 12. Inputting position information for designating a viewpoint of an image to be reproduced, and performing the designation based on model information composed of coordinate values and luminance values of respective points on the object plane and the input position information. A pixel position to be referred to from the multi-viewpoint image in order to reproduce an image from the obtained viewpoint, and for each of the obtained pixel positions, synthesizes the prediction image corresponding thereto and the decoding result of the prediction error information. 12. The multi-view image decoding method according to claim 11, wherein an image from the specified viewpoint is reproduced. . 13. The encoded stream includes model information including coordinate values and luminance values of respective points on the object plane, and an information position on the encoded stream to which the prediction error information is multiplexed. Are multiplexed together in another information position different from the above, the generation of the predicted image of each pixel position is performed by accessing the model information, and the decoding of the prediction error information of each pixel position is performed. 13. The multi-view image decoding method according to claim 12, wherein the method is performed by accessing only prediction error information necessary for decoding the pixel position. 14. A recording medium on which is recorded a program for encoding a multi-view image including a plurality of viewpoint images of the subject obtained from a plurality of viewpoints whose positional relationship with the subject is known. In the program, a viewpoint image to be used as a model is determined for each point on the object plane, and the coordinate value and the luminance value of each point on the object plane are determined by using corresponding luminance information in the viewpoint image. Generating model information composed of: and encoding the model information composed of coordinate values and luminance values of each point on the object plane as image information of a viewpoint image used as a model of each point. A step of predicting an image of each viewpoint image from the model information, and encoding a prediction error between the predicted image and each of the input viewpoint images. The model information and by multiplexing the prediction error, the recording medium characterized in that a procedure for generating an encoded stream of the multi-viewpoint image is described. 15. A recording medium on which an encoded stream of a multi-view image including a plurality of viewpoint images of the subject obtained from a plurality of viewpoints whose positional relationship with the subject is known is recorded. In the stream, the model information composed of the coordinate value and the luminance value of each point on the object plane, and the image of each viewpoint image is predicted from the model information, and the prediction result and each of the input viewpoint images are obtained. The prediction error information obtained by encoding the prediction error is multiplexed, and the luminance of each point of the model information is the corresponding luminance information in the viewpoint image used as a model of the point. A recording medium characterized by being encoded. 16. A multi-viewpoint image for inputting a plurality of viewpoint images of the subject obtained from a plurality of viewpoints whose positional relationship with the subject is known, and encoding a multi-viewpoint image including the plurality of viewpoint images. In the encoding device, a modeling unit configured to model the multi-viewpoint image to generate model information including coordinate values and luminance values of respective points on the subject surface; and Means for predicting an image, encoding a prediction error between the predicted image and each of the input viewpoint images, and means for encoding model information composed of coordinate values and luminance values of each point on the subject plane. Multiplexing means for multiplexing the coded model information and the prediction error to generate a coded stream of the multi-view image. 17. A multi-view image for inputting a plurality of viewpoint images of the subject obtained from a plurality of viewpoints whose positional relationship with the subject is known, and encoding a multi-view image including the plurality of viewpoint images. In the encoding method, a viewpoint image to be used as a model is determined for each point on the object plane, and coordinate values and luminance of each point on the object plane are determined by using corresponding luminance information in the viewpoint image. Generating model information composed of values, encoding model information composed of coordinate values and luminance values of respective points on the object plane, predicting an image of each viewpoint image from the model information, and predicting the image. Encoding a prediction error between an image and each of the input viewpoint images; multiplexing the encoded model information and the prediction error to generate an encoded stream of the multi-view image; Multi-view image encoding method comprising.