JP2023073003A - Rendering device and its program - Google Patents

Abstract

A rendering device capable of generating a natural CG image even from an imprecise model is provided. A rendering device (1) includes a first physics-based rendering unit (10) that applies physics-based rendering with a first virtual illumination, and a second physics-based rendering unit (10) that applies physics-based rendering with a second virtual illumination different from the first virtual illumination. An output CG is obtained from a rendering unit 11, an image-based rendering unit 12 that applies image-based rendering to an image-based model, and pixel values of corresponding pixels of the first physical-based CG, the second physical-based CG, and the image-based CG. and a pixel value calculation unit 13 for calculating the pixel value of the corresponding pixel. [Selection drawing] Fig. 1

Description

The present invention relates to a rendering device and its program.

A physics-based model is a model that models not only the shape of an object but also optical properties such as reflection, refraction, and transmission on surfaces and interfaces, and scattering of light inside. As a rendering method for this physics-based model, physics-based rendering that generates a CG (Computer Graphics) image by simulating the behavior of light rays based on the optical properties with a computer or device is known (for example, patent Reference 1). US Pat. No. 6,200,000 discloses a method of measuring subsurface scattering of the face, etc., to form a skin reflectance model of the face that combines bidirectional reflectance functions, albedo maps, and translucency maps.

Also known is image-based rendering, which synthesizes an image at a virtual viewpoint by performing geometric transformation or interpolation based on a photographed image. In image-based rendering, a technique is known in which, in addition to the actual image, a texture map is used in which the coordinates on the surface of the shape model of the subject are associated with the coordinates on the actual image.

JP 2006-277748 A

Physics-based rendering can produce natural CG images if the shape and optical properties of the subject are modeled with sufficient accuracy. However, when trying to model objects and living things that exist in reality with high precision, sensors are required to measure various physical quantities, and with physics-based rendering, the time and computational costs required for sensing and analysis are enormous. .

For example, when measuring the bidirectional reflectance distribution function (BRDF), light close to a point light source or coded pattern light is irradiated onto the object surface from various angles, and the reflection from various angles is obtained. I need to take a picture. Therefore, a mechanism for changing two or more of the camera position, the light source position, and the object orientation is required, or a plurality of cameras or light sources are required, which increases the scale of the apparatus. Furthermore, in the measurement of the bidirectional reflectance distribution function, acquisition of information is not completed with only one photographing.

In addition, in physics-based rendering, there is a limit to the description of the shapes and optical characteristics of objects and living things in the natural world, and the parts that cannot be described appear as errors in CG. Especially in physically-based rendering, the last step before photo-realistic rendering can lead to the so-called "uncanny valley" of cognitive unnaturalness, which is a barrier.

On the other hand, in image-based rendering, since the image includes the effects of light that cannot be fully represented by the physical model, natural CG generation is possible, and the "uncanny valley" is less likely to occur. However, since image-based rendering is not a technique that strictly models optical properties, it is difficult to change lighting conditions. In particular, in the case of an object with specular reflection, the reflection and gloss change depending on the viewing angle, but image-based rendering basically does not support the expression of such reflection and gloss.

As described above, both physical-based rendering and image-based rendering have advantages and disadvantages, and it is difficult to generate a natural CG image from an imprecise model.

SUMMARY OF THE INVENTION Accordingly, it is an object of the present invention to provide a rendering device capable of generating a natural CG image even from an imprecise model, and a program therefor.

In order to solve the above problems, a rendering device according to the present invention is a rendering device that renders an output CG image from a model of a subject using physically-based rendering and image-based rendering, comprising: a first physically-based rendering unit; The configuration includes a second physical-based rendering section, an image-based rendering section, and a pixel value calculation section.

According to this configuration, the first physics-based rendering section generates the first physics-based CG image by applying physics-based rendering to the physics-based model under the predetermined first virtual lighting conditions.
The second physics-based rendering unit generates a second physics-based CG image by applying physics-based rendering to the physics-based model under a second virtual lighting condition different from the first virtual lighting condition.

The image-based rendering section generates an image-based CG image by applying image-based rendering to the image-based model.
The pixel value calculation unit calculates the pixel value of the corresponding pixel of the output CG image from the pixel value of the corresponding pixel of the first physical base CG image, the second physical base CG image and the image base CG image.

As described above, the rendering device calculates pixel value components that cannot be completely reproduced in the first physical rendering by performing calculations between the image-based CG image and the second physical-based CG image, and complements the first physical-based CG image. . This allows the rendering device to generate natural CG images even from models that are not highly accurate.

The present invention can also be realized by a program for causing a computer to function as the rendering device described above.

According to the present invention, it is possible to generate a natural CG image even from an imprecise model.

1 is a block diagram showing a configuration example of a rendering device according to an embodiment; FIG. In the embodiment, (a) is the first physics-based CG, (b) is the second physics-based CG, (c) is the image-based CG, (d) is the output CG, and (e) is an explanation explaining the expected output. It is a diagram. 1 is an explanatory diagram for explaining the principle of a rendering device in an embodiment; FIG. 2 is a flow chart showing an operation example of the rendering device of FIG. 1;

BEST MODE FOR CARRYING OUT THE INVENTION Hereinafter, embodiments of the present invention will be described with reference to the drawings. However, the embodiments described below are for embodying the technical idea of the present invention, and unless there is a specific description, the present invention is not limited to the following. Further, the same reference numerals may be given to the same means, and the description thereof may be omitted.

(embodiment)
[Configuration example of rendering device]
A configuration example of a rendering device 1 according to an embodiment will be described with reference to FIG.
The rendering device 1 renders output CG (output CG image) from a subject model using physical-based rendering and image-based rendering. As shown in FIG. 1, the rendering device 1 includes a first physically based rendering section 10, a second physically based rendering section 11, an image based rendering section 12, and a pixel value calculating section 13.

The first physics-based rendering unit 10 applies physics-based rendering to the physics-based model under a predetermined first virtual lighting (first virtual lighting condition) to generate a first physics-based CG (first physics-based CG image ). In this embodiment, the first physics-based rendering unit 10 receives a physics-based model, first virtual lighting and virtual camera parameters, and performs a physics-based rendering based on the input physics-based model, first virtual lighting and virtual camera parameters. Rendering is performed to generate the first physics-based CG. The first physics-based rendering unit 10 then outputs the generated first physics-based CG to the pixel value calculation unit 13 .

A physics-based model models not only the shape of an object but also the reflection, refraction, attenuation, and scattering of light on the surface and inside of the object.
The virtual camera parameters are information such as the position, orientation, and angle of view of a virtual camera that virtually captures an object described in the physical base model.

The first virtual lighting is information about the light illuminating the object described in the physics-based model. For example, the first virtual lighting includes the number of light sources, the shape of the light emitting part of the light source such as a point light source, a linear light source, and a surface light source, the color or spectrum characteristics of the light source, the light distribution characteristics of the light source, the optical axis direction of the light source, and the arrangement of the light source. and so on. Also, the first virtual lighting may be described according to specific lighting equipment such as ambient light, directional light, area light, point light, and spotlight. Additionally, the first virtual lighting may be described in accordance with the global rendering of the lighting, such as global illumination.

Here, the first physics-based rendering unit 10 can use known physics-based rendering. For example, physically-based rendering includes parameters and maps such as albedo, specular reflection, roughness, and microfacets. In addition, physically-based rendering may approximate the reflection characteristics on the surface of an object with relatively few parameters, such as Lambert shading and Phong shading.

The second physics-based rendering unit 11 applies physics-based rendering to the physics-based model with a second virtual lighting (second virtual lighting condition) different from the first virtual lighting, thereby rendering a second physics-based CG (second physics-based CG image). In this embodiment, the second physics-based rendering unit 11 receives the physics-based model, the second virtual lighting, and the virtual camera parameters, and converts the physics-based rendering unit 11 into the physics-based model based on the input second virtual lighting and virtual camera parameters. Rendering is performed to generate a second physics-based CG. The second physics-based rendering unit 11 then outputs the generated second physics-based CG to the pixel value calculation unit 13 .

Here, the information (virtual lighting) regarding the light that illuminates the object differs between the first physical-based rendering unit 10 and the second physical-based rendering unit 11 . That is, the first physics-based rendering unit 10 and the second physics-based rendering unit 11 apply the same physics-based rendering with the same virtual camera parameters to the same physics-based model.

In the present embodiment, the first virtual illumination is the illumination condition in the output CG (output CG image) to be generated finally, that is, the illumination condition of the output CG finally output by the rendering device 1 .
The second virtual illumination is an approximation of the illumination conditions under which the image-based model, which will be described later, was generated. For example, the second virtual lighting may be approximated as ambient light when the image-based model is generated by illuminating the subject from various azimuths and elevation angles using a plurality of lighting devices. Also, for example, if the image-based model is generated by illuminating the image-based model with a relatively small number of lighting devices (for example, 10 or less), the second virtual lighting can be approximated by the same number and arrangement of area lights and point lights as the lighting devices. You may Note that both the first virtual illumination and the second virtual illumination are manually set, and the reason for the difference between the two will be described later.

The “approximation” of the second virtual illumination includes, for example, the number of light sources, the shape of the light emitting part of the light source such as a point light source, a linear light source, and a surface light source, the color or spectral characteristics of the light source, the light distribution characteristics of the light source, the light source It refers to approximating all or part of the optical axis direction, light source arrangement, etc. to the lighting conditions when the image-based model was generated. Of course, "approximating" the second virtual lighting includes making it identical to the lighting conditions under which the image-based model was generated.

The image-based rendering unit 12 applies image-based rendering to the image-based model to generate image-based CG (image-based CG image). In this embodiment, the image-based rendering unit 12 receives an image-based model and virtual camera parameters, performs image-based rendering on the image-based model based on the input virtual camera parameters, and generates image-based CG. The image-based rendering section 12 then outputs the generated image-based CG to the pixel value calculation section 13 .

Here, the image-based rendering unit 12 can use known image-based rendering. For example, based on the virtual camera parameters, the image-based rendering unit 12 performs a method of transforming a photographed image, a method of synthesizing a plurality of photographed images (including deformed images) by blending, mosaicing, multiple-frame super-resolution, or the like. An image-based CG is generated by appropriately combining methods such as projecting a model generated based on an image.

An image-based model is a model based on a photographed image taken with a plurality of illumination lights, shadowless lights, or planar illumination lights.
The method of transforming the photographed image, the method of synthesis, the method of projecting the image-based model generated from the photographed image, and the pixel value information or texture information extracted from the photographed image are called image-based CG.

For example, the image-based rendering unit 12 may use an image-based model including the shape model and a texture map that associates coordinates on a shape model such as polygons with image coordinates of a photographed image. In this case, a photographed image taken from one viewpoint may be used, or a series of images taken from a plurality of viewpoints may be used, and the image to be used may be changed according to the location on the shape model.

Further, the image-based rendering unit 12 may generate image-based CG by, for example, using a point cloud and assigning pixel values of a photographed image to each point constituting the point cloud. Also in this case, the actual image may be an image shot from one viewpoint, or an image sequence shot from a plurality of viewpoints may be used and the image used may be changed according to the points forming the point cloud.

Also, the image-based rendering unit 12 may generate an intermediate-viewpoint image-based CG by interpolating a sequence of images taken at a plurality of viewpoints. In this case, the pixel value calculator 13, which will be described later, may perform local correspondence between the images forming the image sequence, and multiply this correspondence vector by a predetermined multiple to determine the pixel value of the intermediate viewpoint.

Furthermore, the image-based rendering unit 12 performs local correspondence between the two or more images using a series of images shot from a plurality of viewpoints, and sets the camera parameters of the camera that shot each of the two or more images. The three-dimensional coordinates of the locally associated points may be identified by triangulation based on the above. In this case, the pixel value calculator 13 may project the specified three-dimensional coordinates using desired camera parameters to specify the corresponding pixel coordinates in the output CG. Further, of the two or more images used when obtaining the specified three-dimensional coordinates, using the pixel values on the corresponding points of one or more images (for example, by the average value), the "output CG You may determine the pixel value in "corresponding pixel coordinates".

The pixel value calculation unit 13 receives the first physics-based CG input from the first physics-based rendering unit 10, the second physics-based CG input from the second physics-based rendering unit 11, and the image-based rendering unit 12. The pixel value of the corresponding pixel of the output CG is calculated from the pixel value of the corresponding pixel of the image-based CG image.

<Calculation by Pixel Value Calculator>
Calculations by the pixel value calculator 13 will be described in detail below.
Pixels at the same image coordinates (x, y) in the first physical base CG, the second physical base CG, the image base CG, and the output CG are defined as corresponding pixels. Further, the pixel value P(x, y) of the corresponding pixel of the first physical base CG, the pixel value Q(x, y) of the corresponding pixel of the second physical base CG, the pixel value R(x , y), and the pixel value S(x, y) of the corresponding pixel of the output CG. Note that the pixel values may be scalar (eg, grayscale image) or vector (eg, color image consisting of red, green, and blue components).

Here, from the pixel value P of the corresponding pixel of the first physical base CG, the pixel value Q of the corresponding pixel of the second physical base CG, and the pixel value R of the corresponding pixel of the image base CG, As shown in the following equation (1), the pixel value of the corresponding pixel of the output CG is calculated using the binary operation μ.

For example, the pixel value calculator 13 converts the binary operator μ and the function f according to the following equation (2) to calculate the pixel value S(x, y) of the corresponding pixel of the output CG.

Note that Q ⁻¹ is the inverse of Q in the binary operation μ. Q ⁻¹ means the reciprocal of each vector component when the binary operation μ is the Hadamard product and Q is a D-dimensional vector (where D is an integer equal to or greater than 1). That is, Q ⁻¹ is represented by the following formula (3). In particular, if D=1, ie if the binary operation μ is a remainder calculation and Q is a scalar, then Q ⁻¹ means 1/Q. On the other hand, when the binary operation μ is addition and subtraction, Q ⁻¹ means Q.

The function f(v) is assumed to be independent for each component as shown in the following equation (4), and the function f _d (v) is assumed to increase monotonically in a broad sense with respect to v (d is an integer equal to or greater than 1 and equal to or less than D ).

Also, the function f(v) may be the following equation (5). In this case, the formula (1) is represented by the following (6).

Here, the binary operation μ of equation (1) can be performed by addition and subtraction of equation (7) or remainder calculation of equation (9). For example, the pixel value calculator 13 calculates the pixel value of the corresponding pixel of the output CG using the binary operation μ, which is addition and subtraction, as shown in Equation (7) below. That is, the pixel value calculator 13 performs the calculation represented by the following equation (8) to calculate the pixel value S(x, y) of the corresponding pixel of the output CG.

In addition, the pixel value calculation unit 13 uses the binary operation μ, which is a remainder calculation, as shown in the following equation (9) for pixels and color components in which the component of Q is not zero, and the correspondence of the output CG A pixel value of a pixel may be calculated. That is, the pixel value calculator 13 performs the calculation represented by the following equation (10) to calculate the pixel value S(x, y) of the corresponding pixel of the output CG.

Here, when the d-th components of P, Q, R and S are respectively _Pd , _Qd , _Rd and _Sd , the pixel value _Sd is calculated by the following equation (11).

Exceptionally, for the component of Q _d (x, y)=0, the pixel value S _d is calculated by any of the following equations (12) to (15) (where m and n are positive integer).

<Principle of Rendering Device>
The principle of the rendering device 1 will be described with reference to FIGS. 2 and 3. FIG.
Here, an example of CG rendering of a sphere with a pattern (for example, a billiard ball) will be described. Also, the accuracy of physical-based rendering in the first physical-based rendering unit 10 and the second physical-based rendering unit 11 is inferior to the accuracy of image-based rendering in the image-based rendering unit 12 . Also, when the image-based model is generated, the object is illuminated with a uniform light beam from the entire periphery.

FIG. 2(a) shows the first physical base CG. In this first physics-based CG, although an image with shadows and glossiness is obtained by the first physics-based rendering, blurring occurs due to the lack of accuracy of the physics-based model.

FIG. 2(b) shows the second physical base CG. In this second physics-based CG, in order to approximate the lighting at the time of acquiring the image-based model, a virtual illumination that irradiates a uniform light beam from the entire periphery of the subject is used. As a result, the second physics-based CG renders an image in which only the pattern of the subject appears without shadows or glossiness. Also, in the second physics-based CG, blurring occurs due to insufficient accuracy of the physics-based model.

FIG. 2(c) shows the image-based CG. This image-based CG is CG generated from a photographed image captured by illuminating the subject with uniform light rays from all around. In image-based CG, an image without shadows or glossiness is rendered, while the image-based model is more precise than the physical-based model, as described above, so that a high degree of detail can be obtained.

FIG. 2(d) shows the output CG calculated by Equation (6) for the GCs of FIGS. 2(a) to 2(c). This output CG is of high quality with shading, luster, and no blurring. This output CG approximates the CG obtained when the accuracy of the physics-based model is sufficiently high, as shown in FIG. 2(e).

As shown in FIG. 3, in the first physically-based CG, while the shadows of the output CG under the desired illumination are reproduced, texture shape deterioration due to shape errors increases. Image-based CG reproduces the shadows of the image-based model while reducing texture shape deterioration due to shape errors. In the second physics-based CG, the shadows of the image-based model are reflected, and texture shape deterioration due to shape errors increases.

Therefore, in the rendering device 1, by adding the first physical base CG and the image base CG and subtracting the second physical base CG, the shadow under the desired illumination is reproduced and the texture shape is deteriorated due to the shape error. An output CG with less is obtained. As described above, according to the rendering apparatus 1, even when the performance of physically-based rendering is inferior to the performance of image-based rendering, it is possible to generate CG that achieves both texture and detail.

[Example of operation of rendering device]
An operation example of the rendering device 1 will be described with reference to FIG.
As shown in FIG. 4, in step S1, the first physics-based rendering unit 10 generates the first physics-based CG by applying physics-based rendering to the physics-based model with the first virtual lighting.
In step S2, the second physics-based rendering unit 11 generates a second physics-based CG by applying physics-based rendering with a second virtual illumination to the physics-based model.
In step S3, the image-based rendering unit 12 generates image-based CG by applying image-based rendering to the image-based model.
Note that the processing of steps S1 to S3 can be executed in parallel.

In step S4, the pixel value calculation unit 13 calculates the correspondence of the output CG from the pixel values of the corresponding pixels of the first physical base CG in step S1, the second physical base CG in step S2, and the image base CG in step S3. Calculate the pixel value of a pixel.

[Action/effect]
As described above, the rendering apparatus 1 calculates pixel value components that cannot be completely reproduced by the first physical rendering through calculations between the image-based CG and the second physical-based CG, and complements the first physical-based CG. This allows the rendering device 1 to generate a natural CG image even from a model that is not highly accurate.
Furthermore, since the rendering device 1 can obtain a realistic output CG even if the accuracy of the physics-based model is reduced, it is possible to reduce equipment and processing time required for modeling and the amount of model data.

That is, the rendering device 1 complements the pixel value components that cannot be reproduced in the first physically based CG with the different components between the image based CG and the second physically based CG. Thus, in the rendering device 1, when the image-based CG has finer texture components than the second physics-based CG, the difference component can be complemented to the first physics-based CG, thereby improving the sense of detail.

Furthermore, the rendering device 1 uses subtraction for the calculation of the difference component and addition for complementation. In this case, the rendering device 1 can improve the scale and processing speed of the device because the computation cost of the difference computation is low.

Furthermore, the rendering device 1 uses division for the calculation of the difference component and multiplication for complementation. In this case, the rendering device 1 generally applies these effects multiplicatively in the processes of reflection on the object surface and scattering, reflection, and transmission inside the object. By performing multiplication and division, it is possible to perform more physically strict complementation.

Further, the rendering device 1 can approximate the shadow of the second physical base CG to the shadow of the image base CG by approximating the second virtual lighting to the lighting environment at the time of generating the image base model, and the shadow can remove the effects of As a result, the rendering device 1 can make the complemented information specialize in information related to fineness, such as the presence or absence of detailed structures.

Furthermore, since the rendering device 1 uses an image-based model based on a photographed image taken with a plurality of illumination lights, a shadowless lamp, or planar illumination light, it is possible to generate an image-based CG with reduced influence of the illumination light. becomes. In this case, in the pixel value calculation unit 13, by performing calculations between the image-based CG in which the influence of the illumination light is reduced and the second physical-based CG having the same shadow as the shadow of the image-based CG, the illumination light It is possible to perform highly accurate complementation that eliminates the adverse effects that depend on , and improve the image quality of the output CG.

(Modification)
Although the embodiments have been described in detail above, the present invention is not limited to the above-described embodiments, and includes design changes and the like within the scope of the present invention.

In the above-described embodiment, the first virtual lighting is the lighting conditions in the output CG, and the second virtual lighting is the approximation of the lighting conditions when the image-based model is generated. However, the present invention is not limited to this. . That is, if the second virtual illumination is different from the first virtual illumination, both illumination conditions can be arbitrarily set. For example, by intentionally changing the second virtual lighting, special video effects such as edge enhancement and strong gloss can be obtained.

In the above-described embodiment, the first physical-based rendering section and the second physical-based rendering section are configured independently, but they may be shared. In other words, both are implemented as a common circuit or program, the first virtual illumination and the second virtual illumination are switched at different timings, physical base rendering is performed, and the result is the first physical base CG and the second physical base CG. may be

In the above-described embodiment, the rendering device is described as independent hardware, but the present invention is not limited to this. For example, the present invention can also be realized by a program for causing hardware resources such as a CPU, memory, and hard disk provided in a computer to function as the above-described rendering device. This program may be distributed via a communication line, or may be distributed by being written in a recording medium such as a CD-ROM or flash memory.

1 Rendering Device 10 First Physically Based Rendering Unit 11 Second Physically Based Rendering Unit 12 Image Based Rendering Unit 13 Pixel Value Calculating Unit

Claims (7)

A rendering device that renders an output CG image from a model of a subject using physically-based rendering and image-based rendering,
a first physics-based rendering unit that generates a first physics-based CG image by applying the physics-based rendering to the physics-based model under a predetermined first virtual lighting condition;
a second physics-based rendering unit that generates a second physics-based CG image by applying the physics-based rendering to the physics-based model under a second virtual lighting condition different from the first virtual lighting condition;
an image-based rendering unit that generates an image-based CG image by applying the image-based rendering to the image-based model;
a pixel value calculation unit for calculating the pixel value of the corresponding pixel of the output CG image from the pixel value of the corresponding pixel of the first physical base CG image, the second physical base CG image, and the image base CG image;
A rendering device comprising: the first virtual lighting condition is a lighting condition in the output CG image;
3. The rendering apparatus of claim 1, wherein the second virtual lighting conditions approximate lighting conditions under which the image-based model was generated. The pixel value calculation unit calculates a pixel value P of the corresponding pixel of the first physically-based CG image, a pixel value Q of the corresponding pixel of the second physically-based CG image, and a pixel value of the corresponding pixel of the image-based CG image. From R, as shown in the following equation (1), using the binary operation μ (where Q ⁻¹ is the inverse of Q in the binary operation μ),

3. The rendering device according to claim 1, wherein the pixel value of the corresponding pixel of the output CG image is calculated. The pixel value calculator uses the binary operation μ, which is addition and subtraction, as shown in the following equation (2):

4. The rendering device according to claim 3, wherein the pixel value of the corresponding pixel of said output CG image is calculated. The pixel value calculator uses the binomial operation μ, which is a remainder calculation, for pixels and color components in which the component of Q is not zero, as shown in the following equation (3):

4. The rendering device according to claim 3, wherein the pixel value of the corresponding pixel of said output CG image is calculated. 6. The image-based model according to any one of claims 1 to 5, wherein the image-based model is a model based on a photographed image taken with a plurality of illumination lights, a shadowless lamp, or planar illumination light. rendering equipment. A program for causing a computer to function as the rendering device according to any one of claims 1 to 6.

Images