KR100939080B1 - Synthetic image generation method and apparatus, display method and apparatus using synthetic image - Google Patents

Abstract

Disclosed are a method and apparatus for generating a composite image, and a display method and apparatus using a composite image. According to another exemplary embodiment, a display apparatus includes a first display unit or a main object that outputs either a composite image or a backlight; And a second display unit for outputting the main image photographed, and a lens unit for restoring the synthesized image to a stereoscopic image. The present invention has the effect of providing a complex 3D image display apparatus combining a floating image method and an integrated image.

Integrated image, floating image, display

Description

Method and Apparatus for generating composited image, Method and Apparatus for generating composited image, Method and Apparatus for displaying using composited image}

The present invention relates to a display apparatus, and more particularly, to a method and apparatus for generating a composite image, and a display method and apparatus using a composite image.

Recently, the demand for 3D image has been increased due to the advancement of display technology and the high level of technology integration. Accordingly, many studies on 3D stereoscopic image and display technology have been conducted. 3D display technology can be said to be the ultimate image realization technology in that the actual image information of the object can be shown to the viewer.

On the other hand, the display device can be briefly classified into viewing and daily work according to the purpose. Display devices used in movies and animations, TVs, game consoles and advertisements are for viewing. Display devices used for reading and writing documents and e-mails, 2D graphic work, and searching the Internet are suitable for daily business use. The application of 3D display technology to the viewing video apparatus meets the needs of users who desire a more realistic display system. By the way, users who perform daily work tasks are accustomed to the 2D display-based screen having a high resolution. Therefore, there is a need for a display technology capable of switching between 2D image output mode and 3D image output mode according to the purpose of use of the individual. In addition, a lot of research has been conducted on the display technology capable of 2D-3D compatibility.

As a display technology capable of using 2D-3D in the related art, there is a 2D-3D display based on stereoscopy. In this method, the observer sees the 3D stereoscopic image by the action of stereo disparity, but there is a disadvantage in that dizziness and eye fatigue occur due to disparity of the two images and inconsistency of the focus function of the eyes.

In the 2D-3D switchable display method, a lot of studies have been conducted to apply an integrated imaging (InIm) method in order to solve the above-mentioned problems of the prior art. The integrated image method is one of the most actively studied 3D display methods in recent years, and it enables to view 3D images having continuous viewpoints and full colors within a certain viewing angle range without using special assistive devices. However, this method requires many improvements in view angle and depth.

In addition, stereoscopic-based 2D-3D display and integrated image technology have one common limitation. More specifically, the resolution of the 3D image implemented by the display device having the same performance is smaller in inverse proportion to the number of view points when compared to the resolution of the existing 2D image. That is, assuming that the number of viewpoints is N, the resolution of the 3D image implemented by the multi-view method is lowered at a ratio of 1 / N compared to the resolution of the 2D image. In the case of a 3D integrated image implemented with N ㅧ N lens arrays (or element images), the resolution drops to 1 / (N ㅧ N). As such, the stereoscopic-based 2D-3D display method or integrated image has a problem in that the resolution decreases rapidly depending on the number of viewpoints.

Recently, in order to display a 3D image having a high resolution, a new 3D image display method incorporating an integrated image technology in a floating image display technology has been proposed. However, in general, the floating image implemented on one image plane has difficulty in improving stereoscopic effect due to the incompleteness of the 2D image in principle. In addition, occlusion region problems (i.e., translucence or overlap, observation of invalid region) occur between the front and back images in systems using only two 2D image planes.

The present invention proposes a complex 3D image display apparatus combining a floating image method and an integrated image.

The present invention proposes an apparatus for representing a high resolution primary image by a floating image method and displaying a secondary image or a background image by an integrated image method.

According to an aspect of the invention, the first display unit for outputting any one of a composite image or a backlight; A second display unit configured to output a main image photographed by a main object; And a lens unit for reconstructing the synthesized image into a stereoscopic image, wherein the main image is a two-dimensional image, and the synthesized image is a white image having the same shape and size as the main image and an integrated image composed of element images not corresponding to the main object. the display device, characterized in that a zero-phase synthesis an image is provided.

The display apparatus may further include a controller configured to control the first display unit to output one of a background image and a backlight according to an output mode. The output mode may be one of a two-dimensional mode and a three-dimensional mode.

The second display unit may be a transmissive spatial light modulator (SLM).

The lens unit may include any one of a lens array or a lenslet array or a micro lens array.

According to another aspect of the present invention, a display method of a display device having a first display unit and a second display unit, comprising: (a) outputting one of a composite image or a backlight by the first display unit; (b) the second display unit outputting a main image of the main object photographed; And (c) restoring the synthesized image to a stereoscopic image, wherein the main image is a two-dimensional image, and the synthesized image is composed of an element image that does not correspond to the main object, and has a shape and a size of the main object. A display method is provided which is an image obtained by synthesizing an image of the same white color.

Step (a) is a step of outputting the background image or the backlight according to the output mode, the output mode may be any one of the two-dimensional mode or three-dimensional mode.

The background image may be a composite image in which the pixel corresponding to the main object of the pixels of the element image corresponding to the background is changed to white.

According to another aspect of the invention, the lens unit including a plurality of lenses; A sensor unit for photographing an object using the lens to generate a mask image and a background image; And a synthesizer configured to generate a synthesized image by masking a background image with the mask image.

The mask image may be a binary image of an element image of a main object.

The background image may be an integrated image including element images of a background other than a main object.

According to another aspect of the invention, generating a mask element image and a background image through a lens array (lens array) or a lenslet array (lenslet array); There is provided a composite image generating method comprising generating a composite image by masking the background image with the mask image.

The masking may be to change the pixel of the background image corresponding to the white pixel of the mask image to white.

The mask image may be a binary image of an integrated image composed of element images of a main object.

The background image may be an integrated image including element images of a background other than a main object.

The present invention has the effect of providing a complex 3D image display apparatus combining a floating image method and an integrated image.

The present invention has the effect of providing a device for displaying a primary image of a high resolution in a floating image method, and displays a secondary image or a background image having a full parallax in an integrated image method.

The present invention has the effect of improving the low resolution of the integrated image system by using a high resolution floating image.

The present invention has the effect of expressing a 3D image having a high resolution of 2D level only in the main image area represented by the floating image.

The present invention has the effect of improving the three-dimensional effect of the floating image by using the integrated image background.

As the invention allows for various changes and numerous embodiments, particular embodiments will be illustrated in the drawings and described in detail in the written description. However, this is not intended to limit the present invention to specific embodiments, it should be understood to include all transformations, equivalents, and substitutes included in the spirit and scope of the present invention. In the following description of the present invention, if it is determined that the detailed description of the related known technology may obscure the gist of the present invention, the detailed description thereof will be omitted.

Terms such as first and second may be used to describe various components, but the components should not be limited by the terms. The terms are used only for the purpose of distinguishing one component from another.

The terminology used herein is for the purpose of describing particular example embodiments only and is not intended to be limiting of the present invention. Singular expressions include plural expressions unless the context clearly indicates otherwise. In this application, the terms "comprise" or "have" are intended to indicate that there is a feature, number, step, operation, component, part, or combination thereof described in the specification, and one or more other features. It is to be understood that the present invention does not exclude the possibility of the presence or the addition of numbers, steps, operations, components, components, or a combination thereof.

Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings.

1 is a diagram illustrating a pickup process of an integrated image method, FIG. 2 is a diagram illustrating a real image restoration process of a 3D image, and FIG. 3 is a diagram illustrating a virtual image restoration process of a 3D image.

Referring to FIG. 1, light rays emitted from the 3D object 110 pass through a lens array 120 and then reach the image sensor 130. The image sensor 130 may generate one element image for each lens of the lens array 120 by detecting the rays.

In this case, the lens array 120 may be a lenslet array, a micro lens array, or an MLA.

Element images may be reconstructed into three-dimensional images through a reconstruction process. For example, when the focal length of one lens is f, the distance between the lens array 220 and the display 210 is g, and the distance between the lens array 220 and the object is L, the lens formula 1 / L According to + 1 / g = 1 / f, a real image is projected when g≥f as shown in FIG. 2, and a virtual image is observed by an observer when g≤f as shown in FIG. If g = f, several 3D images having different depths, including real and virtual images, can be simultaneously displayed. Hereinafter, a two-dimensional and three-dimensional display device using the above-described element image will be described in detail with reference to FIGS. 4 to 8.

4 is a view for explaining the configuration of a display device according to an embodiment of the present invention, Figure 5 is a view illustrating the structure of a display device according to an embodiment of the present invention, Figure 6 is an embodiment of the present invention FIG. 7 is a view for explaining the principle of operation of the display device in the two-dimensional output mode, FIG. 7 is a view for explaining the principle of operation of the background screen display screen in the three-dimensional output mode according to an embodiment of the present invention, FIG. 8 is a diagram illustrating an operation principle of a display apparatus displaying a virtual background screen in a 3D output mode according to an exemplary embodiment of the present invention.

Referring to FIG. 4, a display device according to an exemplary embodiment of the present invention may include a controller 410, a first display unit 420, a second display unit 430, and a lens unit 440.

The controller 410 may control the first display unit 420 and the second display unit 430 to display an image according to an output mode (ie, a 2D output mode or a 3D output mode).

For example, in the 2D output mode, the controller 410 may output 2D image data to the second display unit 430, and control the second display unit 430 to display the 2D image data. In this case, the controller 410 may output a backlight signal to the first display unit 420 to control the first display unit 420 to maintain all the pixels in an 'on' state to serve as a light source (backlight). .

The controller 410 controls the first display unit 420 and the second display unit 430 to display the composite image and the main image, respectively, in the 3D output mode. The main image is a two-dimensional representation of an image corresponding to a main object among images to be observed by an observer. The composite image is composed of an integrated image composed of element images that do not correspond to a main object (hereinafter referred to as a background image) and a white image of the same shape and size as the main object (hereinafter referred to as a "mask image"). As an image, it will be described in detail later with reference to FIG. 9. At this time, the portion corresponding to the white image of the above-described composite image serves as a backlight function of the main image.

In this case, the controller 410 may receive the composite image from the outside and control the first display 420 to output the synthesized image. In addition, the controller 410 may synthesize a white image and a background image to generate and output a synthesized image. A process of generating a synthesized image will be described in detail with reference to FIG. 9.

In addition, the controller 410 outputs the main image to the second display unit 430 and controls the second display unit 430 to display the main image. The main image has the resolution of a normal 2D image. As described above, an observer may view a new high resolution composite stereoscopic image as a whole by using the composite image displayed by the first display unit 420 and the main image displayed by the second display unit 430.

The first display unit 420 may display the composite image received from the controller 410. In addition, when the first display unit 420 receives a backlight signal, the first display unit 420 may maintain a 'on' state to serve as a light source (backlight). The first display unit 420 may be a two-dimensional display device such as an LCD, a PDP, or a CRT, or may be a projection display device in which a projector and a screen are combined.

The second display unit 430 may display a main image received from the controller 410. The second display unit 430 may be an LCD panel in which a back light unit (BLU) is removed by a transmissive spatial light modulator (SLM). In this case, since the second display unit 430 does not have its own backlight function, the first display unit 420 may serve as a backlight according to an exemplary embodiment of the present invention. In addition, the second display unit 430 may display a 2D image received from the controller 410.

The lens unit 440 may be located between the first display unit 420 and the second display unit 430. The lens unit 440 may restore the integrated image projected by the first display unit 420 into a stereoscopic image. The lens unit 440 may be any one of a lens array or a lenslet array or a micro lens array.

As illustrated in FIG. 5, the above-described first display unit 420 and the second display unit 430 may be continuously arranged in a line, and the lens unit 440 may be positioned between the two display units. Therefore, in the 3D output mode, the observer may recognize the background image displayed through the first display unit 420 and the lens unit 440 through the second display unit 430 which is a transmissive display device. In addition, the observer may recognize a main image displayed by the second display unit 430 as a background image to feel a 3D effect.

In addition, as illustrated in FIG. 6, in the two-dimensional output mode, the observer may recognize the two-dimensional image displayed by the second display unit 430 by using the backlight output from the first display unit 420. .

The background screen may be formed on the real surface as illustrated in FIG. 7 through the first display unit 420 and the lens unit 440, or may be formed on the virtual image surface as illustrated in FIG. 8. In this case, when the background image is formed on the virtual image, the stereoscopic feeling of the entire 3D image may be improved.

9 is a view for explaining the configuration of the apparatus for generating a composite image according to an embodiment of the present invention, Figure 10 is a view for explaining a process of generating a composite element image according to an embodiment of the present invention.

Referring to FIG. 9, an apparatus for generating a composite image may include a lens unit 910, a sensor unit 920, and a synthesis unit 930.

The lens unit 910 is arranged by a plurality of lenses (lens) or lenslets (pickles) for picking up the element image of the object. The lens unit 910 may collect and output light to the sensor unit 920 through each lens or lenslet.

The sensor unit 920 detects light collected through the lens unit 910, generates an element image corresponding to each lens or lenslet of the lens unit 910, and outputs the element image to the synthesis unit 930. can do. In this case, the sensor unit 920 may separately generate a mask image and a background image. The mask image represents an integrated image composed of element images of main objects as a binary image, and serves as a mask of a background image. The background image is an integrated image composed of element images of a background other than a main object. The synthesis unit 930 may synthesize a mask image and a background image received from the sensor unit 920. For example, the synthesis unit 930 may mask a mask image with respect to a background image. That is, the synthesis unit 930 may change the pixel at the position corresponding to the white pixel of the mask image to white in the background image. 10 illustrates a composite image obtained by synthesizing a background image and a mask image according to an embodiment of the present invention. Subsequently, the synthesizer 930 outputs the synthesized image to an external device.

Hereinafter, a stereoscopic image display method using a direct image according to an embodiment of the present invention will be described with reference to FIG. 11.

11 is a flowchart illustrating a stereoscopic image display process using a direct image according to an embodiment of the present invention.

Hereinafter, the present invention will be described with reference to FIG. 11, but for the convenience of understanding and explanation of the present invention, the functional units illustrated in FIG. 4 will be clearly described.

Referring to FIG. 11, in operation 1105, the controller 410 determines whether an output mode of an input image is a 3D output mode. The controller 410 may determine whether the input image is the 3D mode by referring to at least one of a header file of the input image, a user's setting, and / or a default setting.

If the input image is in the 3D mode, in operation 1110, the first display unit 420 outputs the synthesized image in a form that the user can visually recognize.

In operation 1120, the lens unit 440 restores the displayed composite image through the lens array. At this time, the reconstructed image may be formed on the real plane or the virtual plane. If the reconstructed image is formed on the virtual image, the stereoscopic feeling of the entire 3D image may be improved.

In operation 1130, the second display unit 430 outputs the main image in a form that the user can visually recognize.

If not in the 3D mode, in operation 1140, the first display unit 420 maintains all pixels in an 'on' state to perform a backlight function.

In operation 1150, the second display unit 430 outputs the main image in a form that the user can visually recognize.

Hereinafter, a method of generating a composite image which is restored through the first display unit 420 and the lens unit 440 and used as a background will be described.

12 is a flowchart illustrating a process of generating a composite image according to an embodiment of the present invention.

9 and 12, in operation 1210, the sensor unit 920 generates a mask image through a lens array (or lenslet array) of the lens unit 910. The mask image is a binary image and serves as a mask of the background image.

In operation 1220, the sensor unit 920 generates a background image through the lens array (or lenslet array) of the lens unit 910.

In operation 1230, the synthesis unit 930 generates a composite image by masking a background image using a mask image. As a result, the synthesized image is an image obtained by converting an object part of the mask image into a white image in the background image. Later, these white pixels will serve as the backlight for the main image.

Although the above steps 1210 to 1220 have been described as being sequentially performed, it is obvious that the steps 1210 to 1220 may be performed in parallel according to an implementation method.

Although the above has been described with reference to a preferred embodiment of the present invention, those skilled in the art to which the present invention pertains without departing from the spirit and scope of the present invention as set forth in the claims below It will be appreciated that modifications and variations can be made.

1 is a diagram illustrating a pickup process of an integrated image method.

2 is a diagram illustrating a real image restoration process of a 3D image.

3 is a diagram illustrating a virtual image restoration process of a 3D image.

4 is a view for explaining the configuration of a display device according to an embodiment of the present invention.

5 is a diagram illustrating a structure of a display device according to an embodiment of the present invention.

6 is a view for explaining the principle of operation of the display device in the two-dimensional output mode according to an embodiment of the present invention.

7 is a view for explaining the operation principle of the display device to the actual background screen in the three-dimensional output mode according to an embodiment of the present invention.

8 is a view for explaining the principle of operation of the display device a virtual background screen in the three-dimensional output mode according to an embodiment of the present invention.

9 is a view for explaining the configuration of an apparatus for generating a composite image according to an embodiment of the present invention.

10 is a view for explaining a process of generating a composite element image according to an embodiment of the present invention.

11 is a flowchart illustrating a stereoscopic image display process using a direct image according to an embodiment of the present invention.

12 is a flowchart illustrating a process of generating a composite image according to an embodiment of the present invention.

Claims (14)

A first display unit configured to output one of a composite image and a backlight; A second display unit configured to output a main image photographed by a main object; And It includes a lens unit for restoring the composite image to a stereoscopic image, The main image is a two-dimensional image, The synthesized image is a display device comprising a composite image composed of an element image not corresponding to a main object and an image of a white image having the same shape and size as the main object. According to claim 1, According to an output mode further comprises a control unit for controlling the first display unit to output any one of the composite image or the backlight, The output mode is a display device, characterized in that the two-dimensional mode or three-dimensional mode. According to claim 1, And the second display unit is a transmissive spatial light modulator (SLM). According to claim 1, The lens unit may include any one of a lens array or a lenslet array or a micro lens array. In the display method of the display device having a first display unit and a second display unit, (a) outputting, by the first display unit, either a composite image or a backlight; (b) outputting a main image of the main object by the second display unit; And (c) restoring the synthesized image to a stereoscopic image, The main image is a two-dimensional image, The synthesized image is a display method comprising a composite image composed of an element image not corresponding to a main object and an image of a white image having the same shape and size as the main object. The method of claim 5, Step (a) is the step of outputting the composite image or the backlight according to the output mode, And the output mode is a two-dimensional mode or a three-dimensional mode. The method of claim 6, And wherein the composite image is a composite image obtained by changing a pixel corresponding to a main object of a pixel of an element image corresponding to a background to white. A lens unit including a plurality of lenses; A sensor unit for photographing an object using the lens to generate a mask image and a background image; And And a synthesizer configured to change the pixel of the background image corresponding to the position corresponding to the white pixel of the mask image to white to generate a synthesized image. The mask image is a composite image generating device characterized in that the elementary image for the main object represented as a binary image (binary image). delete The method of claim 8, And the background image is an integrated image including element images of a background other than a main object. Generating a mask image and a background image through a lens array or a lenslet array; Generating a composite image by changing a pixel of the background image corresponding to a white pixel of the mask image to white; The mask image is a composite image generation method characterized in that the integrated image consisting of the element image of the main object represented as a binary image (binary image). delete delete The method of claim 11, wherein The background image is a composite image generation method, characterized in that the integrated image consisting of the element image for the background other than the main object.

Images