JP2013090170A - Stereoscopic image reproduction device - Google Patents

Abstract

【Task】
Provided is a stereoscopic video reproduction device that enables natural stereoscopic vision without impairing the stereoscopic effect of each content when reproducing a plurality of stereoscopic video content in a superimposed manner.
[Solution]
Interference determining means for determining that the first content hinders viewing of part or all of the second content by determining that the first content forms an image before the second content; A decoration unit that applies a frame-shaped decoration around the second stereoscopic video content, a synthesis unit that superimposes the stereoscopic video content, and when the interference determination unit determines that there is interference, the decoration unit The decoration is given around the content, and the synthesizing unit synthesizes the second content and the decoration given to the second content with priority over the first content.
[Selection] Figure 2

Description

  The present invention relates to a display device that reproduces or displays a stereoscopic image or a stereoscopic video, and particularly relates to a technique for displaying or reproducing a plurality of stereoscopic images and stereoscopic video in an overlapping manner.

  In recent years, a stereoscopic display function using binocular parallax is being installed in stationary devices such as BD recorders and BD players, and portable devices such as small game machines, imaging devices, and mobile phones. There are various stereoscopic display methods, but the basic principle of stereoscopic display using binocular parallax is to show videos (including moving images and still images) with different degrees of parallax in the left eye and right eye By introducing a mechanism and using the parallax between the eyes, a stereoscopic image is perceived.

In addition to devices capable of viewing stereoscopic images, devices such as stereoscopic cameras, conversion lenses, and stereoscopic camcorders that have the ability to capture stereoscopic photographs and stereoscopic videos have begun to spread. An environment for viewing stereoscopic content shot with the camera is becoming widespread.
On the other hand, in a 2D photograph which is an example of a conventional still image, when many 2D photographs are displayed as a list, they may be displayed in a state where they overlap each other. Here, “superimposed display” refers to a state in which a subject of one 2D photo covers a part of a subject of another 2D photo.

Thus, for example, when two 2D photographs are displayed in a superimposed manner, the image of the 2D photograph that is to be positioned in the foreground from the viewing direction is displayed and positioned in the superimposed portion. The 2D photograph to be displayed is displayed so as to be obscured by the 2D photograph to be positioned in front.
In the above example, the appearance of two 2D photographs is described. However, even if there are a plurality of 2D photographs, if the above description is applied to two target 2D photographs, a plurality of 2D photographs are used. When the images are displayed in a superimposed manner, the image of the overlapped portion is displayed as the image of the 2D photograph that is to be positioned closest to that portion, and the 2D photograph that is to be positioned deeply is positioned in front. The display is made so as to be covered by the 2D photograph to be covered.

Even in the real world, when a part of multiple photos overlaps, you can see the photograph that is located in front of the viewer from the direction of viewing the overlapped part. Therefore, the user can view a plurality of 2D photographs displayed without a sense of incongruity.
Similarly, consider a case where a plurality of stereoscopic photographs, which are examples of still images, are displayed in a superimposed manner. In a stereoscopic photograph, each photograph is displayed with a depth from each other. In this case, when the images are displayed so as to be overwritten so as to be overwritten in a predetermined order as in the case of a plurality of 2D photographs without considering the depth of the image of the stereoscopic photograph, the part where the overlapping occurs is different from the real world. There are cases where the display is made in the way that it looks.

For example, if the image that should be visible in the back is overwritten on the image that should be visible in front of the main body, the overlapping part of multiple stereoscopic photographs will be displayed in a way that looks different from the real world. It becomes difficult to see.
Depth interferes with the display in a different view from the real world due to the superimposition of the stereoscopic image displayed in the back so as to overwrite the image of the photo displayed in the foreground ( Or simply interfere).

  In order to eliminate interference generated when superimposing such stereoscopic photographs, there is a technique of superimposing after manipulating the depth. For example, in Patent Document 1, when a stereoscopic image and a stereoscopic video are superimposed, the depth ranges of the stereoscopic image and the stereoscopic video are compressed so that interference does not occur. Here, the depth range means the difference in depth between the pixel displayed at the front and the pixel displayed at the back, and compression of the depth range refers to the depth of each pixel so that the depth range becomes smaller. It means shifting to the front or back. FIG. 18 shows an example of compression of the depth range according to Patent Document 1. The depth range 44 is a range of depth that can be displayed by a display that outputs a stereoscopic video. Superimposition is performed after adjusting the depth ranges of the first stereoscopic video and the second stereoscopic video to be within the depth subrange 41 and the depth subrange 43, respectively.

Special table 2010-505174

  However, the conventional configuration has a problem that as the number of stereoscopic contents to be overlapped is increased, the depth range of each stereoscopic content is reduced, and as a result, a sufficient stereoscopic effect cannot be obtained. For example, when 10 stereoscopic photographs having the same depth range are displayed in an overlapping manner, the depth range of the stereoscopic photographs per sheet after the overlapping is reduced to 1/10 compared with before the overlapping. For this reason, in the state after overlapping, the stereoscopic effect of each stereoscopic photograph becomes poor, and a sufficient stereoscopic effect cannot be obtained.

  The present invention solves the above-described conventional problems, and an object of the present invention is to provide a stereoscopic video reproduction apparatus that can superimpose a large number of photographs without reducing the stereoscopic degree of each photograph.

In order to solve the above problems, the present invention
A reading unit for reading the first stereoscopic video content and the second stereoscopic video content;
Display for determining the display position of the first stereoscopic video content and the second stereoscopic video content, and the order in which the first stereoscopic video content and the second stereoscopic video content are overwritten so as to be overwritten Coordinate determination unit,
When the first stereoscopic video content and the second stereoscopic video content are overlapped according to the overlapping order, a part of the display area of the first stereoscopic video content and the second stereoscopic video content An interference determination unit that determines whether interference occurs when a part of the display area of the stereoscopic video content overlaps;
In the interference determination unit, when it is determined that interference has occurred, an interference cancellation processing unit that generates a decorative image from the end of the display image of the stereoscopic video content to be overwritten to the outside,
An image synthesis unit that synthesizes the first stereoscopic video content, the second stereoscopic video content, and the decoration image;
It is characterized by providing.

  According to the present invention, the reading unit that reads the first stereoscopic video content and the second stereoscopic video content, the display positions of the first stereoscopic video content and the second stereoscopic video content, and the first A display coordinate determining unit for determining an order of superimposing the one stereoscopic video content and the second stereoscopic video content so as to overwrite the first stereoscopic video content, the first stereoscopic video content, and the second stereoscopic video content; When overlapping according to the overlapping order, interference occurs when a part of the display area of the first stereoscopic video content and a part of the display area of the second stereoscopic video content overlap. A table of stereoscopic video content to be overwritten when the interference determination unit determines whether interference has occurred in the interference determination unit. An interference cancellation processing unit that generates a decorative image from the end of the image to the outside, and an image combining unit that combines the first stereoscopic video content, the second stereoscopic video content, and the decorative image. When displaying and displaying a plurality of stereoscopic video contents in a superimposed manner, the stereoscopic effect of each stereoscopic content is not impaired, and the discomfort in the vicinity of the boundary of the overlapping portion is reduced by displaying a decorative image. Is possible.

  In the first stereoscopic image content and the second stereoscopic video content, the interference determination unit has a value corresponding to the depth of each pixel of the color image from a specific viewpoint in the first stereoscopic video content. A second depth image having a value corresponding to the depth of each pixel of a color image from a specific viewpoint as a pixel value is superimposed on the display position determined in the display coordinate determination unit, respectively, in the display coordinate determination unit. When the first depth image and the second depth image are overlapped in order to be overwritten, the position is located on the side that is the end of the overwritten depth image adjacent to the overwritten depth image. An interference determination function determined using a shooting point of stereoscopic content corresponding to a pixel to be overwritten and a depth image to be overwritten, and the depth image to be overwritten Since than side determining whether interference with the values of pixels of the overwritten by depth image positioned outside occurs, it can be determined whether the interference is occurring.

  In the case where the overlap determination order determined by the display coordinate determination unit is an order in which the interference determination unit overlaps the first stereoscopic video content so as to overwrite the second stereoscopic video content. Comparing the value of the pixel of the first depth image located outside the side of the second depth image with the value when the interference determination function is applied to the pixel, If the pixel value of the pixel of the image is larger, it is determined that interference has occurred, so it can be determined whether interference has occurred.

In addition, since the interference cancellation processing unit includes a decoration width calculation unit that determines the horizontal width of the decoration image generated from the horizontal end of the second stereoscopic content to the outside, the decoration with an appropriate width is provided. An image can be generated.
In addition, the decoration width calculation means includes a plane including the side of the second stereoscopic video content on the side adjacent to the first stereoscopic video content and the coordinates of the shooting point of the first stereoscopic video content. In addition, when the pixel width of the first stereoscopic video content imaged on the shooting point side is set as a decoration width, an overlap occurs when a plurality of stereoscopic video content are reproduced and displayed. It is possible to appropriately determine the width of the decorative image that reduces the uncomfortable feeling in the vicinity of the boundary between the portions.

Further, the decoration means is characterized in that the decoration is applied with a pattern image configured such that the interval becomes narrower toward the back perceived side, thereby making it easier to obtain a three-dimensional effect of the decoration image, and a more natural three-dimensional image. Visualization is possible.
Further, the decoration means is characterized in that the decoration is gradually brightened as it goes from the upper part to the lower part of the decoration, thereby facilitating obtaining a stereoscopic effect of the decoration image and enabling a more natural stereoscopic view. Become.

In addition, the decoration means can perform optimum decoration according to the degree of interference by switching the decoration method according to the decoration width determined by the decoration width calculation means.
Further, according to the present invention, when at least one of the first stereoscopic video content and the second stereoscopic video content is a video video, the interference determination unit is configured to perform the predetermined interval of the video video. The decoration width calculation means performs interference determination and calculates the decoration width at predetermined intervals of the video image, thereby superimposing the moving image and the still image without overlapping the depth, and overlapping the moving images. Is possible.

System diagram in Embodiment 1 of the present invention The figure which shows the internal structure of the reproducing | regenerating apparatus in Embodiment 1 of this invention. The figure which shows an example of the input image in Embodiment 1 of this invention Flowchart of image overlay processing in Embodiment 1 of the present invention The figure which shows the depth image generation method in Embodiment 1 of this invention. The figure which shows an example of the color image in Embodiment 1 of this invention, and a depth image. The figure which shows an example of the superimposition of the two images in Embodiment 1 of this invention The figure which shows the flowchart of the interference cancellation determination in Embodiment 1 of this invention, and interference cancellation processing The figure which shows an example of the interference determination function in Embodiment 1 of this invention. The figure which shows an example of the interference cancellation process of the color image in Embodiment 1 of this invention The figure which shows an example of the interference cancellation process of the depth image in Embodiment 1 of this invention The figure which shows an example of the color image after the interference cancellation in Embodiment 1 of this invention, and a depth image The figure which shows an example of the interference which generate | occur | produces at the time of the superimposition of the 3 or more photograph in Embodiment 1 of this invention. The figure which shows an example of the decoration part in Embodiment 1 of this invention System diagram in Embodiment 2 of the present invention The figure which shows the internal structure of the reproducing | regenerating apparatus in Embodiment 2 of this invention. The figure which shows the internal structure of the reproducing | regenerating apparatus in Embodiment 3 of this invention. The figure which shows the interference elimination method at the time of 3D image superposition in the conventional invention

Embodiments of the present invention will be described below with reference to the drawings.
(Embodiment 1)
First, a mode of use will be described among implementations of the stereoscopic video playback apparatus according to the present invention. FIG. 1 is a diagram showing an example of a form of usage of a playback apparatus according to the present invention. In FIG. 1, a playback apparatus 101 is an example of a playback apparatus according to the present invention.

For example, the playback apparatus 101 supplies stereoscopic video content to a home theater system formed by a remote control 102 and a stereo television 103 capable of simultaneously displaying two viewpoints, which is an example of a stereoscopic display apparatus. To be served. In addition, a stereoscopic video recorded on the removable medium 104 can be viewed.
Next, the configuration of the playback apparatus 101 according to the present invention will be described. FIG. 2 is a block diagram showing an example of the internal configuration of the playback apparatus 101. The playback apparatus 101 typically includes a stereoscopic video reading unit 201, a depth image generation unit 202, a display coordinate determination unit 203, an interference determination unit 204, an interference cancellation processing unit 205, an image synthesis unit 206, a viewpoint image generation unit 207, A video external output unit 208 is provided.

  The stereoscopic video reading unit 201 is configured to read stereoscopic video content recorded on a removable medium such as an SD card inserted in a slot (not shown) included in the playback device 101. Here, the stereoscopic video content includes, for example, a still image or a stereoscopic video content of a moving image. In this embodiment, a still image (for example, a photograph) will be described as an example of the stereoscopic video content.

  In this embodiment, a semiconductor memory such as an SD card is used as an example of a removable medium. However, the present invention is not necessarily limited to this. For example, a portable hard disk or an optical disk may be a removable disk. I do not care. In this case, in addition to the slot (not shown), a drive device (not shown) for reading information recorded on the optical disc, a wired / wireless connection means (not shown) for connecting the hard disk and the playback device of the present embodiment. )Is required.

The stereoscopic video content is data storing a plurality of viewpoint images necessary for stereoscopic viewing. In the following description, the stereoscopic video content is an image (left-eye image) and right-eye image of the same subject taken from the left-eye viewpoint. It is a pair of images (right eye images) taken from the viewpoint.
Here, in order to simplify the description, the stereoscopic video reading unit 201 will be described by taking, as an example, a case where two sets of left and right image pairs of a photograph 1 and a photograph 2 shown in FIG. 3 are read. The data format of the stereoscopic video content may be a pair of image files in an arbitrary format such as JPEG or PNG, or a file format such as MPO (Multi-Picture Format). I do not care. MPO is a file format for storing a plurality of images defined by the camera video equipment industry association standard in a single file.

With reference to FIG. 2, the depth image generation unit 202 is a processing unit that receives the left-right image pair read by the stereoscopic video reading unit 201 as an input and generates a color image and a depth image.
Here, the color image corresponds to an image obtained by photographing the subject from a certain viewpoint, and represents either the left eye image or the right eye image of the left and right image pair read by the stereoscopic video reading unit 201. The depth image is an image in which a pixel value is assigned according to the distance of the subject from the shooting position.In this embodiment, when the pixel value is 255, the subject in the image exists at a position closest to the shooting position. When the pixel value is 0, it is assumed that the subject in the image exists at a position farthest from the shooting position.

In this embodiment, the pixel value assigned to the pixel of the depth image is an example, and it is not always necessary to set the pixel value farthest from the shooting position to 0 and the pixel value closest to the shooting position to 255. The distance from the shooting position to the subject may be indicated by a relative numerical value.
Hereinafter, the pixel value recorded in the depth image is referred to as a depth value. Detailed operation of the depth image generation unit 202 will be described later.

  The display coordinate determination unit 203 determines the x-coordinate and y-coordinate of each of the photos 1 and 2 when the photos 1 and 2 are displayed in a stereoscopic view, and the Z order of each photo. In this embodiment, the x coordinate is a horizontal (or left / right) coordinate as viewed from the viewer, the y coordinate is a vertical (vertical) direction coordinate as viewed from the viewer, and is the number of pixels from the origin (lower left position on the screen). Represent. The Z order is a numerical value indicating which picture is viewed from the viewer's side. In the present embodiment, an example in which a smaller numerical value is assigned to the Z order of the photograph shown on the near side as viewed from the viewer will be described, so an image with a small Z order is displayed in the foreground. For example, if a larger numerical value is assigned to the Z order of the photograph shown to the near side as viewed from the viewer, an image with a large Z order is displayed in the foreground.

  The interference determination unit 204 includes the color images and depth images of the photos 1 and 2 generated by the depth image generation unit 202, and the x and y coordinates of the photos 1 and 2 determined by the display coordinate determination unit 203. Receives Z-order as input, determines whether there is depth interference (determines whether the depth of the image to be displayed on the back side is closer to the image to be displayed on the front side), and eliminates interference It is a processing part which determines the width | variety of a decoration part required for this. Detailed procedures for determination processing and decoration width determination will be described later.

The interference cancellation processing unit 205 is a processing unit that cancels interference by processing the color images and depth images of the photos 1 and 2 based on the decoration width determined by the interference determination unit 204. A detailed procedure of processing will be described later.
The image composition unit 206 performs processing for superimposing the color images and depth images of the photos 1 and 2 according to the x coordinate, the y coordinate, and the Z order determined by the display coordinate determination unit 203. When the display areas of Photo 1 and Photo 2 overlap, the pixel of the image with the smaller Z order is adopted as the post-combination pixel.

The viewpoint image generation unit 207 is a processing unit that generates a right-eye image from the color image (left-eye image) and the depth image that have been generated by the image composition unit 206 and a left-right image pair.
The video external output unit 208 is a processing unit that outputs the left and right image pairs generated by the viewpoint image generation unit 207 to an external device. Communication with an external device is preferably performed by a video / audio input / output interface such as HDMI (High-Definition Multimedia Interface). In addition, as a data format of stereo image data transmitted using HDMI, a format such as a side-by-side format, a frame sequential format, or a checker pattern format is used. The video external output unit 207 also performs processing for transforming the left and right image pair data generated by the viewpoint image generation unit 207 in accordance with the output data format.

Next, the playback apparatus 101 according to the present invention will be described. FIG. 4 shows a flowchart of processing for displaying a stereoscopic photograph in a superimposed manner.
First, the stereoscopic video reading unit 201 reads a stereoscopic video (Photo 1, Photo 2) recorded on a removable medium such as an SD card inserted in a slot (not shown) included in the playback device 101 (Step S401). ).

Next, the depth image generation unit 202 generates depth images of the photos 1 and 2 read by the stereoscopic video reading unit 201 (step S402).
FIG. 5 is a diagram illustrating an example of processing for generating a depth image from a left-eye image and a right-eye image. In the present embodiment, a configuration in which a parallax image is generated with reference to the left eye image will be described as an example, but a configuration in which a parallax image is generated with reference to the right eye image may be used.

  When generating a parallax image based on the left-eye image, a search is performed in the horizontal direction where the pixel of the right-eye image corresponding to each pixel of the left-eye image exists for the right eye. As a method for detecting the right eye pixel corresponding to the left eye pixel, for example, there is a method of searching for a point where the absolute value difference between the pixel values in the block is minimized in units of n * n blocks as in the present embodiment. When pixels corresponding to each pixel for the left eye are detected in the right eye image, how many pixels of the corresponding corresponding right eye image are detected with reference to the pixel for the left eye image and the detected right eye Whether the pixel of the image for use is on the right side or the left side of the pixel for the image for the left eye as a reference is detected. If the pixel on the right-hand side or the left-hand side of the reference left-eye image is expressed using polarity (that is, plus “+” or minus “−”), the distance is represented by an integer value. A function for associating a corresponding integer value with a value of 0 to 255 in a one-to-one correspondence is prepared in advance, and the distance (including direction) between the left and right pixels is adjusted to any one of 0 to 255 using this function. This is converted into a numerical value, and the converted value is mapped as the pixel value of the pixel of the depth image, and a depth image can be generated by performing this process for each pixel of the image for the left eye.

FIG. 6 shows the depth image of photo 1 generated by the depth image generation unit 202 when the left-eye image of photo 1 and the left-eye image of photo 2 are the color image of photo 1 and the color image of photo 2, respectively. An example of the depth image of the photograph 2 is shown.
The upper left part of FIG. 6 is a diagram showing the positional relationship between the subject (subject 1) and the photographing point (photographing point 1) at the time of taking Photo 1 (the image for the left eye of Photo 1). It is a figure which shows the positional relationship of the to-be-photographed object and the imaging | photography point 1 at the time. The center of the left side is a color image obtained when the subject 1 is taken from the shooting point 1 (the left eye image of Photo 1), and the lower left part is generated by the depth image generation unit 202 using the left eye image and the right eye image of Photo 1. It is a depth image of Photo 1.

  Similarly, the upper right side of FIG. 6 is a diagram showing the positional relationship between the subject (subject 2) and the photographing point (photographing point 2) at the time of photographing Photo 2 (the left-eye image of Photo 2). FIG. 6 is a diagram illustrating a positional relationship between a subject 2 and a shooting point 2 when viewed from above. The center of the right side is generated by the depth image generation unit 202 using the color image when the subject 2 is captured from the shooting point 2 (the left eye image of Photo 2), and the lower right side is generated using the left eye image and the right eye image of Photo 2. It is a depth image of photograph 2.

  As can be understood as shown in the upper diagram of FIG. 6, the direction in which the subject 1 is viewed from the imaging point 1 and the direction in which the subject 2 is viewed from the imaging point 2 are defined as the depth direction, When the values in the depth direction of the shooting point 2 are matched, the object in the back of the photograph 2 is photographed in comparison with the photograph 1, so that the subject in the depth image of the photograph 1 is also obtained in the depth image obtained. Compared to the pixel value, the pixel value of the subject in the depth image corresponding to Photo 2 is a value indicating that the pixel value is farther from the shooting point.

In the present embodiment, as shown in FIG. 5, the depth image is assigned a pixel value closer to 0 (a value closer to black) as the distance from the shooting point increases. Therefore, in the example illustrated in FIG. The subject in the depth image is closer to black.
Referring to FIG. 4, next, display coordinate determination unit 203 determines the x-coordinate, y-coordinate, and Z-order for displaying photo 1 and photo 2 (step S403). In the present embodiment, display coordinate determination section 203 superimposes input photo 1, photo 2 on the display screen, photo 1 and photo 2 according to the operation of remote controller 102 by the user of playback apparatus 101. From the information regarding the order, the x-coordinate and y-coordinate on the display screen for displaying Photo 1 and Photo 2, and the Z order of Photo 1 and Photo 2 are determined.

Here, the x-coordinate and y-coordinate determined by the display coordinate determining unit 203 shown in FIG. 4 are the coordinates of the left-eye image displayed on the display device (not shown), for example.
FIG. 7 shows an example of the overlapping position of the photograph 1 and the photograph 2 in the present embodiment. 7 is a coordinate system for specifying the display position on the display screen of the stereo television 103. As shown in the figure, the upper left vertex of Photo 1 is (0, y ₃ ), the upper left vertex of Photo 2 is (x ₁ , y ₂ ), and the Photo 2 side is displayed in front (Photo 2 It is assumed that the remote controller 102 selects and inputs the Z order so that the Z order is smaller than the Z order of Photo 1).

A rectangular area whose diagonals are (x ₁ , y ₁ ) and (x ₂ , y ₂ ) is an area in which the color image of Photo 1 and the color image of Photo 2 are superimposed and reproduced. Although the subject of photo 1 is closer to the shooting position than the subject of photo 2, the photo 2 is displayed with priority over the photo 1, and thus there is a possibility of interference. FIG. 7 shows only the color image, but the same applies to the depth image.

The process of step S403 may be performed before steps S401 and S402.
Next, the interference determination unit 204 determines that the photo 1 and the photo 2 from the color image of the photo 1 determined by the display coordinate determination unit 203, the x-coordinate and the y-coordinate of the color image of the photo 2, and the Z order of the photos 1 and 2. It is determined whether interference occurs and the width of the decoration portion is determined (step S404).

FIG. 8 is a flowchart showing an example of the algorithm of the method for determining the width of the decoration portion in S404.
The flowchart in FIG. 8 is based on the premise that the left end of the photograph 2 is located on the right side of the center of the photograph 1. Accordingly, when the display coordinate determination unit 203 determines that the left end of the photo 2 is located on the left side of the center of the photo 1, a warning for prompting the movement of the photo 2 is displayed on the display screen or a warning sound is generated. Is desirable.

First, initialize variables WIDTH1, _{y i,} respectively 0, _{y 1} (step S801).
Then, it initializes the variable _{x i} with _x 2/2 (step S802).
x ₂ corresponds to the x coordinate of the right end of the photograph 1. x _2/2 is the center of the x-coordinate of the photo 1, it corresponds to the x-axis direction of the position of the imaging point of the photograph 1.

Next, an interference determination function f (x) at y = y ₁ is obtained (step S803). The interference judgment function is a photo that is placed in the back because the depth of the subject of the photo that appears in the back is apparently deeper than that of the subject in the front of the photo when multiple stereoscopic photos are stacked This is a function representing a boundary line where interference occurs where a photograph placed in front is hidden.
A method of obtaining the interference determination function at y = y ₁ will be described with reference to FIG. FIG. 9 shows the relative positional relationship in the case where the depth image of photo 1 and the depth image of photo 2 are arranged at the positions where the color image of photo 1 and the color image of photo 2 determined by the display coordinate determination unit 203 are arranged. FIG.

In the present embodiment, an example in which a photograph 2 is superimposed on a photograph 1 will be described. In this example, the part where the photograph 1 and the photograph 2 overlap each other means that the photograph 2 is preferentially displayed.
In FIG. 9, the x-axis is the horizontal position of the display screen, and the y-axis is the vertical position of the display screen. In FIG. 9, the z-axis indicates the position in the direction perpendicular to the display screen, and numerical values indicate the depth such as the degree of protrusion from the display screen and the degree of retraction from the display screen, as described in FIGS. Are indicated by numerical values from 0 to 255. More specifically, it is the value of the pixel at the position specified by the x-axis and y-axis.

In the upper diagram of FIG. 9, (0, y ₁ ) is the coordinates of the lower left vertex of Photo 1, (0, y ₃ ) is the coordinates of the upper right vertex of Photo 1, and (x ₂ , y ₁ ) is the right of Photo 1 The coordinates of the lower vertex, (x ₂ , y ₃ ) are the coordinates of the upper right vertex of photo 1, (x ₁ , 0) are the coordinates of the lower left vertex of photo 2, and (x ₁ , y ₂ ) are the upper left vertex of photo 2 The coordinates, (x ₃ , 0) are the coordinates of the lower right vertex of Photo 2, and (x ₃ , y ₂ ) are the coordinates of the upper right vertex of Photo 2.

The lower side of FIG. 9 shows pixel values up to 0 ≦ x ≦ x ₃ when y = y ₁ is fixed. That is, it shows the xz plane when y = y ₁ when the depth image of photo 1 and the depth image of photo 2 are overlapped, and the pixel values of each depth image are plotted with solid lines. . Shooting point Photo point _{A 1 (x 2/2,} (y 3 -y1) / 2,255) were projected on the x-z plane position _(x 2 / 2,255), point B pictures 2 y = straight line connecting the AB when the leftmost pixel _(x 1, 0) in the _{y 1} is the interference determination function in y = _{y 1.}

  In general, there are many cases in which a distant background sufficiently far from the subject is shown at both ends in the horizontal direction of a photograph. In such a case, the depth image of the photo (photo 2 in the present embodiment) located in the Z-order on the side adjacent to the photo (photo 1 in the present embodiment) arranged in the back in the Z-order. The side in the vertical direction (side in the y-axis direction) is a straight line parallel to the y-axis direction in the xyz space.

In such a case, the interference determination function f (x) is, for example, a shooting point of a photograph (photo 1 in the present embodiment) arranged in the back in the Z order and a photograph (front) in the Z order described above ( In the present embodiment, the plane including the vertical side (the side in the y-axis direction) of the depth image in photo 2) is the xz plane on y = y _i (i is a value from y ₁ to y ₂ ). It will be shown as a straight line when projected.

Referring to FIG. 8, in step S804, temporary variables z ₁ and z ₂ are calculated. z ₁ is a pixel value in (x _i , y _i ) of the depth image of photograph 1, and z ₂ is a value at x = x _i of the interference determination function f (x). Next, compare the values of _{z 1} and _{z 2} (step _{S805), z1> (x i} , y i) of the depth image of _{z 2} That Picture 1 pixel value at is greater than the value of collision determination function, ie , if such be located more forward (more photographing point side) stores the difference between the left edge coordinates x ₁ and x _i pictures 2 to the variable w (step S806).

Next, the values of the variable w and the variable width1 are compared (step S807). If w> width1, the value of the width1 is updated with w (step S808). If it is not determined that z ₁ <z ₂ in step S805, the execution of steps S806 to S808 is skipped, and step S809 is executed. In step S809, the value of x _i is incremented by 1 (step S809). While x _i <x ₂ , the execution of steps S804 to S809 is repeated (step S810). In step S811, the value of y _i is incremented by 1 (step S811). While y _i <y ₂ , the execution of steps S802 to S811 is repeated (step S812).

Referring to FIG. 4, if it is determined in step S404 that there is interference, that is, if it is determined that the size (width1) of the decoration portion to be added to photo 2 is larger than 1 pixel, the photo 2 is decorated. (Step S405).
FIG. 10 shows a color image decoration example and FIG. 11 depth image decoration example. In the color image, the decoration is filled with a single color in an area having diagonal lines (x ₁ -width1, y ₂ ) and (x ₁ , y ₁ ). In the depth image, a region where (x ₁ −width1, y ₂ ) and (x ₁ , y ₁ ) are diagonal, and a column of x = x ₁ −width 1 is a column of the depth image of photograph 1 x = x ₁ − minimum value of the pixel in WIDTH1, columns of x = x1 is a maximum value of the pixel in the column x = x ₂ depth image of the photographic 2, the pixels between the two fill as depth changes continuously It is.

Note that the decoration area shown in the figure is the minimum area calculated by applying the algorithm of FIG. 4 when interference occurs, and there is no problem even if a decoration larger than this is applied. For example, a decoration having the same width as that in FIGS. 10 and 11 may be provided on the upper side, the right side, and the lower side of the photograph 2.
Referring to FIG. 4, if it is determined in step S404 that there is no interference, step S405 is not executed, but step S406 is executed. In step S406, the image composition unit 206 performs a process of superimposing the color images and depth images of the photos 1 and 2 according to the x and y coordinates determined by the display coordinate determination unit 203. FIG. 12 shows an example of a color image and a depth image after the decoration is added and superimposed. The coordinates of the points 1201 and 1202 correspond to (x ₁ , y ₂ ) in FIG.

  Next, the viewpoint image generation unit 207 generates a stereo image pair obtained by combining the photos 1 and 2 (step S407). The generation of a stereo image from a color image and a depth image is a technique generally called DIBR (Depth Image based Rendering). In DIBR, a pixel of a new viewpoint is generated by shifting a pixel of a color image of an original image in a horizontal (or horizontal) direction according to a depth image.

For example, it is assumed that the value corresponding to the shift amount of the pixel (x _i , y _i ) of the color image is the pixel value of the pixel (x _i , y _i ) of the depth image. In this case, first, the pixel value (any one of 0 to 255) of the pixel (x _i , y _i ) of the depth image corresponding to the pixel (x _i , y _i ) of the color image is read, and the corresponding pixels Convert to distance. As described in the description of FIG. 5, a function that has a one-to-one correspondence between the pixel value of the depth image (any value from 0 to 255) and the distance between pixels (including direction (that is, polarity)) is used. do it. For example, in the copy destination of the pixel value of (x _i , y _i ) of the color image in the original image, the distance calculated from the pixel value at the coordinates (x _i , y _i ) of the depth image of the original image is 3 In this case, it is copied to the pixel at the coordinates (x _{i + 3} , y _i ) of the viewpoint image. By performing such a shift process for each pixel of the color image, a new viewpoint image is generated.

This color image is used as a new left-eye image, and the generated viewpoint image is used as a new right-eye image.
If there is interference between Photo 1 and Photo 2, the color image and depth image are decorated as described in FIGS. 10 and 11, so that a new left eye image, In the right-eye image, the part where the interference between Photo 1 and Photo 2 occurs is an image with decoration.

  Finally, the video external output unit 208 uses the stereo image pair generated by the viewpoint video generation unit 207 (that is, an image pair in which the color image is a new left eye image and the generated viewpoint image is a new right eye image). Output to an external device (step S408). Prior to output, the video external output unit 208 may convert the stereo image pair generated by the viewpoint video generation unit 207 into a format such as a side-by-side format, a frame sequential format, or a checker pattern format.

  By configuring as described above, when a plurality of stereoscopic contents are displayed in an overlapped manner, a portion in which a plurality of stereoscopically displayed contents are overlapped while maintaining a stereoscopic effect of each stereoscopic video content. When content that is displayed in the back is prioritized and content that is displayed in the foreground is displayed, a decorative part that mitigates the sense of discomfort is created and displayed at the boundary of the content, reducing the sense of discomfort caused by interference It becomes possible.

  In this embodiment, the stereoscopic video reading unit 201 is configured to read stereoscopic video content recorded on a removable medium such as an SD card. However, an HDD (Hard Disc Drive), SSD (Solid) included in the playback device 101 is used. A stereoscopic image stored in a storage device (not shown) such as State Drive may be read.

In addition, the stereoscopic video reading unit 201 reads stereoscopic video content recorded in an electronic device such as a flash memory drive, a mobile phone, or a digital camera connected through a USB interface (not shown) provided in the playback device 101. It may be configured.
In addition, the stereoscopic video reading unit 201 reads the stereoscopic video content recorded on an optical disk such as a DVD or a BD-ROM inserted in a DVD drive or a BD drive (not shown) included in the stereoscopic video reproduction device 101. It may be configured.

  The stereoscopic video reading unit 201 may be configured to download stereoscopic video content from an external server connected to the network through a wired LAN or a wireless LAN network interface (not shown) included in the playback device 101. In this case, the download of the stereoscopic video content can be realized using a known protocol such as HTTP (Hypertext Transfer Protocol) or FTP (File Transfer Protocol).

  Further, the stereoscopic video reading unit 201 may be configured to download stereoscopic video content from an external server connected to the network through a Bluetooth interface (not shown) provided in the playback apparatus 101. In this case, downloading of stereoscopic video content can be realized using a known profile such as FTP (File Transfer Profile), BIP (Basic Imaging Profile).

In the present embodiment, the stereoscopic video reading unit 201 reads a pair of stereo images. However, the form of the stereoscopic video content is not limited to a stereo image pair, for example, a pair of a color image and a depth image. There may be. In this case, the depth image generation unit in FIG. 2 is not necessary.
In the flowchart shown in FIG. 4, in particular, step S404 and step S405 may be realized as an LSI in addition to the realization by software. By doing so, it is possible to speed up processing necessary for image generation after interference cancellation.

In the present embodiment, the example in which the photograph 1 side is disposed on the left back side and the photograph 2 is disposed on the right front side has been described. However, even if the photograph 2 is on the left front side and the photograph 1 is on the right back side, Can be implemented similarly.
Further, the present invention can be similarly implemented even when three or more photographs are superimposed.
FIG. 13 shows examples (a) and (b) in which three photographs are displayed in an overlapping manner. In each example, interference occurs in two areas of interference 1 and interference 2. When a plurality of interferences occur and the sizes of the decorative parts change, the maximum value among the decorative part sizes calculated for each interference part may be used as the size of all the decorative parts. Since the difference in the size of the decorative part is perceived as a difference in the apparent depth, by unifying the decorative size in the different interference parts in this way, the apparent depth of the decorative part becomes uniform, reducing the sense of incongruity It becomes possible to do.

  In the present embodiment, the color image decoration method shown in FIG. 10 has been described as being painted with a single color, but it may be configured to be painted with a specific color and pattern using a psychological factor of stereoscopic vision. The psychological factor of stereoscopic vision is based on empirical knowledge obtained in human life, and it is said that a stereoscopic effect is obtained by analyzing images under various implicit assumptions. Painting the decorative part in conformity with these assumptions has the effect of enhancing the stereoscopic effect of the decorative part and making it easier for the viewer to feel the stereoscopic effect.

For example, it is said that the depth is sensed using the implicit assumption that there is one light source above the field of view. By utilizing this, for example, as shown in FIG. 14A, the decoration part attached to the upper side is darkened, the decoration part attached to the lower side is brightened, and the decoration part is shaded to enhance the stereoscopic effect of the decoration part.
Also, for example, it is said that when there is a pattern on the surface of an object, the depth is felt using an implicit assumption that the texture of the pattern is uniform. By utilizing this, for example, as shown in FIG. 14B, the three-dimensional effect of the decoration portion can be enhanced by making the texture of the pattern coarser toward the front and the texture of the pattern becoming finer toward the back.

Moreover, you may comprise so that how to paint may be changed with the magnitude | size of a decoration part. By doing in this way, it becomes possible to give an optimal decoration according to the magnitude | size of a decoration part.
In the present embodiment, it has been described that Photo 1 and Photo 2 are overlapped in the horizontal and vertical directions, but it is not always necessary to overlap. For example, even if Photo 1 and Photo 2 are arranged side by side in the horizontal direction, if Photo 1 is displayed so as to form a larger image in the depth direction than Photo 2, the subject of Photo 1 is Interference may occur due to the appearance of popping out.

Further, in the flowchart shown in FIG. 8, it is assumed that the left end of photo 2 is located on the right side of the center of photo 1, but the left end of photo 2 is located on the right side of the center of photo 1. For example, processing is performed using the origin (0, 0) as the point A, the initial value at the time of initialization is set to 0 in S802 of the flowchart described in FIG. 8, and 0 ≦ between _{x i <(x 2/2} ), if to repeat the execution of steps S804~ step S809, decoration process can have you if you are located left end of the photograph 2 on the right side than the center of the photograph 1 Can be done.

(Embodiment 2)
In the first embodiment, an example in which video is output to a stereoscopic display compatible with the outside of the playback device has been described. However, in the second embodiment, a case where the playback device itself includes a stereoscopic display is described. In the second embodiment, detailed description of the same parts as those in the first embodiment is omitted, and only the differences from the first embodiment are described. Parts not described are the same as those in the first embodiment.

  FIG. 15 is a diagram showing an example of a form of usage of the playback apparatus according to the present invention. In FIG. 1, the playback apparatus according to the present invention is a playback apparatus 1501. As a specific example of the playback device 1501, a smartphone, a tablet computer, a liquid crystal TV, a plasma TV, or the like having a stereoscopic function can be considered. In the following description, it is assumed that the playback device 1501 includes a stereoscopic display that can display two viewpoints simultaneously.

  FIG. 16 shows the internal configuration of the playback apparatus 1501. The playback apparatus 1501 typically includes a stereoscopic video reading unit 1601, a depth image generation unit 1602, a display coordinate determination unit 1603, an interference determination unit 1604, an interference elimination processing unit 1605, an image composition unit 1606, a viewpoint image generation unit 1607, The video output unit 1608 is configured. Since the stereoscopic video reading unit 1601 to the viewpoint image generation unit 1607 are the same as the stereoscopic video reading unit 201 to the viewpoint image generation unit 207 in FIG.

The video output unit 1608 is a display device that displays the left eye image and right eye image generated by the viewpoint image generation unit 1607 to the left eye and right eye of the user, respectively.
Various methods are conceivable as a method for realizing the viewpoint image generation unit 1607. For example, one of the commonly used methods is a method using shutter glasses (not shown).
In this method, the viewer's left and right eye fields are alternately covered at high speed with glasses, and the display image on the display is updated at high speed for the left and right eyes in synchronization with the movement of the glasses. Thus, the left eye image displayed on the display can be seen only by the shutter glasses, and the right eye image can be seen only by the right eye. As another method, there is a method using a lenticular lens. A lenticular lens is a lens in which bowl-shaped lenses are arranged in the horizontal direction. By refracting light emitted from the display in units of pixels of the display image, different pictures can be displayed on the left and right eyes.

By configuring as described above, even when the playback device itself is provided with a stereoscopic display, while reproducing a plurality of stereoscopic content, while maintaining the stereoscopic effect of each stereoscopic video content, Depth interference can be avoided.
In the present embodiment, the description has been given assuming that the playback apparatus 1501 has two viewpoints, but a multi-view display having three or more viewpoints may be used. When a multi-viewpoint display is used, two cases can be considered depending on the arrangement of viewpoints.

  One is a case where there is parallax only in the left-right direction and no parallax in the vertical direction. In this case, the viewpoint image generation unit 1607 may generate three or more viewpoints, and the video output unit 1608 may be configured so that the picture of each viewpoint can be correctly viewed from each viewing position. By doing so, the decorative portion for eliminating the interference can also see a stereoscopic image corresponding to the viewing position, so that a more natural stereoscopic effect can be obtained.

  Another case where a multi-viewpoint display is used is a case having a vertical parallax. In this case, in step S404 shown in FIG. 4, it is possible to determine whether or not vertical interference has occurred by calculating the photograph 1 after rotating the photograph 2 by 90 degrees about the z axis. Similarly, the processing in step S405 can calculate the size of the decoration part required in the vertical direction by calculating the photograph 1 after rotating the photograph 2 by 90 degrees about the z-axis. By configuring in this way, the decorative portion for eliminating the interference can view a stereoscopic image corresponding to the viewpoint position even when the viewpoint is moved in the vertical direction, so that a more natural stereoscopic effect can be obtained. Can be obtained.

(Embodiment 3)
In the first embodiment and the second embodiment, the stereoscopic video content to be displayed in a superimposed manner is the processing when the still image, that is, the photograph 1 and the photograph 2 are superimposed. An example in which at least one of the stereoscopic video contents to be performed is a moving image will be described. In the third embodiment, detailed description of the same parts as those in the first and second embodiments is omitted, and only changes from the first and second embodiments are described. Parts not described are the same as those in the first and second embodiments.

  FIG. 17 is a diagram showing the internal structure of the playback apparatus 101 in this embodiment. The playback apparatus 101 typically includes a stereoscopic video reading unit 1701, a depth image generation unit 1702, a display coordinate determination unit 1703, an interference determination unit 1704, an interference cancellation processing unit 1705, an image composition unit 1706, a viewpoint image generation unit 1707, A video external output unit 1708 and a video decoder 1709 are provided. Since the display coordinate determining unit 1703 to the video external output unit 1708 are the same as the stereoscopic video reading unit 201 to the video external output unit 208 in FIG.

  A stereoscopic video reading unit 1701 reads a stereoscopic video stream. Stereoscopic video stream is a side-by-side, top-and-bottom format in which left and right viewpoint video is recorded in each frame, or a stream for the right eye and a stream for the left eye are provided as different transport stream files, There is a format in which a stream for the right eye and a stream for the left eye are provided by being multiplexed into a single transport stream file. In this embodiment, the stereoscopic video reading unit 1701 will be described as reading a moving image stream recorded in a side-by-side format.

Details of the depth image generation unit 1702 will be described later.
The video decoder 1709 decodes the video stream read by the stereoscopic video reading unit 1701 to obtain a frame. The frame includes a left eye frame image and a right eye frame image. In the present embodiment, it is assumed that the left half of the frame is the left eye frame image and the right half of the frame is the right eye frame image. A left-eye frame image and a right-eye frame image are obtained by enlarging the left half and the right half of the frame obtained by decoding twice in the horizontal direction. The video decoder 1709 needs a number proportional to the number of moving images to be reproduced simultaneously. For example, as many video decoders 1709 as the number of moving images to be reproduced simultaneously are required, such as two when two moving images are overlapped and one when overlapping a moving image and a still image.

With reference to FIG. 4, the flowchart of the process which superimposes and displays the stereoscopic video stream in this Embodiment is demonstrated. In the following, Photo 1 and Photo 2 in FIG. 4 are read as frame images obtained by decoding 1709 the stereoscopic video stream read by the stereoscopic video reading unit 1701 by the video decoder.
In step S401, the video images 1709 and 1710 decode the two stereoscopic video streams 1 and 2 read by the stereoscopic video reading unit 1701 to obtain frame image 1 and frame image 2, respectively.

In step S402, a depth image is obtained from each of the frame image 1 and the frame image 2.
Various methods are conceivable as the depth image acquisition method.
For example, a method of calculating the depth value in each frame in real time by the same method as in the first embodiment can be considered. This is a calculation method based on video frame data obtained by decoding a left-eye video stream and a right-eye video stream. At this time, the obtained depth image may be stored in association with the media playback time, and may be used for the second and subsequent times of the same video stream.

As another method, a method of multiplexing data relating to depth in advance on the video stream side is also conceivable. This can be realized by obtaining a depth image of each frame in advance and multiplexing the change of the depth image in time with the video stream.
At this time, instead of obtaining the temporal change of the depth image for each frame of the video data, for example, the maximum value and the minimum value for each arbitrary unit such as GOP may be obtained. This configuration has the effect of reducing the size of the video stream and the effect of reducing the decoding load. In particular, when only the maximum value and the minimum value of the depth value through the reproduction of the video stream are stored, it can be recorded as a still image, so that the decoding burden is greatly reduced.

Steps S403 to S408 are the same as those in the first embodiment, and thus detailed description thereof is omitted.
By configuring as described above, it is possible to avoid interference even in the case of superimposing a moving image and a still image or overlapping each other.

The playback device constituting the present invention is:
When contents such as photographs and videos that can be viewed stereoscopically are displayed in an overlapping manner, natural stereoscopic vision can be realized without impairing the stereoscopic degree of each stereoscopic content. In particular, it can be used in the consumer equipment industry, which is involved in photo and video content playback machines.

DESCRIPTION OF SYMBOLS 101 Playback apparatus 102 Remote control 103 Output monitor 104 Removable media 201 Stereoscopic image reading part 202 Depth image generation part 203 Display coordinate determination part 204 Interference determination part 205 Interference cancellation process part 206 Image composition part 207 Viewpoint image generation part 208 Image | video external output part

Claims (9)

A reading unit for reading the first stereoscopic video content and the second stereoscopic video content;
Display for determining the display position of the first stereoscopic video content and the second stereoscopic video content, and the order in which the first stereoscopic video content and the second stereoscopic video content are overwritten so as to be overwritten Coordinate determination unit,
When the first stereoscopic video content and the second stereoscopic video content are overlapped according to the overlapping order, a part of the display area of the first stereoscopic video content and the second stereoscopic video content An interference determination unit that determines whether interference occurs when a part of the display area of the stereoscopic video content overlaps;
In the interference determination unit, when it is determined that interference has occurred, an interference cancellation processing unit that generates a decorative image from the end of the display image of the stereoscopic video content to be overwritten to the outside,
An apparatus for reproducing stereoscopic video, comprising: an image synthesis unit that synthesizes the first stereoscopic video content, the second stereoscopic video content, and the decoration image. The interference determination unit
A first depth image having a pixel value corresponding to a depth corresponding to each pixel of a color image from a specific viewpoint in the first stereoscopic video content and a color image from a specific viewpoint in the second stereoscopic video content A second depth image having a pixel value corresponding to the depth of each pixel is overwritten at the display position determined by the display coordinate determining unit in the overlapping order determined by the display coordinate determining unit. When superimposed,
In the portion where the first depth image and the second depth image overlap, the three-dimensional image corresponding to the pixel located on the side that is the edge of the overwritten depth image adjacent to the overwritten depth image and the overwritten depth image It is determined whether interference has occurred using an interference determination function determined using a shooting point of visual content and a pixel value of the overwritten depth image located outside the side of the overwritten depth image. The stereoscopic video reproduction device according to claim 1. The interference determination unit
When the overlapping order determined by the display coordinate determination unit is an order to superimpose the second stereoscopic video content over the first stereoscopic video content, the second depth image The pixel value of the first depth image located outside the side is compared with the value when the interference determination function is applied to the pixel, and the pixel value of the pixel of the first depth image is compared. The stereoscopic video reproduction device according to claim 2, wherein if the value is large, it is determined that interference has occurred.   The said interference cancellation process part is provided with the decoration width calculation means which determines the width | variety of the said horizontal direction of the decoration image produced | generated from the edge part of the horizontal direction of a said 2nd stereoscopic content outside. The stereoscopic video reproduction apparatus described. The decoration width calculation means includes:
The image is formed closer to the shooting point than the plane including the side of the second stereoscopic video content on the side adjacent to the first stereoscopic video content and the coordinates of the shooting point of the first stereoscopic video content. The stereoscopic video reproduction device according to claim 1, wherein a horizontal pixel width of the first stereoscopic video content is a decoration width.   6. The stereoscopic video reproduction device according to claim 1, wherein the decoration means decorates with a pattern image configured such that the interval becomes narrower toward the side perceived in the back.   7. The stereoscopic video reproduction apparatus according to claim 1, wherein the decoration means performs decoration so that the decoration gradually becomes brighter from the upper part to the lower part of the decoration.   8. The stereoscopic video reproduction apparatus according to claim 1, wherein the decoration means switches a decoration method according to the decoration width determined by the decoration width calculation means. When at least one of the first stereoscopic video content and the second stereoscopic video content is a moving image,
The said interference judgment means performs interference judgment for every predetermined interval of the said video image, and the said decoration width calculation means calculates the decoration width for every predetermined interval of the said video image. The stereoscopic video reproduction device according to claim 8.

Images