[go: up one dir, main page]

WO2018187724A1 - Système, procédé et logiciel destinés à convertir des images capturées par une caméra de champ lumineux en images tridimensionnelles qui semblent s'étendre verticalement au-dessus d'un support d'affichage ou devant celui-ci - Google Patents

Système, procédé et logiciel destinés à convertir des images capturées par une caméra de champ lumineux en images tridimensionnelles qui semblent s'étendre verticalement au-dessus d'un support d'affichage ou devant celui-ci Download PDF

Info

Publication number
WO2018187724A1
WO2018187724A1 PCT/US2018/026516 US2018026516W WO2018187724A1 WO 2018187724 A1 WO2018187724 A1 WO 2018187724A1 US 2018026516 W US2018026516 W US 2018026516W WO 2018187724 A1 WO2018187724 A1 WO 2018187724A1
Authority
WO
WIPO (PCT)
Prior art keywords
light field
subject
image data
appear
cause
Prior art date
Application number
PCT/US2018/026516
Other languages
English (en)
Other versions
WO2018187724A8 (fr
Inventor
Richard S. FREEMAN
Scott A. HOLLINGER
Original Assignee
Maxx Media Group, LLC
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Priority claimed from US15/481,447 external-priority patent/US10136121B2/en
Priority claimed from US15/829,861 external-priority patent/US10475233B2/en
Application filed by Maxx Media Group, LLC filed Critical Maxx Media Group, LLC
Publication of WO2018187724A1 publication Critical patent/WO2018187724A1/fr
Publication of WO2018187724A8 publication Critical patent/WO2018187724A8/fr

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/50—Depth or shape recovery
    • G06T7/55—Depth or shape recovery from multiple images
    • G06T7/557—Depth or shape recovery from multiple images from light fields, e.g. from plenoptic cameras
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/50—Depth or shape recovery
    • G06T7/55—Depth or shape recovery from multiple images
    • G06T7/593—Depth or shape recovery from multiple images from stereo images
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
    • H04N13/20—Image signal generators
    • H04N13/204—Image signal generators using stereoscopic image cameras
    • H04N13/207—Image signal generators using stereoscopic image cameras using a single 2D image sensor
    • H04N13/232—Image signal generators using stereoscopic image cameras using a single 2D image sensor using fly-eye lenses, e.g. arrangements of circular lenses
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10052—Images from lightfield camera

Definitions

  • the present invention relates to systems, methods, and software that are used to create three-dimensional images that are viewed on a display medium, such as an electronic display or printed paper. More particularly, the present invention relates to systems, methods and software that create three-dimensional images with enhanced effects that cause the three-dimensional images to appear to extend above, or in front of, the display medium being viewed.
  • the present invention is a system, method and software for producing imagery with enhanced 3D effects that is viewed on a display medium.
  • the imagery with enhanced 3D effects can be a fixed image or a dynamic video.
  • the imagery contains elements that appear, at least in part to extend vertically above, or in front of the display medium that is showing the imagery.
  • a light field camera images a scene that contains at least one subject .
  • the subject is positioned in an initial orientation.
  • the light field camera Upon imaging, the light field camera produces light field image data that digitally represents the scene being imaged.
  • the light field image data may be interpreted dimensionally different ways, including converting to left and right eye stereo images, or converting to left and right eye side-by-side images
  • the light field image data corresponding to the subject in the scene is edited to add enhanced 3D effects .
  • the light field image data becomes enhanced image data .
  • the added 3D effects cause the subject to appear, at least in part, to extend vertically above, or in front of, the display medium depending upon the orientation of the display medium.
  • FIG. 1 shows system hardware needed to create and utilize the present invention system, method, and software
  • FIG. 2 is a perspective view of the scene representing the data collected by imaging a physical scene
  • FIG. 3 is a side view showing tilt digital modifications made to the data collected by imaging
  • FIG. 4 is a side view showing bend digital modifications made to the data collected by imaging
  • FIG. 5 is a front view showing taper digital modifications made to the data collected by imaging
  • FIG. 6 is a front view showing stretch digital modifications made to the data collected by imaging
  • FIG. 7 and FIG. 8 show left and right offset images, respectively, of the virtual scene
  • FIG. 9 is a top view of the offset images showing the superimposition of guidelines
  • FIG. 10 shows a digitally corrected offset image created using the guidelines previously shown in Fig. 9;
  • FIG. 11 shows a final imagery with left and right offset images superimposed
  • FIG. 12 shows a block diagram logic flow for the software methodology utilized by the present invention .
  • imagery 10 with enhanced 3D effects that is viewed on a display medium, such as a printed page or the illustrated example of a display 12 of an electronic device 14.
  • the imagery 10 can be a stationary image or a video. Regardless, the imagery 10 appears to have elements that are three-dimensional when viewed on the
  • the elements embody enhanced 3D effects that cause those elements to appear to extend vertically above, or in front of, the surface plane of the display 12. If the electronic device 14 has a traditional LED or LCD display, the imagery 10 will have to be viewed with 3D glasses in order to observe the three- dimensional effects in the imagery 10. If the
  • the electronic device 14 has an autostereoscopic display, then the three-dimensional effects in the imagery 10 can be observed with the naked eye.
  • the imagery 10 that contains enhanced 3D effects starts as a physical scene 15 that is captured by a light field camera 17.
  • the physical scene 15 captured by the light field camera 17 typically contains a primary subject 20.
  • the primary subject 20 is a toy dinosaur 22.
  • any subject or collection of subjects can be imaged.
  • the light field camera 17 Upon imaging, the light field camera 17 creates light field image data 16.
  • the light field data 16 is read by a computer 19.
  • the computer 19 runs specialized image editing software 18 that adds enhanced 3D effects to the light field image data 16
  • the light field image data 16 is a digital representation of the original physical scene 15.
  • the light field image data 16 can be viewed, wherein the resulting image would appear as it does in Fig. 2.
  • a reference plane 24 is selected for the light field image data 16.
  • the reference plane 24 can be any plane in the light field image data 16 from which objects are to appear above and/or below.
  • the reference plane 24 is oriented with the ground upon which the toy dinosaur 22 stands.
  • the reference plane 24 of the light field image data 16, when displayed on an electronic display 12, is going to be oriented along the surface plane of the electronic display 12. As such, when the imagery 10 is viewed, any part of a subject imaged above the reference plane 24 will project vertically above, or in front of the display 12, depending on the orientation of the display 12.
  • any part of a subject imaged below the reference plane 24 will appear to be rearwardly projected and will appear below or behind the plane of the display 12, when the imagery 10 is viewed.
  • the reference plane 24 is selected to correspond with the plane of the paper upon which the imagery 10 is printed .
  • the light field camera 17 is positioned at an angle in front of and above the primary subject 20 in the physical scene 15.
  • the angle of elevation Al of the light field camera 17 is dependent upon the height of the subjects being imaged and the degree of surrounding area desired to be captured by the light field camera 17.
  • the field of view imaged by the light field camera 17 is limited and it is understood that the light field image data 16 is intended to be shown on an electronic display 12.
  • Most electronic displays are rectangular, having a width that is between 50% and 80% of the length. Accordingly, the light field image data 16 is created within boundaries appropriate in size and scale for a typical electronic display 12.
  • the boundaries include a front boundary 27, a rear boundary 28, and two side boundaries 29, 30. Any imagery 10 that is to be displayed on the electronic display 12 must exist within the boundaries 27, 28, 29, 30 in order to be seen.
  • a rear image boundary 28 is set for the light field image data 16. All of the subjects to appear in the final imagery 10 must be positioned forward of the rear image boundary 28.
  • the primary subject 20 has a height HI.
  • the light field camera 17 is set to a second height H2.
  • the second height H2 is a function of the subject height HI and the rear image boundary 28.
  • the second height H2 of the light field camera 17 is high enough so that the top of the primary subject 20, as viewed from the light field camera 17 does not extend above the rear image boundary 28. Accordingly, the elevation angle of the light field camera 17 depends upon the scene boundaries 27, 28, 29, 30 and height HI of the primary subject 20.
  • the light field image data 16 from the light field camera 17 is not used directly. Rather, the light field image data 16 digitally manipulated using the custom image editing software 18 being run by the computer 19.
  • the custom image editing software 18 is used to produce various enhance 3D effects.
  • the digital manipulations performed by the custom image editing software 18 includes, but is not limited to:
  • the digital manipulations that are used depend upon the details of the subjects being imaged.
  • Fig. 3 illustrates the possible tilt
  • the light field image data 16 that corresponds to the primary subject 20 and/or other imaged subjects can be digitally tilted toward or away from the position of the light field camera 17.
  • the primary subject 20 itself can be physically tilted prior to imaging.
  • one or more of the various subjects can be titled with an angle of inclination Al .
  • the preferred tilt angle is generally between 1 degree and 20 degrees from the original orientation.
  • the whole of the reference plane 24 can be tilted toward or away from the light field camera 17. This also can be achieved either digitally or physically.
  • the preferred tilt angle A2 is generally between 1 degree and 20 degrees from the horizontal, depending upon the final perceived height of the primary subject 20.
  • the reference plane 24 can be digitally and/or physically manipulated to tilt forward or backward.
  • the tilt angle T2 of the reference plane 24 and the tilt angle Tl of the primary subject 20 are independent of one another.
  • the tilting of the reference plane 24 changes the position of the rear image boundary 28 relative to the perceived position of the primary subject 20. This enables the height of the primary subject 20 to be increased proportionately within the confines of the mathematical relationship.
  • Fig. 4 in conjunction with Fig. 1 and Fig. 1, a preferred bend manipulation is explained.
  • the primary subject 20B is shown as a rectangle, rather than a dinosaur, for ease of explanation. A bend in the complex shape of a dinosaur would be difficult to perceive.
  • a bend point Bl is selected along the height of the primary subject 20B.
  • the bend point Bl is between 1/3 and 2/3 the overall height of the primary subject 20B.
  • the primary subject 20B is also divided into three regions 31, 33, 35 along its height. In the first region 31, the primary image 20B is not manipulated. In the second region 33, no manipulation occurs until the bend line Bl . Any portion of the primary subject 20B above the bend line Bl and within the second region 33 is digitally tilted by a first angle AAl. In the third region 35, the primary subject 20B is tilted at a second angle AA2, which is steeper than the first angle AAl.
  • the first angle AAl and the second angle AA2 are measured in relation to an imaginary vertical plane that is parallel to the vertical plane in which the light field camera 17 is set. The result is that the imagery 10 can be made larger and taller without extending above the rear image boundary 28. When viewed from the perspective of the light field camera 17, the primary subject 20B appears taller and has a more pronounced forward or vertical projection .
  • the primary subject 20B is shown as a representative rectangle, rather than a dinosaur for ease of explanation .
  • the primary subject 20B is divided into two regions 37, 39 along its height. In the first region 37, the primary subject 20B is not manipulated. In the second region 39, the primary subject 20B is reduced in size using a taper from front to back of an angle AA3 of between 1 degree and 25 degrees. The point where the taper begins is positioned between 1/3 and 2/3 up the height of the primary subject 20B. The result is that the imagery 10 can be made wider without extending beyond the side image boundaries 29, 30. When viewed, the primary subject 20B appears taller and has a more pronounced forward or vertical projection .
  • the primary subject 20B is shown as a representative rectangle, rather than a dinosaur for ease of explanation.
  • This imaginary line 41 represents the upper limit of what can be viewed with enhanced 3D effects.
  • the various subjects, including the primary subject 20 have heights that may not reach the imaginary line 41. If the height of a subject, such as the primary subject 20, is below the imaginary line 41, then the height of the primary subject 20B can be stretched vertically until it approaches the height of the imaginary line 41. The result is that all or some of the imaged subjects can be made taller without extending beyond the image boundaries When viewed, the primary subject 20B appears taller and has a more pronounced forward or vertical projection .
  • digitally enhanced image data 43 is created.
  • the digitally enhanced image data 43 is used to create two offset images 40, 42. Since the digitally enhanced image data 43 is initially obtained from a light field camera 17, this data can be skewed to produce two images from two different viewpoints in the same manner as traditional light field camera data . Referring to Fig. 7 and Fig. 8 in conjunction with Fig. 1 and Fig. 2, it can be seen that the two offset images 40, 42 can be considered as
  • Each offset image 40, 42 has a fading perspective due to the elevated orientation of the light field camera 17. This causes the front image boundary 27 to appear to be wider than the rear image boundary 28 in both offset images 40, 42.
  • a top view of one of the offset images 40, 42 from Fig. 7 or Fig. 8 is shown Although only one of the offset images is shown, it will be understood that the described process is performed on both of the offset images 40, 42.
  • the reference numbers 40, 42 of both offset images are used to indicate that the processes affect both
  • Temporary reference guides are superimposed upon the stereoscopic images 40. 42.
  • the reference guides include a set of inner guidelines 44 and a set of outer guidelines 46.
  • the inner guidelines 44 are parallel lines that extend from the rear image boundary 28 to the front image boundary 27.
  • the inner guidelines 44 begin at points P2 where in stereoscopic images 40, 42 met the rear boundary line 28.
  • the outer guidelines 46 are also parallel lines that extend from the rear image boundary 28 to the front image boundary 27.
  • the position of the outer guidelines 46 depends upon the dimensions of the electronic display 12 upon which the imagery 10 is to be displayed.
  • the width between the outer guidelines 46 corresponds to the pixel width of the electronic display to be used.
  • the peripheries of the offset images 40, 42 are digitally altered to fit within the parameters of the outer guidelines 46.
  • the offset images 40, 42 are widened toward the rear image boundary 28 and compressed toward the front image boundary 27. This creates corrected offset images 40A, 42A.
  • the inner guidelines 44 remain on the corrected offset images 40A, 42A.
  • the corrected left and right offset images 40A, 42A are superimposed.
  • the inner guidelines 44 from both corrected offset images 40A, 42A are aligned. Once alignment is achieved, the inner guidelines 44 are removed. This creates the final imagery 10.
  • the corrected offset images 40A, 42A can be colored in red or cyan.
  • the offset images 40A, 42A can be can be oppositely polarized or set into a side-by-side format. In this manner, when the final imagery 10 is viewed using 3D glasses or is viewed on an autostereoscopic display, the final imagery 48 will appear to be three-dimensional.
  • a content producer uses a light field camera 17 to image one or more subjects 20 that are to appear having enhanced 3D effects. See prior description of Fig. 1 and Fig. 2.
  • the content producer also selects a reference plane 24 for the light field image data produced by the light field camera. See Block 52. Using the reference plane 16 and the selected subjects 20, the content producer can determine the boundaries of the image field to be used by the final imagery 10. See Block 54.
  • the light field image data 16 collected by the light field camera 17 can be
  • the light field image data 16 is digitally altered using tilt manipulations, bend manipulations, taper
  • the enhanced image data 43 is then utilized to create two offset images 40, 42. See Block 64. Also see prior

Landscapes

  • Engineering & Computer Science (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Multimedia (AREA)
  • Signal Processing (AREA)
  • Testing, Inspecting, Measuring Of Stereoscopic Televisions And Televisions (AREA)
  • Processing Or Creating Images (AREA)

Abstract

L'invention concerne un système, un procédé et un logiciel destinés à produire des images présentant des effets 3D améliorés, visualisées sur un support d'affichage (14). Pour créer les images présentant des effets 3D améliorés, une caméra de champ lumineux (17) forme l'image d'une scène contenant au moins un sujet (20). A un moment donné dans le temps, le sujet (20) est positionné dans une orientation initiale. Lors de la formation d'images, la caméra de champ lumineux (17) produit des données d'images de champ lumineux (16) qui représentent la scène. Les données d'images de champ lumineux (16) sont modifiées afin d'y ajouter des effets 3D améliorés. Une fois modifiées, les données d'images améliorées (43) sont reproduites sur un support d'affichage (12), les effets 3D ajoutés faisant en sorte que le sujet semble, au moins en partie, s'étendre verticalement au-dessus du support d'affichage (14), ou devant celui-ci.
PCT/US2018/026516 2017-04-06 2018-04-06 Système, procédé et logiciel destinés à convertir des images capturées par une caméra de champ lumineux en images tridimensionnelles qui semblent s'étendre verticalement au-dessus d'un support d'affichage ou devant celui-ci WO2018187724A1 (fr)

Applications Claiming Priority (4)

Application Number Priority Date Filing Date Title
US15/481,447 US10136121B2 (en) 2016-04-08 2017-04-06 System, method and software for producing virtual three dimensional images that appear to project forward of or above an electronic display
US15/481,447 2017-04-06
US15/829,861 US10475233B2 (en) 2016-04-08 2017-12-01 System, method and software for converting images captured by a light field camera into three-dimensional images that appear to extend vertically above or in front of a display medium
US15/829,861 2017-12-01

Publications (2)

Publication Number Publication Date
WO2018187724A1 true WO2018187724A1 (fr) 2018-10-11
WO2018187724A8 WO2018187724A8 (fr) 2019-05-02

Family

ID=63712660

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/US2018/026516 WO2018187724A1 (fr) 2017-04-06 2018-04-06 Système, procédé et logiciel destinés à convertir des images capturées par une caméra de champ lumineux en images tridimensionnelles qui semblent s'étendre verticalement au-dessus d'un support d'affichage ou devant celui-ci

Country Status (1)

Country Link
WO (1) WO2018187724A1 (fr)

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20120026289A1 (en) * 2009-03-31 2012-02-02 Takeaki Suenaga Video processing device, video processing method, and memory product
US20120242777A1 (en) * 2011-03-21 2012-09-27 Sony Corporation Establishing 3d video conference presentation on 2d display
US8749620B1 (en) * 2010-02-20 2014-06-10 Lytro, Inc. 3D light field cameras, images and files, and methods of using, operating, processing and viewing same
US20170294052A1 (en) * 2016-04-08 2017-10-12 Maxx Media Group, LLC System, Method and Software for Producing Virtual Three Dimensional Images that Appear to Project Forward of or Above an Electronic Display

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20120026289A1 (en) * 2009-03-31 2012-02-02 Takeaki Suenaga Video processing device, video processing method, and memory product
US8749620B1 (en) * 2010-02-20 2014-06-10 Lytro, Inc. 3D light field cameras, images and files, and methods of using, operating, processing and viewing same
US20120242777A1 (en) * 2011-03-21 2012-09-27 Sony Corporation Establishing 3d video conference presentation on 2d display
US20170294052A1 (en) * 2016-04-08 2017-10-12 Maxx Media Group, LLC System, Method and Software for Producing Virtual Three Dimensional Images that Appear to Project Forward of or Above an Electronic Display

Also Published As

Publication number Publication date
WO2018187724A8 (fr) 2019-05-02

Similar Documents

Publication Publication Date Title
US7983477B2 (en) Method and apparatus for generating a stereoscopic image
JP4707368B2 (ja) 立体視画像作成方法および装置
US8953023B2 (en) Stereoscopic depth mapping
US7486817B2 (en) Apparatus for and method of generating image, and computer program product
CN101390131B (zh) 呈现输出图像
AU2018249563B2 (en) System, method and software for producing virtual three dimensional images that appear to project forward of or above an electronic display
EP2323416A2 (fr) Édition stéréoscopique pour la production vidéo, la post-production et l'adaptation d'affichages
WO2011052389A1 (fr) Dispositif et procédé de traitement d'image
US20110216160A1 (en) System and method for creating pseudo holographic displays on viewer position aware devices
US8094148B2 (en) Texture processing apparatus, method and program
JP2003209858A (ja) 立体画像生成方法及び記録媒体
KR20110049039A (ko) 능동 부화소 렌더링 방식 고밀도 다시점 영상 표시 시스템 및 방법
JP4489610B2 (ja) 立体視可能な表示装置および方法
JP2003348621A (ja) 二視点カメラの設定手段
CN119487830A (zh) 共享视场共同部分的相机的观看者调整立体图像显示器
KR101754976B1 (ko) 적층형 홀로그램용 콘텐츠 변환방법 및 변환장치
JP2012105172A (ja) 画像生成装置、画像生成方法、コンピュータプログラムおよび記録媒体
US10475233B2 (en) System, method and software for converting images captured by a light field camera into three-dimensional images that appear to extend vertically above or in front of a display medium
WO2018187724A1 (fr) Système, procédé et logiciel destinés à convertir des images capturées par une caméra de champ lumineux en images tridimensionnelles qui semblent s'étendre verticalement au-dessus d'un support d'affichage ou devant celui-ci
NZ757902B2 (en) System, method and software for producing virtual three dimensional images that appear to project forward of or above an electronic display
HK40022702B (en) System, method and software for producing virtual three dimensional images that appear to project forward of or above an electronic display
HK40022702A (en) System, method and software for producing virtual three dimensional images that appear to project forward of or above an electronic display
JP2005284183A (ja) 立体視画像作成方法および装置
JP2012010302A (ja) 画像表示装置および画像表示方法

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 18781716

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 18781716

Country of ref document: EP

Kind code of ref document: A1