[go: up one dir, main page]

WO2018187655A1 - Système et procédé de production d'images tridimensionnelles d'une production vidéo en direct qui semblent se projeter vers l'avant d'une unité d'affichage électronique ou au-dessus de celle-ci - Google Patents

Système et procédé de production d'images tridimensionnelles d'une production vidéo en direct qui semblent se projeter vers l'avant d'une unité d'affichage électronique ou au-dessus de celle-ci Download PDF

Info

Publication number
WO2018187655A1
WO2018187655A1 PCT/US2018/026406 US2018026406W WO2018187655A1 WO 2018187655 A1 WO2018187655 A1 WO 2018187655A1 US 2018026406 W US2018026406 W US 2018026406W WO 2018187655 A1 WO2018187655 A1 WO 2018187655A1
Authority
WO
WIPO (PCT)
Prior art keywords
location
electronic device
video file
display screen
images
Prior art date
Application number
PCT/US2018/026406
Other languages
English (en)
Inventor
Richard S. FREEMAN
Scott A. HOLLINGER
Original Assignee
Maxx Media Group, LLC
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Priority claimed from US15/481,447 external-priority patent/US10136121B2/en
Priority claimed from US15/665,423 external-priority patent/US10230939B2/en
Priority claimed from US15/705,142 external-priority patent/US10469803B2/en
Application filed by Maxx Media Group, LLC filed Critical Maxx Media Group, LLC
Publication of WO2018187655A1 publication Critical patent/WO2018187655A1/fr

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T15/00—3D [Three Dimensional] image rendering
    • G06T15/10—Geometric effects
    • G06T15/20—Perspective computation
    • G—PHYSICS
    • G03—PHOTOGRAPHY; CINEMATOGRAPHY; ANALOGOUS TECHNIQUES USING WAVES OTHER THAN OPTICAL WAVES; ELECTROGRAPHY; HOLOGRAPHY
    • G03B—APPARATUS OR ARRANGEMENTS FOR TAKING PHOTOGRAPHS OR FOR PROJECTING OR VIEWING THEM; APPARATUS OR ARRANGEMENTS EMPLOYING ANALOGOUS TECHNIQUES USING WAVES OTHER THAN OPTICAL WAVES; ACCESSORIES THEREFOR
    • G03B35/00—Stereoscopic photography
    • G03B35/08—Stereoscopic photography by simultaneous recording
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T19/00—Manipulating 3D models or images for computer graphics
    • G06T19/20—Editing of 3D images, e.g. changing shapes or colours, aligning objects or positioning parts
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
    • H04N13/10—Processing, recording or transmission of stereoscopic or multi-view image signals
    • H04N13/194—Transmission of image signals
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
    • H04N13/20—Image signal generators
    • H04N13/204—Image signal generators using stereoscopic image cameras
    • H04N13/207—Image signal generators using stereoscopic image cameras using a single 2D image sensor
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
    • H04N13/30—Image reproducers
    • H04N13/332—Displays for viewing with the aid of special glasses or head-mounted displays [HMD]

Definitions

  • the present invention relates to systems and methods that are used to create three- dimensional images that are displayed on an
  • the present invention relates to systems, methods used to creating three-dimensional images that are based upon live camera footage obtained by photographic or video recording of live subjects or real objects.
  • voice interaction interfaces Although interacting with a computer via a voice recognition interface is far more dynamic than a keypad or touch pad, it still has drawbacks .
  • the artificial intelligence algorithms used for voice interaction interfaces are very limited. As such, voice interaction interfaces can only answer certain basic queries and those queries must contain
  • communication queues used in the conversation are visual in natural .
  • the avatar can be customized and personalized by a user.
  • a similar system is found in U.S. Patent Application Publication No. 2006/0079325 to Trajkovic which shows an avatar system for smart phones .
  • the avatar can be customized, where aspects of the avatar are selected from a database.
  • An obvious problem with such prior art systems is that the video image of the person you are talking to or the avatar you are viewing is two- dimensional. Furthermore, if a smart phone is being used, the image being viewed is on a screen that may be less than two inches wide. Accordingly, much of the visual information being communicated can be difficult to see and easy to miss.
  • the present invention is a system and method for communicating between a first location and one or more remote locations .
  • a production set is established at the first location.
  • a first person or object is imaged with stereoscopic video cameras to obtain stereoscopic footage.
  • the stereoscopic footage is digitally enhancing with 3D effects to create a production video file.
  • the production video file is transmitted to an electronic device at a second location through a data network.
  • the electronic device has a display screen that can be viewed by a second person at the second location.
  • the production video file is played on the electronic device, wherein the production video fil creates images of the first person or object at the first location. On the display screen, the images appear three dimensional to the second person.
  • the images appear to extend in front or vertically above, the display screen of the electronic device .
  • FIG. 1 is a block diagram schematic
  • FIG. 2 is a diagram illustrating the present invention system being utilized in a first exemplary use
  • FIG. 3 is a diagram illustrating the present invention system being utilized in a second
  • FIG. 4 is a diagram illustrating the present invention system being utilized in a third exemplary use.
  • FIG. 5 is a diagram illustrating the present invention system being used for a conference call with multiple parties simultaneously.
  • 3D images for a variety of purposes, the embodiments illustrated show only a few exemplary applications of the technology. Additionally, although the 3D images can be displayed on any type of electronic display, the illustrated embodiments show the 3D images displayed on the screen of a smart phone and on a screen of a stationary display.
  • a communications system is established between at least two locations.
  • enhanced 3D images are transmitted in at least one direction.
  • the 3D images are created by imaging a live person, or some other object, at a live production set.
  • an exemplary embodiment is provided that shows a two-way communication system 10.
  • the communication system 10 transmits 3D images 12 in one direction.
  • the communication system 10 enables communications between a first person who creates the 3D images 12 and a second person who views the 3D images 12.
  • the 3D images 12 are viewed on the display screen 14 of an electronic device 16.
  • the display screen 14 is accessible by a viewer 18, wherein the 3D image 12 appears to have enhanced three dimensional effects when viewed.
  • the enhanced 3D effects cause the 3D images to appear to extend vertically above, or in front of, the display screen 14 of the electronic device 16 displaying the 3D images 12. If the
  • the electronic device 16 has a traditional display screen 14, such as an LED or LCD, then the 3D images 12 must be viewed through some form of 3D glasses 19 to appear three dimensional to the viewer 18. If the electronic device 16 has an auto-stereoscopic display or similar specialized 3D display, then the 3D image 12 can appear three dimensional to the naked eye .
  • the real person or object appearing in the 3D images 12 is referred to as the camera subject 20.
  • the camera subject 20 is imaged at a live production set 22.
  • the live production set 22 has stereoscopic video cameras 24 and other hardware necessary to image the camera subject 20 and produce the 3D images 12. It should be understood that the live production set 22 need not be an existing place. Rather, the live production set 22 can be created wherever needed. For example, the live production set 22 can be temporarily erected in a college classroom to create a 3D image 12 from a professor teaching a class.
  • the 3D images 12 are based upon the imaging of the camera subject 20. Therefore, the 3D images 12 move as the camera subject 20 at the live production set 22 moves .
  • the 3D images 12 also appear three dimensional, extending either vertically above or in front of the display screen 14 depending upon the orientation of the display screen 14.
  • the communication system 10 can be fully interactive. That is, not only can the camera subject 20 at the live production set 22 communicate with the viewer 18 through the 3D images 12, but the viewer 18 also can communicate with the camera subject 20. This two-way communication enables the viewer 18 of the 3D images 12 to ask a question and the camera subject 20 can react to the question and answer that question through the 3D images 12.
  • the creation of the 3D images 12 is first explained.
  • the live production set 22 there are at least one set of stereoscopic cameras 24.
  • stereoscopic cameras 24 include a left view camera and a right view camera.
  • the stereoscopic cameras 24 produce raw footage 26 that is processed by a production computer system 28.
  • the production computer system 28 runs specialized 3D effects software 30 that alters the raw footage 26 and superimposes the raw footage 26 from the
  • the 3D effects are engineered to cause the 3D image 12 to appear to extend vertically above, or in front of, the display screen 14 when viewed.
  • the operation of the 3D effects software needed to produce the stated 3D effects in 3D image 12 is explained in co-pending U.S. Patent Application No. 15/665,423, entitled "System, Method And Software For Producing Live Video Containing Three-Dimensional Images That Appear To Project Forward Of Or Vertically Above A Display", the disclosure of which is herein incorporated by reference.
  • the production video file 32 is transmitted to the electronic device 16 of the viewer 18 via a dat network 34.
  • the data network 34 can be a cellular network or a computer network, such as the World Wide Web.
  • the production video file 32 can be recorded for later play by a viewer. For example, i the electronic display 14 where an information station at a museum display, a prerecorded
  • production video file 32 can be played each time an information button is pressed at the museum display However, since the production video file 32 is created from a live camera subject 20, the
  • communication system 10 is especially well suited for live streaming the production video file 32 to the electronic device 16 of the viewer 18. In this manner, the viewer 18 can also stream questions and comments back to the camera subject 20 in the live production set 22.
  • the viewer 18 can send audio messages back to the live production set 22.
  • the electronic device 16 has a camera and a microphone, as do most smart phones and tablet computers, then the viewer 18 can send audio and video messages back to the live production set 22.
  • Messages sent upstream from the viewer 18 to the live production set 22 are displayed on an off- camera display 36 that can be viewed and/or heard by the camera subject 20, if the camera subject is a person.
  • the message can be fielded by an off-camera production worker. In either scenario, someone at the live production set 22 can hear and/or see questions and comments posted by the viewer 18.
  • the camera subject 20 a production worker, or a software operated AI assistant can then react to those questions and comments.
  • the viewer 18 receives reactions and answers from the camera subject 20 through the 3D images 12 extending vertically above, or in front of, the display screen 14 of the viewer's electronic device 16.
  • the production video file 32 can be augmented to better communicate
  • FIG. 1 it can be seen that the production computer system 28 has access to a 3D model database 38.
  • the 3D image database 38 contains virtual models 40 that have been previously created and engineered to contain enhanced 3D effects.
  • the enhanced 3D effects in the virtual models 40 are also the type that cause the virtual models 40 to appear to project forward, or above, the display screen 14 of the electronic device 16 when viewed.
  • the creation of virtual models 40 with such effects is disclosed in co-pending U.S. Patent Application No. 15/665,423, entitled "System, Method And
  • the selection of a virtual model 40 from the 3D image database 38 can be done by the camera subject 20 or someone else at the live production set 22 using an effects control unit 42.
  • the effects control unit 42 can be as simple as a computer mouse or slide show controller.
  • the camera subject 20 can select from a menu posted on an off-camera
  • the image or video can be recalled from the 3D model database 38 by a production assistant who understands the choreography of the presentation. If an AI assistant is being used, then the selection of a virtual model 40 may be managed by an automated system programmed to respond according to a predetermined routine, or to respond based on input or requests from a viewer.
  • the exemplary camera subject 20 is a professor 20A who is providing a live online lecture.
  • the viewer 18 may be a medical student who has a question about some aspect of the heart .
  • the viewer 18 communicates that question to the camera subject 20 using an electronic device 16 in the form of a smart phone 16A.
  • the professor 20A can recall a 3D virtual model 40 of a heart 40A from the 3D model database 38.
  • the professor 20A selects the image and
  • Such a communication system 10 is highly useful for customer support centers, wherein a customer support representative can
  • Augmenting the production video file 32 with a virtual model 40 requires more than simply
  • the virtual model 40 must be oriented to match the camera settings used in the creation of the production video file 32.
  • the camera subject 20 is being imaged by stereoscopic cameras 24.
  • orientation of the stereoscopic cameras 24, and the distance between the stereoscopic cameras 24 are all very important parameters when creating the enhanced 3D effects. These settings, among others, are read by the production computer system 28.
  • the virtual models 40 held in the 3D model database 38 are computer constructs . Accordingly, they can be viewed at any angle from any viewpoint in a virtual environment.
  • the selected virtual model 40 is configured to be at viewed with the same camera settings as are being used in the live
  • both images have continuity in viewpoint and camera settings. This makes the virtual model 40 and the 3D images 12 visually consistent in the same virtual environment. Being visually consistent, the
  • the viewer 18 is viewing the 3D image 12 or the augmented 3D image using his/her own electronic device 16.
  • the electronic display 16 has a traditional display that is designed to display 2D images. With such displays, 3D glasses 19 are required to see the 3D effects.
  • FIG. 3 an example is provided where a viewer 18 can utilize the present system without their own electronic device and without specialized 3D glasses.
  • FIG. 3 an exemplary reception
  • the reception station 42 may be the concierge desk at a hotel, an information desk at a hospital, or a kiosk in an amusement park. Regardless, the reception station 42 contains a display 44, a camera 46, a microphone 48 and a user interface 50. The reception station 42 is connected to a remote live production set 54 that contains one or more live receptionists 52.
  • a viewer 14 approaches the reception station 42 and activates the display 44.
  • the activation of the reception station may be achieved using passive sensors which detect an approaching user, or may be achieved via the user interface 50.
  • the display 44 here is preferably an auto stereoscopic display based on any technology capable of showing 3D stereoscopic images without the need for specialized glasses .
  • the reception station 42 is linked to the live production set 54 and a 3D image 56 appears on the display 44.
  • the live production set 54 can see and hear the viewer 18 via the camera 46 and microphone 48 at the reception station 42.
  • the user interface 50 at the reception station 42 may also contain a credit card reader, ticket dispenser, or other such device or integrated software that may assist in conducting business .
  • the viewer 18 can interact with the 3D image 56 in the same manner as if the receptionist 52 where
  • markers 62 can be worn by a camera subject 60 in a live production set 64.
  • the camera subject 60 is imaged in the manner previously described.
  • the production computer system 66 runs modeling software 68 that generates a virtual model 70 and manipulates the virtual model 70 as directed by the movement of the markers 62 on the camera subject 60.
  • the 3D effects that cause the virtual model 70 to appear to project forward or above the display screen when viewed are added.
  • the live avatar 72 can look like anyone or anything. In the shown example, the virtual live avatar 72 looks like a cartoon
  • Such a virtual live avatar 72 can be used at reception stations in an amusement park .
  • each production set 22A, 22B, 22C has a camera subject 12A, 12B, 12C with the before-described equipment and software needed to produce enhanced 3D images 82 of each camera subject 12A, 12B, 12C.
  • each production set 22A, 22B, 22C is provided with an electronic display 80 that can display the 3D images 82 from a different location.
  • Each electronic display 80 is part of electronic equipment that runs a conference call management software application 84.
  • the conference call management software application 84 enables any viewer at any of the live production sets to view the 3D image of any or all of the 3D images

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Computer Graphics (AREA)
  • Multimedia (AREA)
  • Signal Processing (AREA)
  • Geometry (AREA)
  • Computing Systems (AREA)
  • Architecture (AREA)
  • Computer Hardware Design (AREA)
  • General Engineering & Computer Science (AREA)
  • Software Systems (AREA)
  • Processing Or Creating Images (AREA)

Abstract

La présente invention concerne un système et un procédé permettant une communication entre une première personne à un premier emplacement et une ou plusieurs personnes à des emplacements distants. Un ensemble de production (22) est établi au niveau du premier emplacement où la première personne (20) est imagée avec des caméras vidéo stéréoscopiques (24). L'enregistrement stéréoscopique (26) est amélioré de manière numérique avec des effets 3D pour créer un fichier vidéo de production (32). Le fichier vidéo de production (32) est transmis à un dispositif électronique (16) au niveau d'un ou de plusieurs emplacements distants. Le fichier vidéo de production (32) est lu et crée des images de la première personne au niveau du premier emplacement. Sur un écran d'affichage (14) au niveau des emplacements distants, les images apparaissent en trois dimensions aux spectateurs distants. En outre, les images semblent s'étendre en avant de l'écran d'affichage (14) du dispositif électronique (16) ou au-dessus dudit écran.
PCT/US2018/026406 2017-04-06 2018-04-06 Système et procédé de production d'images tridimensionnelles d'une production vidéo en direct qui semblent se projeter vers l'avant d'une unité d'affichage électronique ou au-dessus de celle-ci WO2018187655A1 (fr)

Applications Claiming Priority (6)

Application Number Priority Date Filing Date Title
US15/481,447 US10136121B2 (en) 2016-04-08 2017-04-06 System, method and software for producing virtual three dimensional images that appear to project forward of or above an electronic display
US15/481,447 2017-04-06
US15/665,423 US10230939B2 (en) 2016-04-08 2017-08-01 System, method and software for producing live video containing three-dimensional images that appear to project forward of or vertically above a display
US15/665,423 2017-08-01
US15/705,142 2017-09-14
US15/705,142 US10469803B2 (en) 2016-04-08 2017-09-14 System and method for producing three-dimensional images from a live video production that appear to project forward of or vertically above an electronic display

Publications (1)

Publication Number Publication Date
WO2018187655A1 true WO2018187655A1 (fr) 2018-10-11

Family

ID=63712682

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/US2018/026406 WO2018187655A1 (fr) 2017-04-06 2018-04-06 Système et procédé de production d'images tridimensionnelles d'une production vidéo en direct qui semblent se projeter vers l'avant d'une unité d'affichage électronique ou au-dessus de celle-ci

Country Status (1)

Country Link
WO (1) WO2018187655A1 (fr)

Citations (9)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US6614427B1 (en) * 1999-02-01 2003-09-02 Steve Aubrey Process for making stereoscopic images which are congruent with viewer space
US20080079660A1 (en) * 2006-09-28 2008-04-03 Rieko Fukushima 3D image displaying method and apparatus
US20080246759A1 (en) * 2005-02-23 2008-10-09 Craig Summers Automatic Scene Modeling for the 3D Camera and 3D Video
US20110109720A1 (en) * 2009-11-11 2011-05-12 Disney Enterprises, Inc. Stereoscopic editing for video production, post-production and display adaptation
US20120263372A1 (en) * 2011-01-25 2012-10-18 JVC Kenwood Corporation Method And Apparatus For Processing 3D Image
US20130278727A1 (en) * 2010-11-24 2013-10-24 Stergen High-Tech Ltd. Method and system for creating three-dimensional viewable video from a single video stream
US20160086379A1 (en) * 2014-09-22 2016-03-24 Samsung Electronics Company, Ltd. Interaction with three-dimensional video
US20160180590A1 (en) * 2014-12-23 2016-06-23 Lntel Corporation Systems and methods for contextually augmented video creation and sharing
US20170294052A1 (en) * 2016-04-08 2017-10-12 Maxx Media Group, LLC System, Method and Software for Producing Virtual Three Dimensional Images that Appear to Project Forward of or Above an Electronic Display

Patent Citations (9)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US6614427B1 (en) * 1999-02-01 2003-09-02 Steve Aubrey Process for making stereoscopic images which are congruent with viewer space
US20080246759A1 (en) * 2005-02-23 2008-10-09 Craig Summers Automatic Scene Modeling for the 3D Camera and 3D Video
US20080079660A1 (en) * 2006-09-28 2008-04-03 Rieko Fukushima 3D image displaying method and apparatus
US20110109720A1 (en) * 2009-11-11 2011-05-12 Disney Enterprises, Inc. Stereoscopic editing for video production, post-production and display adaptation
US20130278727A1 (en) * 2010-11-24 2013-10-24 Stergen High-Tech Ltd. Method and system for creating three-dimensional viewable video from a single video stream
US20120263372A1 (en) * 2011-01-25 2012-10-18 JVC Kenwood Corporation Method And Apparatus For Processing 3D Image
US20160086379A1 (en) * 2014-09-22 2016-03-24 Samsung Electronics Company, Ltd. Interaction with three-dimensional video
US20160180590A1 (en) * 2014-12-23 2016-06-23 Lntel Corporation Systems and methods for contextually augmented video creation and sharing
US20170294052A1 (en) * 2016-04-08 2017-10-12 Maxx Media Group, LLC System, Method and Software for Producing Virtual Three Dimensional Images that Appear to Project Forward of or Above an Electronic Display

Similar Documents

Publication Publication Date Title
US11403595B2 (en) Devices and methods for creating a collaborative virtual session
US10917613B1 (en) Virtual object placement in augmented reality environments
Orlosky et al. Virtual and augmented reality on the 5G highway
Vertegaal The GAZE groupware system: mediating joint attention in multiparty communication and collaboration
US6466250B1 (en) System for electronically-mediated collaboration including eye-contact collaboratory
US10200654B2 (en) Systems and methods for real time manipulation and interaction with multiple dynamic and synchronized video streams in an augmented or multi-dimensional space
Fink et al. Re-locations: Augmenting personal and shared workspaces to support remote collaboration in incongruent spaces
US8644467B2 (en) Video conferencing system, method, and computer program storage device
JPH07255044A (ja) アニメーション式電子会議室及びビデオ会議システム及びビデオ会議方法
CN111386517A (zh) 用于体验虚拟现实的用户之间的通信的装置和相关方法
WO2003058518A2 (fr) Procede et dispositif de systeme d'interface pour utilisateur d'avatars
JP2017511615A (ja) 物理的場所間のビデオ対話
US12032735B2 (en) Method and apparatus for real-time data communication in full-presence immersive platforms
Steptoe et al. Acting rehearsal in collaborative multimodal mixed reality environments
Regenbrecht et al. Mutual gaze support in videoconferencing reviewed
US20240211093A1 (en) Artificial Reality Coworking Spaces for Two-Dimensional and Three-Dimensional Interfaces
Edwards Telepresence: Virtual reality in the real world [special reports]
Dijkstra-Soudarissanane et al. Virtual visits: life-size immersive communication
US10469803B2 (en) System and method for producing three-dimensional images from a live video production that appear to project forward of or vertically above an electronic display
US10839593B2 (en) System, method and software for adding three-dimensional images to an intelligent virtual assistant that appear to project forward of or vertically above an electronic display
WO2018187640A1 (fr) Système, procédé et logiciel destinés à produire des avatars tridimensionnels virtuels qui répondent activement à des signaux audios tout en semblant être projetés vers l'avant ou au-dessus d'une unité d'affichage électronique
US20240256711A1 (en) User Scene With Privacy Preserving Component Replacements
Lang The impact of video systems on architecture
Chen Collaboration in Multi-user Immersive Virtual Environment
Schäfer Improving essential interactions for immersive virtual environments with novel hand gesture authoring tools

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 18781912

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 18781912

Country of ref document: EP

Kind code of ref document: A1