[go: up one dir, main page]

WO1999026198A2 - Systeme et procede pour fusionner des objets en une suite d'images sans connaitre a priori la scene representee dans ladite suite d'images - Google Patents

Systeme et procede pour fusionner des objets en une suite d'images sans connaitre a priori la scene representee dans ladite suite d'images Download PDF

Info

Publication number
WO1999026198A2
WO1999026198A2 PCT/SG1998/000092 SG9800092W WO9926198A2 WO 1999026198 A2 WO1999026198 A2 WO 1999026198A2 SG 9800092 W SG9800092 W SG 9800092W WO 9926198 A2 WO9926198 A2 WO 9926198A2
Authority
WO
WIPO (PCT)
Prior art keywords
image sequence
set forth
clone
objects
features
Prior art date
Application number
PCT/SG1998/000092
Other languages
English (en)
Inventor
Kiem Ching Ong
Heng Chuan Teh
Tiow Seng Tan
Original Assignee
National University Of Singapore
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by National University Of Singapore filed Critical National University Of Singapore
Publication of WO1999026198A2 publication Critical patent/WO1999026198A2/fr

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T17/00—Three dimensional [3D] modelling, e.g. data description of 3D objects

Definitions

  • the present invention relates generally to computer graphics, image processing and augmented reality, and more particularly, to systems and methods for merging computer-generated objects or any images into a video or a sequence of photographs.
  • World Patent No. 95/30312 "Improved Chromakeying System” issued to Tamir et al., 9 November 1995
  • World Patent No. 97/26758 "Method and Apparatus for Insertion of virtual Objects into a Video Sequence” issued to Goodman, 24 July 1997. They discuss the simple occlusion of inserting objects/images in front of all other objects in the video sequence.
  • a qualitative approach on modeling the depth information of objects in a video sequence for the purpose of video compression is disclosed in World Patent No. 96/29678, “Method and Apparatus for Depth Modeling and providing Depth Information of Moving Objects” issued to Martens et al., 26 September 1996. Such qualitative assessment of depth is inadequate for our purposes.
  • the object of this invention to provide methods and apparatus to seamlessly merge objects into an image sequence, without a priori knowledge about the real scene nor the availability of any scene model, and without much user intervention.
  • the inserted objects may be static or dynamic like 2D windows or 3D virtual objects, and the image sequence may be monocular video sequence or a collection of photographs of a virtual or real environment that can be interpolated into a video sequence.
  • the invention can be used to build a software system to assist an architect evaluate the impact of his or her design by merging a computer model of the architecture into a video of the location where it is to be built.
  • the augmented video conveys the idea more realistically compared to the conventional methods of using still images or physical models.
  • changes in the design can be more readily realized and visualized in such a system.
  • the invention can also find its usage in the movie industry, especially for special effects and in computer-animated films.
  • Computer animation in real environment which can be seen in today's TV commercials, can also be created using this invention.
  • the invention described herein satisfies the above-identified needs. It provides methods and apparatus to seamlessly merge objects into an image sequence to obtain a montage image sequence, without knowing a priori the scene content of the image sequence. Such montage image sequence can be shown to a viewer or recorded for future display. The inventors have determined that many details in the process of merging can be automated with little user intervention.
  • Occlusion is the phenomenon that allows one to only see the front object when two objects fall on the same line of sight. It is one of the most important visual cues that helps us perceive depth, thus survive in a 3D environment.
  • correct occlusion among the inserted objects and the objects in the image sequence is resolved to ensure convincing merging.
  • the basic principle employed in this invention to resolve occlusion is to construct a 3D clone for each important object of the scene in the image sequence.
  • Such 3D clones serving the same purpose of the real object in the image sequence, provide the proper occlusion to objects to be merged into the image sequence.
  • Each 3D clone is usually an approximation of its corresponding real object since the complete information of the real object may not be available in the image sequence. More importantly, each 3D clone, when projected into the image sequence with the correct camera parameters, matches closely the image of the real object in the image sequence.
  • the process of constructing 3D clones is mostly automated with minimal user intervention.
  • the method processes the image sequence to derive relative locations of important features (such as corners of buildings) as well as the camera parameters for each image in the image sequence.
  • the method obtains silhouettes of selected objects in selected images in the image sequence, either through automated or semi-automated segmentation techniques. With all this information, the recovered 3D features can be clustered according to the real objects they belong and then initial estimates of the 3D clones can then be derived, which are later improved using some optimization methods.
  • the method includes the following steps:
  • Structure Recover selected features of objects in the input image sequence are tracked whereby the spatial locality of these features are then computed.
  • FIG. 1 is a block diagram of an exemplary raster graphics systems
  • FIG. 2 is a block diagram of the raster graphics system of FIG. 1 that shows the input data
  • FIG. 3 is a flowchart that depicts the steps of the method of the invention.
  • FIG. 4 is a flowchart that depicts the operation of the method of the invention.
  • FIG. 5 is an approximate 3D clone that projects to real object silhouettes
  • FIG. 6 is an initial estimate ( ⁇ w f ) of camera position (p f );
  • FIG. 7 shows the clone construction process for an exemplary box in (a);
  • FIG. 8 is a mapping of viewing frustum to a 2D physical window
  • FIG. 9 shows the slices of voxels used in the construction of clone
  • FIG. 10 shows the rendering pipeline used to produce a merged image
  • FIG. 11 is a transformation function that maps viewing coordinates to graphics subsystem coordinates
  • FIG. 12 shows how the depth value is rectified when objects are rendered using different projection models.
  • FIG. 1 illustrates an exemplary raster graphics system, which includes a main (Host) processor and a graphics subsystem.
  • the Host processor executes an application program and dispatches graphics tasks to the graphics subsystem.
  • the graphics subsystem includes a pipeline of several components that perform operations necessary to prepare geometric entities for display on a raster display device.
  • a model of the graphics subsystem is employed that contains the following functional units. It should be realized that this particular model is not to be construed in a limiting sense upon the practice of the invention.
  • a Geometric Processor unit performs geometric and perspective transformations, exact clipping on primitives against screen (window) boundaries, as well as lighting computations.
  • the resulting graphics primitives e.g. points, lines, triangles, etc., are described in screen space (integral) coordinates.
  • a Scan Conversion (Rasterization) unit breaks down the graphics primitives into raster information, i.e. a description of display screen pixels that are covered by the graphics primitives.
  • a Graphics Buffer unit receives, stores, and processes the pixels from the Scan Conversion unit in accordance with the method of the invention.
  • the most common buffers associated to the Graphics Buffer unit are the frame buffer and the z-buffer.
  • the frame buffer is where we store the RGB color components (in
  • the z-buffer is used primarily for visible surface determination. It keeps track of the nearest z-coordinate (distance to the camera) at each pixel. Before we write any incoming pixel value into the frame buffer, we compare the z-coordinate of the geometry to that of the z-buffer. If the incoming z value shows that the new geometry is closer to the camera than the existing geometry, the value of the old pixel in the frame buffer and the old z value in the z-buffer are replaced by the new ones.
  • a Display unit receives pixels from the Graphics Buffer unit and transforms these pixels into information to either be displayed on the output device or recorded on some storage devices for future display.
  • FIG. 3 is a flowchart that depicts the flow of these steps of the invention.
  • Step (1) sparse 3D features from the scene using structure-from- motion algorithm from computer vision.
  • some 2D image features in the starting frame are identified and subsequently tracked across the whole image sequence. These 2D image coordinates are then used to derive the 3D scene coordinates of the features.
  • Step (2) the recovered 3D features together with their corresponding 2D features are used to derive the unknown camera parameters like the zoom factor and the camera position in each frame. This is simply a minimization problem where the camera parameters are adjusted such that the difference between the tracked 2D features and the computed 2D projections of the corresponding 3D features is minimized.
  • Step (3) some frames are also selected, either automatically or semi-automatically, from the image sequence as key frames where the object silhouettes of some selected real objects are to be segmented.
  • the output from Step (3) on object silhouettes may be utilized by Step (1), in which case Step (3) is run before Step (1) and (2), otherwise, Step (3) can run in parallel to Step (1) and (2).
  • Step (4) approximate models, which are called clones to the real objects.
  • these clones when projected onto each key frame of the image sequence, are to cover exactly the silhouettes of their respective real objects as shown in FIG. 5.
  • the subsequent merging process in Step (5) is simply a depth comparison where proper occlusion is realized.
  • Solving structure-from-motion problem using long image stream has two advantages. Firstly, feature correspondence problem between successive frames can be more easily solved due to small change in the image content. Secondly, data redundancy can be exploited to counteract noise in the images.
  • the structure-from-motion problem with perspective projection is inherently a nonlinear minimization problem. This class of problems is susceptible to local minima in the parameter space and a good formulation is essential to better manage its complexity during minimization. In Szeliski and Kang, "Recovering 3D shape and motion from image streams using non-linear least squares" (Journal of Visual Communications and Image Representation, 5(1), pp.
  • the factorization method or any other structure-from- motion algorithms may be modified to work with video sequence that violates the specific requirements of the type of video sequence required by them. For example, a video sequence that moves towards or away from the scene can be corrected, with some user guidance, by incorporating scaling factors to correct each column in the matrix W.
  • this step of structure recovery can work with a collection of photographs instead of a video sequence.
  • a collection of photographs instead of a video sequence.
  • it is possible to derive other images between the two see, for example, Mark, McMillan and Bishop, "Post-rendering 3D Warping", Proceedings of ACM Symposium on 3D Interactive Graphics, 1997, pp. 7 — 16).
  • a video sequence of the scene can be derived and treated as input to any structure-from-motion algorithms as before.
  • Structure-from-motion algorithms in general, do not output a complete description of camera parameters.
  • the factorization method produces only the orientation R of the camera but not the position and zoom factor of the camera in each frame.
  • 4F camera parameters to be derived, namely 3F camera position coordinates (3 coordinates per frame) and F zoom factors.
  • 3F camera position coordinates (3 coordinates per frame)
  • F zoom factors Given the tracked 2D features and the recovered 3D features, we would like to compute the 4F parameters such that the 3D features are as closely projected to their respective 2D features as possible in all the frames.
  • P number of feature points.
  • a vector containing the 4F camera parameters to be fitted.
  • One other possible error function is to use maximum distance instead of summation in the above formula.
  • the features are line segments, besides choosing appropriate points on these line segments to compute the error function based on distance as before, a different metric like area or angle between the tracked and the projected line features can also be employed.
  • the Levenberg-Marquardt minimization is an iterative method that requires an initial estimate of a.
  • the initial estimates of the camera positions (p f ) in each frame may be yw f where ⁇ is some constants (u f , v f and w f define the camera orientation which is obtained from matrix R in the factorization method), and the initial estimates of zoom factors may be 1.
  • One may terminate the iteration process when the fitted parameters do not improve a few times in a row.
  • the 4F parameters may be reduced to 3F+1 if the camera is having the same zoom factor for all the frames.
  • the structure-from-motion algorithm (Step (1)) only yields a sparse set of 3D features of the real objects, which is insufficient for us to construct 3D computer models, called clones.
  • clones 3D computer models
  • a few frames are selected automatically or semi-automatically from the image sequence as key frames where the silhouettes of some selected real objects are to be segmented.
  • a frame should be made a key frame when there is substantial geometrical change like the appearance and disappearance of facets on real objects in the image.
  • the more key frames selected the more accurate will be the constructed clones.
  • the method of this invention relies on correct object silhouette segmentation in the key frames for the construction of the clones. From the image sequence, users decide which of the objects need to build clones, and then obtain their silhouettes through automated or semi-automated tools such as those (or their variants) mentioned above. ln another embodiment, segmentation of objects in all frames may be used to build clones. This can be achieved as before by making all frames as key frames, or intelligently interpolate object silhouettes from a few key frames to all other frames by taking advantage of coherency in the image sequence. For example, object silhouette obtained in one frame may be used as the initial estimate of the silhouette of the same object in the next frame.
  • segmentation results of this step may be utilized by structure recovery (Step (1)) to better select important features for tracking and to better track them across the image sequence.
  • Step 4.1 Initial model definition
  • FIG. 7a shows the object silhouettes (see FIG. 7b) defined in the key frames.
  • FIG. 7b shows the object silhouettes (see FIG. 7b) defined in the key frames.
  • FIG. 7c shows the world-coordinate-axis-aligned bounding box of the derived 3D features (in this case, feature points) on the box.
  • an approximated bounding volume (rather than a bounding box) of the object may be supplied by users to expedite the subsequent processing.
  • Step 4.2 Model orientation determination
  • the centroid-based coordinate system has an arbitrary orientation.
  • Having a correct orientation, though not critical, can speed up subsequent processing.
  • users may be provided with an interactive mechanism to change the orientation of the initial model.
  • FIG. 7d shows that user has interactively adjusted (select the square tag, near the upper right hand corner, and drag) the orientation of the bounding box so that it is closer to that of the white box.
  • Step 4.3 Model enlargement
  • the algorithm proceeds by going through all the key frames in turn to progressively enlarge the model. For each key frame, it computes the 2D (axis parallel) bounding box of the object silhouette and locates two points (possibly one point in degenerate cases), each on the object silhouette, that are furthest apart on each boundary edge of the bounding box.
  • each of these 8 (or less) points it checks any enlargement is necessary as in the next paragraph. For each of the 8 (or less) points, it casts a ray from the point on the image plane to the 3D space using the computed camera parameters from Step 2. If the ray intersects the current model, then no enlargement is necessary. Otherwise, the ray is outside the current model indicating that enlargement is necessary because the model does not project to cover this point. In this case, it finds the corner, say C, of the current model closest to the ray. Let 3D point (x, y, z) be the point on the ray closest to C.
  • This step trims away 3D region on the model which does not project to the 2D object silhouette.
  • a voxel representation for the model.
  • Each model is initially subdivided into voxels of similar size.
  • the longest possible straight line enclosed within the 2D projection of a 3D voxel is the diagonal of its opposite corners.
  • x is the dimension of the voxel, then the length of the diagonal of its opposite corners is 3 x.
  • 3 x must be equivalent to one-pixel width.
  • the viewing frustum defined in the computer will be eventually mapped to the 2D physical window, which has the same dimension as the real image.
  • the 2D physical window which has the same dimension as the real image.
  • frustum height x 3 x image height
  • All the voxels of the model are initially assigned "OUT" to indicate that they are not part of the model.
  • the word "trim” actually means form the model rather than trim away unwanted region from the model in the following algorithm.
  • Step 4.5 Dynamic patch voxel derivation
  • the patch voxels needed for each key frame is computed with two passes through all pixels within the object silhouette.
  • the second pass we create patch voxels for pixels marked in the first pass. For each such pixel, it estimates the d 1 (and d 2 respectively) of the pixel as the average of the d 1 values (and d 2 values respectively) of surrounding 8 (or less) pixels. Note that the estimation has to be done in some order of pixels closest to pixels with known d 1 and d 2 values and so on. Next, it casts a ray from the pixel so as to encode the ray segment between d,and d 2 as patch voxels.
  • each key frame has a set of 3D patch voxels, which can be employed in the following manner.
  • the frame is a key frame
  • the clone for intermediate frames, which are not key frames, we may define the clone to be the over-trimmed model together with the patch voxels of the nearest key frame. So, when we step through the image sequence, the clone, which is the over-trimmed model, is dynamically patched up using patch voxels belonging to different key frames.
  • FIG. 7g and FIG. 7h show two different views of the clone with patch voxels drawn.
  • the content of the non-key frames may be analyzed automatically to include only selected patch voxels, possibly from all the key frames.
  • the over-trimming of a clone is the result of either occlusion of the object in some key frame or inaccuracy in the computed camera parameters.
  • the accuracy of the computed camera parameters can be improved by formulating another optimization process to minimize, using the Levenberg- Marquardt algorithm, the number of patch voxels due to inaccuracy in the computed camera parameters.
  • another more time-efficient method is to simply use a 3D plane of any polygonal shape.
  • this 3D plane will be masked by the object silhouettes in the frames, like being cut using cookie cutters.
  • This method requires the object silhouettes in the key frames to be interpolated to all the non-key frames.
  • its silhouettes in the key frames are firstly broken into vertices and the vertices from the silhouettes of two consecutive key frames are paired up according to proximity. (A vertex can be defined as a pixel location where the boundary changes direction.) This effectively generates a 2D morphing path for the vertices on the silhouette boundary.
  • the 3D plane will be centered at this centroid with its orientation always directly facing the camera.
  • the size of the 3D plane is computed in such a way that when the plane is projected to the screen, it at least covers the entire silhouette in the frame. With such an approach, the 3D clone (in terms of a 3D plane masked by the silhouettes) will project to its object silhouette in every frame.
  • Lighting parameters of the image sequence are important in rendering computer objects so that these objects blend nicely to the scene in the image sequence.
  • Other relevant references in the area also include: (1) Foumier, Gunawan and Romanzin, "Common illumination between real and computer generated scenes", Proceedings of Graphics Interface, 1993, pp.
  • each image of the merged image sequence can be produced with the following rendering pipeline.
  • Step 5.1 Display image in frame buffer (refer to FIG. 10a).
  • Step 5.2 Render each clone (both over-trimmed model and patch voxels) in only the z-buffer. This step ensures that the real object has its depth information registered in the Graphics Buffer unit (refer to FIG. 10b).
  • Step 5.3 Render each inserted object in both frame buffer and z-buffer. Each of these computer objects will appear occluded in the merged image if it is behind some real objects (refer to FIG. 10c).
  • the z-values used in depth comparison may no longer conform to the correct viewing order. For example, assuming that point A and B fall on the same line of sight and A is closer to the camera. When both of the points are rendered with the same projection model, point A will always occlude point B. However, when A is rendered with perspective and B with orthographic projection, the visibility order may be wrong even though point A is still closer to the camera. This is because the mapping of the depth values from the viewing coordinate to the graphics subsystem coordinate is different for
  • z system z value in graphics subsystem coordinate; z v ⁇ ew : z value in viewing coordinate; near : distance between camera and near plane of viewing frustum; far : distance between camera and far plane of viewing frustum.
  • mapping is linear under orthographic projection but non-linear under perspective projection.
  • a and B occurs when they take the values indicated in the graph: A is in front of B when they are both rendered with the same projection model, but A appears behind B when A is rendered with perspective and B with orthographic projection.
  • the depth values of the clones can be adjusted in the following manner.
  • R 1 is mapped to T in the graphics subsystem coordinate. Since we use orthographic projection to render the clone, we therefore must compute the range of values in the viewing coordinate that maps to T under orthography. Refer to FIG. 12, R 2 turns out to be the required range.
  • R 1 is mapped to T in the graphics subsystem coordinate.
  • R 2 turns out to be the required range.
  • FIG. 4 is a flowchart that depicts the operation of the method of the invention.
  • the system loads an image sequence.
  • This may be an image sequence of a site where a new infrastructure is to be constructed.
  • the input image sequence may be processed by image processing tools that, for example, filter or mask out unnecessary details or noise before further processing by the method described in this invention.
  • a clone is constructed as depicted in FIG. 3.
  • Some user interactions may be necessary in the silhouette computation, and further information like the speed of the camera or the path of the camera while taking the image sequence is helpful to ease the computation of structure and camera parameters recovery.
  • a portion of the image sequence is sufficient to complete the structure and motion recovery, and the computed result may be extended to the full image sequence.
  • the image sequence and clones for objects are stored in the memory as shown in FIG. 2. At this point, the system is ready to begin an interactive display session with a user.
  • Commands from the user may include the change of the position and the lighting parameters for the inserted object as well as the viewing of the image sequence from a novel viewpoint (see, for example, Horry, Anjyo and Arai, "Tour into the pictures: Using a spidery mesh interface to make animation from a single image", Proceedings of SIGGRAPH '97, pp. 225—232).
  • the system accesses the memory to obtain clone and image information and computes, using interpolation when necessary, the merged image sequence which is either displayed by the graphics subsystem on the display unit or recorded for future display.
  • collision detection between the inserted objects and the clones in the image sequence may also be computed. This is useful when an inserted object is to sit on the real floor/ground in the image sequence for example.

Landscapes

  • Physics & Mathematics (AREA)
  • Engineering & Computer Science (AREA)
  • Computer Graphics (AREA)
  • Geometry (AREA)
  • Software Systems (AREA)
  • General Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Processing Or Creating Images (AREA)

Abstract

L'invention concerne un procédé exécutable par une machine de traitement des données capable de fusionner sans coupure des objects en une suite d'images, pour réaliser un montage sans connaître à priori le contenu de la scène représentée dans la suite d'images d'entrée. La suite d'images montée peut être visualisée par un visualiseur ou enregistrée pour un affichage ultérieur. Le procédé se réalise selon les étapes suivantes: récupération de structure, détermination des paramètres de caméra, calcul de silhouette, construction de clone et extrapolation. La plupart de ces étapes sont automatisées ou semi-automatisées et requièrent une intervention minimale de l'utilisateur. Le procédé se met en oeuvre de manière interactive par le calcul, l'affichage et l'enregistrement d'une suite d'images fusionnée en réponse à des instructions de l'utilisateur concernant, par exemple, le replacement d'objets insérés, la modification des paramètres d'éclairage, etc.
PCT/SG1998/000092 1997-11-14 1998-11-12 Systeme et procede pour fusionner des objets en une suite d'images sans connaitre a priori la scene representee dans ladite suite d'images WO1999026198A2 (fr)

Applications Claiming Priority (4)

Application Number Priority Date Filing Date Title
US6580597P 1997-11-14 1997-11-14
US60/065,805 1997-11-14
US18820298A 1998-11-09 1998-11-09
US09/188,202 1998-11-09

Publications (1)

Publication Number Publication Date
WO1999026198A2 true WO1999026198A2 (fr) 1999-05-27

Family

ID=26746015

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/SG1998/000092 WO1999026198A2 (fr) 1997-11-14 1998-11-12 Systeme et procede pour fusionner des objets en une suite d'images sans connaitre a priori la scene representee dans ladite suite d'images

Country Status (1)

Country Link
WO (1) WO1999026198A2 (fr)

Cited By (35)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2004107272A1 (fr) * 2003-05-29 2004-12-09 Sony Computer Entertainment Inc. Systeme et procede de fourniture d'un environnement interactif tridimensionnel en temps reel
EP1944700A1 (fr) * 2007-01-10 2008-07-16 Imagetech Co., Ltd. Procédé et système pour vidéo interactive en temps-réel
WO2008139251A3 (fr) * 2006-04-14 2009-03-12 Rosenthal Patrick Levy Dispositif de caméra vidéo virtuelle avec suivi tridimensionnel et insertion d'objet virtuel
US7623115B2 (en) 2002-07-27 2009-11-24 Sony Computer Entertainment Inc. Method and apparatus for light input device
US7627139B2 (en) 2002-07-27 2009-12-01 Sony Computer Entertainment Inc. Computer image and audio processing of intensity and input devices for interfacing with a computer program
US7639233B2 (en) 2002-07-27 2009-12-29 Sony Computer Entertainment Inc. Man-machine interface using a deformable device
WO2011029209A3 (fr) * 2009-09-10 2011-09-29 Liberovision Ag Procédé et appareil destinés à générer et à traiter des images à profondeur accrue
US8139793B2 (en) 2003-08-27 2012-03-20 Sony Computer Entertainment Inc. Methods and apparatus for capturing audio signals based on a visual image
US8160269B2 (en) 2003-08-27 2012-04-17 Sony Computer Entertainment Inc. Methods and apparatuses for adjusting a listening area for capturing sounds
US8233642B2 (en) 2003-08-27 2012-07-31 Sony Computer Entertainment Inc. Methods and apparatuses for capturing an audio signal based on a location of the signal
US8279222B2 (en) 2008-03-14 2012-10-02 Seiko Epson Corporation Processing graphics data for a stereoscopic display
US8542907B2 (en) 2007-12-17 2013-09-24 Sony Computer Entertainment America Llc Dynamic three-dimensional object mapping for user-defined control device
US8840470B2 (en) 2008-02-27 2014-09-23 Sony Computer Entertainment America Llc Methods for capturing depth data of a scene and applying computer actions
US8961313B2 (en) 2009-05-29 2015-02-24 Sony Computer Entertainment America Llc Multi-positional three-dimensional controller
US8976265B2 (en) 2002-07-27 2015-03-10 Sony Computer Entertainment Inc. Apparatus for image and sound capture in a game environment
US9177387B2 (en) 2003-02-11 2015-11-03 Sony Computer Entertainment Inc. Method and apparatus for real time motion capture
US9174119B2 (en) 2002-07-27 2015-11-03 Sony Computer Entertainement America, LLC Controller for providing inputs to control execution of a program when inputs are combined
US9381424B2 (en) 2002-07-27 2016-07-05 Sony Interactive Entertainment America Llc Scheme for translating movements of a hand-held controller into inputs for a system
US9393487B2 (en) 2002-07-27 2016-07-19 Sony Interactive Entertainment Inc. Method for mapping movements of a hand-held controller to game commands
US9474968B2 (en) 2002-07-27 2016-10-25 Sony Interactive Entertainment America Llc Method and system for applying gearing effects to visual tracking
US9573056B2 (en) 2005-10-26 2017-02-21 Sony Interactive Entertainment Inc. Expandable control device via hardware attachment
US9589362B2 (en) 2014-07-01 2017-03-07 Qualcomm Incorporated System and method of three-dimensional model generation
US9607388B2 (en) 2014-09-19 2017-03-28 Qualcomm Incorporated System and method of pose estimation
US9682320B2 (en) 2002-07-22 2017-06-20 Sony Interactive Entertainment Inc. Inertially trackable hand-held controller
US9682319B2 (en) 2002-07-31 2017-06-20 Sony Interactive Entertainment Inc. Combiner method for altering game gearing
US9710554B2 (en) 2010-09-23 2017-07-18 Nokia Technologies Oy Methods, apparatuses and computer program products for grouping content in augmented reality
US9911242B2 (en) 2015-05-14 2018-03-06 Qualcomm Incorporated Three-dimensional model generation
US10099147B2 (en) 2004-08-19 2018-10-16 Sony Interactive Entertainment Inc. Using a portable device to interface with a video game rendered on a main display
US10220302B2 (en) 2002-07-27 2019-03-05 Sony Interactive Entertainment Inc. Method and apparatus for tracking three-dimensional movements of an object using a depth sensing camera
US10279254B2 (en) 2005-10-26 2019-05-07 Sony Interactive Entertainment Inc. Controller having visually trackable object for interfacing with a gaming system
US10304203B2 (en) 2015-05-14 2019-05-28 Qualcomm Incorporated Three-dimensional model generation
US10341568B2 (en) 2016-10-10 2019-07-02 Qualcomm Incorporated User interface to assist three dimensional scanning of objects
US10373366B2 (en) 2015-05-14 2019-08-06 Qualcomm Incorporated Three-dimensional model generation
CN110827380A (zh) * 2019-09-19 2020-02-21 北京铂石空间科技有限公司 图像的渲染方法、装置、电子设备及计算机可读介质
USRE48417E1 (en) 2006-09-28 2021-02-02 Sony Interactive Entertainment Inc. Object direction using video input combined with tilt angle information

Cited By (39)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US9682320B2 (en) 2002-07-22 2017-06-20 Sony Interactive Entertainment Inc. Inertially trackable hand-held controller
US9393487B2 (en) 2002-07-27 2016-07-19 Sony Interactive Entertainment Inc. Method for mapping movements of a hand-held controller to game commands
US9474968B2 (en) 2002-07-27 2016-10-25 Sony Interactive Entertainment America Llc Method and system for applying gearing effects to visual tracking
US7623115B2 (en) 2002-07-27 2009-11-24 Sony Computer Entertainment Inc. Method and apparatus for light input device
US7627139B2 (en) 2002-07-27 2009-12-01 Sony Computer Entertainment Inc. Computer image and audio processing of intensity and input devices for interfacing with a computer program
US7639233B2 (en) 2002-07-27 2009-12-29 Sony Computer Entertainment Inc. Man-machine interface using a deformable device
US10220302B2 (en) 2002-07-27 2019-03-05 Sony Interactive Entertainment Inc. Method and apparatus for tracking three-dimensional movements of an object using a depth sensing camera
US10406433B2 (en) 2002-07-27 2019-09-10 Sony Interactive Entertainment America Llc Method and system for applying gearing effects to visual tracking
US8976265B2 (en) 2002-07-27 2015-03-10 Sony Computer Entertainment Inc. Apparatus for image and sound capture in a game environment
US10099130B2 (en) 2002-07-27 2018-10-16 Sony Interactive Entertainment America Llc Method and system for applying gearing effects to visual tracking
US9381424B2 (en) 2002-07-27 2016-07-05 Sony Interactive Entertainment America Llc Scheme for translating movements of a hand-held controller into inputs for a system
US9174119B2 (en) 2002-07-27 2015-11-03 Sony Computer Entertainement America, LLC Controller for providing inputs to control execution of a program when inputs are combined
US9682319B2 (en) 2002-07-31 2017-06-20 Sony Interactive Entertainment Inc. Combiner method for altering game gearing
US9177387B2 (en) 2003-02-11 2015-11-03 Sony Computer Entertainment Inc. Method and apparatus for real time motion capture
US11010971B2 (en) 2003-05-29 2021-05-18 Sony Interactive Entertainment Inc. User-driven three-dimensional interactive gaming environment
WO2004107272A1 (fr) * 2003-05-29 2004-12-09 Sony Computer Entertainment Inc. Systeme et procede de fourniture d'un environnement interactif tridimensionnel en temps reel
US8233642B2 (en) 2003-08-27 2012-07-31 Sony Computer Entertainment Inc. Methods and apparatuses for capturing an audio signal based on a location of the signal
US8160269B2 (en) 2003-08-27 2012-04-17 Sony Computer Entertainment Inc. Methods and apparatuses for adjusting a listening area for capturing sounds
US8139793B2 (en) 2003-08-27 2012-03-20 Sony Computer Entertainment Inc. Methods and apparatus for capturing audio signals based on a visual image
US10099147B2 (en) 2004-08-19 2018-10-16 Sony Interactive Entertainment Inc. Using a portable device to interface with a video game rendered on a main display
US10279254B2 (en) 2005-10-26 2019-05-07 Sony Interactive Entertainment Inc. Controller having visually trackable object for interfacing with a gaming system
US9573056B2 (en) 2005-10-26 2017-02-21 Sony Interactive Entertainment Inc. Expandable control device via hardware attachment
WO2008139251A3 (fr) * 2006-04-14 2009-03-12 Rosenthal Patrick Levy Dispositif de caméra vidéo virtuelle avec suivi tridimensionnel et insertion d'objet virtuel
USRE48417E1 (en) 2006-09-28 2021-02-02 Sony Interactive Entertainment Inc. Object direction using video input combined with tilt angle information
EP1944700A1 (fr) * 2007-01-10 2008-07-16 Imagetech Co., Ltd. Procédé et système pour vidéo interactive en temps-réel
US8542907B2 (en) 2007-12-17 2013-09-24 Sony Computer Entertainment America Llc Dynamic three-dimensional object mapping for user-defined control device
US8840470B2 (en) 2008-02-27 2014-09-23 Sony Computer Entertainment America Llc Methods for capturing depth data of a scene and applying computer actions
US8279222B2 (en) 2008-03-14 2012-10-02 Seiko Epson Corporation Processing graphics data for a stereoscopic display
US8961313B2 (en) 2009-05-29 2015-02-24 Sony Computer Entertainment America Llc Multi-positional three-dimensional controller
WO2011029209A3 (fr) * 2009-09-10 2011-09-29 Liberovision Ag Procédé et appareil destinés à générer et à traiter des images à profondeur accrue
US9710554B2 (en) 2010-09-23 2017-07-18 Nokia Technologies Oy Methods, apparatuses and computer program products for grouping content in augmented reality
US9589362B2 (en) 2014-07-01 2017-03-07 Qualcomm Incorporated System and method of three-dimensional model generation
US9607388B2 (en) 2014-09-19 2017-03-28 Qualcomm Incorporated System and method of pose estimation
US10373366B2 (en) 2015-05-14 2019-08-06 Qualcomm Incorporated Three-dimensional model generation
US10304203B2 (en) 2015-05-14 2019-05-28 Qualcomm Incorporated Three-dimensional model generation
US9911242B2 (en) 2015-05-14 2018-03-06 Qualcomm Incorporated Three-dimensional model generation
US10341568B2 (en) 2016-10-10 2019-07-02 Qualcomm Incorporated User interface to assist three dimensional scanning of objects
CN110827380A (zh) * 2019-09-19 2020-02-21 北京铂石空间科技有限公司 图像的渲染方法、装置、电子设备及计算机可读介质
CN110827380B (zh) * 2019-09-19 2023-10-17 北京铂石空间科技有限公司 图像的渲染方法、装置、电子设备及计算机可读介质

Similar Documents

Publication Publication Date Title
WO1999026198A2 (fr) Systeme et procede pour fusionner des objets en une suite d'images sans connaitre a priori la scene representee dans ladite suite d'images
US9082224B2 (en) Systems and methods 2-D to 3-D conversion using depth access segiments to define an object
US6278460B1 (en) Creating a three-dimensional model from two-dimensional images
Debevec et al. Efficient view-dependent image-based rendering with projective texture-mapping
US5613048A (en) Three-dimensional image synthesis using view interpolation
US6975756B1 (en) Image-based photo hulls
US20080225045A1 (en) Systems and methods for 2-d to 3-d image conversion using mask to model, or model to mask, conversion
US20080225042A1 (en) Systems and methods for allowing a user to dynamically manipulate stereoscopic parameters
US20080228449A1 (en) Systems and methods for 2-d to 3-d conversion using depth access segments to define an object
US20080226160A1 (en) Systems and methods for filling light in frames during 2-d to 3-d image conversion
US20080226181A1 (en) Systems and methods for depth peeling using stereoscopic variables during the rendering of 2-d to 3-d images
US20080226128A1 (en) System and method for using feature tracking techniques for the generation of masks in the conversion of two-dimensional images to three-dimensional images
US20080226194A1 (en) Systems and methods for treating occlusions in 2-d to 3-d image conversion
US11810248B2 (en) Method for processing image data to provide for soft shadow effects using shadow depth information
Slabaugh et al. Image-based photo hulls
CN114972612A (zh) 一种基于三维简化模型的图像纹理生成方法及相关设备
Ong et al. Resolving occlusion in image sequence made easy
Kang et al. Tour into the video: image-based navigation scheme for video sequences of dynamic scenes
US12190444B2 (en) Image-based environment reconstruction with view-dependent colour
Chang et al. Constructing a multivalued representation for view synthesis
Chen et al. Image synthesis from a sparse set of views
Pollard et al. Automatically synthesising virtual viewpoints by trinocular image interpolation
Eisemann et al. Towards plenoptic Raumzeit reconstruction
Eisemann et al. 3-D Cinematography with Approximate or No Geometry
Aliaga Dynamic simplification using textures

Legal Events

Date Code Title Description
AK Designated states

Kind code of ref document: A2

Designated state(s): SG

AL Designated countries for regional patents

Kind code of ref document: A2

Designated state(s): AT BE CH CY DE DK ES FI FR GB GR IE IT LU MC NL PT SE

121 Ep: the epo has been informed by wipo that ep was designated in this application
DFPE Request for preliminary examination filed prior to expiration of 19th month from priority date (pct application filed before 20040101)
122 Ep: pct application non-entry in european phase