CN102568026B - A 3D augmented reality method for autostereoscopic display of multi-viewpoints - Google Patents
A 3D augmented reality method for autostereoscopic display of multi-viewpoints Download PDFInfo
- Publication number
- CN102568026B CN102568026B CN201110412061.5A CN201110412061A CN102568026B CN 102568026 B CN102568026 B CN 102568026B CN 201110412061 A CN201110412061 A CN 201110412061A CN 102568026 B CN102568026 B CN 102568026B
- Authority
- CN
- China
- Prior art keywords
- dimensional
- image
- actual situation
- virtual
- camera
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 31
- 230000003190 augmentative effect Effects 0.000 title description 15
- 238000005286 illumination Methods 0.000 claims abstract description 36
- 230000000007 visual effect Effects 0.000 claims description 27
- 238000004422 calculation algorithm Methods 0.000 claims description 19
- 238000004364 calculation method Methods 0.000 claims description 14
- 230000004927 fusion Effects 0.000 claims description 12
- 238000013507 mapping Methods 0.000 claims description 10
- 238000012545 processing Methods 0.000 claims description 7
- 239000000284 extract Substances 0.000 claims description 6
- 238000000605 extraction Methods 0.000 claims description 5
- 230000002194 synthesizing effect Effects 0.000 claims description 5
- 238000009792 diffusion process Methods 0.000 claims description 3
- 230000000694 effects Effects 0.000 abstract description 9
- 230000002708 enhancing effect Effects 0.000 abstract 1
- 238000005516 engineering process Methods 0.000 description 14
- 239000003550 marker Substances 0.000 description 7
- 230000006870 function Effects 0.000 description 6
- 238000001514 detection method Methods 0.000 description 4
- 239000000463 material Substances 0.000 description 4
- 238000010586 diagram Methods 0.000 description 3
- 238000005315 distribution function Methods 0.000 description 3
- 238000004519 manufacturing process Methods 0.000 description 3
- 238000004458 analytical method Methods 0.000 description 2
- 239000011159 matrix material Substances 0.000 description 2
- 238000009877 rendering Methods 0.000 description 2
- 238000011160 research Methods 0.000 description 2
- 238000012549 training Methods 0.000 description 2
- 230000003044 adaptive effect Effects 0.000 description 1
- 238000013459 approach Methods 0.000 description 1
- 230000002457 bidirectional effect Effects 0.000 description 1
- 230000005540 biological transmission Effects 0.000 description 1
- 238000006243 chemical reaction Methods 0.000 description 1
- 230000000295 complement effect Effects 0.000 description 1
- 238000007405 data analysis Methods 0.000 description 1
- 230000007812 deficiency Effects 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 230000018109 developmental process Effects 0.000 description 1
- 230000012447 hatching Effects 0.000 description 1
- 230000010354 integration Effects 0.000 description 1
- 230000005389 magnetism Effects 0.000 description 1
- 238000012856 packing Methods 0.000 description 1
- 238000004088 simulation Methods 0.000 description 1
- 238000010189 synthetic method Methods 0.000 description 1
Images
Landscapes
- Processing Or Creating Images (AREA)
Abstract
The invention discloses a three-dimensional enhancing realizing method for multi-viewpoint free stereo display, which comprises the following steps: 1) stereoscopically shooting a natural scene by using a binocular camera; 2) extracting and matching a characteristic point of an image of a main camera, generating a three-dimensional point cloud picture of the natural scene in real time, and calculating a camera parameter; 3) calculating a depth image corresponding to the image of the main camera, drawing a virtual viewpoint image and a depth image thereof, and performing hollow repairing; 4) utilizing three-dimensional making software to draw a three-dimensional virtual model and utilizing a false-true fusing module to realize the false-true fusing of the multi-viewpoint image; 5) suitably combining multiple paths of false-true fused images; and 6) providing multi-viewpoint stereo display by a 3D display device. According to the method provided by the invention, the binocular camera is used for stereoscopically shooting and the characteristic extracting and matching technique with better instantaneity is adopted, so that no mark is required in the natural scene; the false-true fusing module is used for realizing the illumination consistency and seamless fusing of the false-true scenes; and the multi-user multi-angle naked-eye multi-viewpoint stereo display effect is supplied by the 3D display device.
Description
Technical field
The present invention relates to three-dimensional enhanced reality method, relate in particular to a kind of three-dimensional enhanced reality method of multi-view auto stereo display.
Background technology
Augmented reality (Augmented Reality, AR), is also referred to as mixed reality, expansion reality.It is applied to real world by computer technology by virtual information, makes true environment and dummy object be superimposed upon in real time same picture or space.With traditional virtual reality (Virtual Reality, VR) effect of immersing completely that technology will reach is different, the object that augmented reality is devoted to computing machine to generate is added on real scenery, the information that has not only represented real world, and virtual information is shown simultaneously, two kinds of information complement each other, superpose, and have strengthened true environment output characteristics, thereby have more significantly advantage than VR technology.
As far back as the sixties in 20th century, U.S. professor Sutherland has invented first head mounted display (Head-mounted display, HMD) in the world, and user can see that line frame graph is superimposed upon on true environment.Because the equipment prices such as HMD are high, some do not rely on hardware device and the AR technology that directly virtual information projected in real space is arisen at the historic moment.For example U.S. University of North Carolina in 1998 develops Spatial Augmented Reality; 1999, Japan developed a set of C/C++ language library ARToolKit, used the relative position between computerized image technique computes video camera and badge card, and programmer can be covered virtual objects on badge card.
The implementation of augmented reality has following a few class: the one, GPS is combined with Sensor, by GPS, obtain precision latitude and height, by earth magnetism digital compass (Sensor), obtain direction and angle of inclination, finally according to these positional informations, obtain Overlapping display after relevant information.Relevant item has PlaceEngine, SekaiCamera etc.This kind of mode is adapted at applying on mobile phone, and current iPhone 3GS and Anroid have possessed hardware unit.The second way is by Marker identification, and the information realization that is about to Marker image is preserved, and by image recognition technology, searches identification Marker image in current image, the relevant information that then superposes, and the aforesaid project ARToolkit that increases income belongs to this type of.Yet the shortcoming of these class methods is to need to realize definition Marker image, and expanded function is restricted, and is applicable to game, advertisement and is specifying the occasion of synthesizing CG on Marker image.The third implementation is graphical analysis identification, and it identifies landscape, object and space by photographs is resolved, then the relevant information that superposes, projects such as PTAM, SREngine.The method does not need customizing messages, and equipment is simple, by analysis diagram, looks like to set up virtual coordinates, can very naturally synthesize real world images and CG, can be described as most important direction in current AR research.Yet these class methods still have a lot of problems urgently to be resolved hurrily, large such as needs deal with data amount, technical difficulty is large etc.
Occlusion detection during actual situation merges is a major issue in augmented reality, and correct hiding relation is the important prerequisite that realizes good syncretizing effect; And existing most of augmented reality system is just superimposed upon dummy object on real scene image simply, cause real scene image by dummy object, to be blocked all the time.Existing most of augmented reality system adopts single camera to obtain scene information, and this kind of method is easy and simple to handle, and equipment cost is low, but is difficult to obtain depth information thereby is difficult to solve the occlusion issue in actual situation fusion.2003, the Masayuki Kanbara of Japan proposed a kind of augmented reality system based on stereoscopic vision, and this system adopts binocular camera to obtain scene information, calculates the depth information of object region scene; By detecting the monumented point position calculation Conversion Matrix of Coordinate in scene, by comparing real scene depth value and dummy object value calculating actual situation fusion depth map in certain limit, solve occlusion detection problem.Yet this system has two weak points: the one, need marking image, and be not suitable for being applied to natural scene; The 2nd, display module needs HMD, can not realize bore hole 3-D display.
The display technique of augmented reality can be divided into following a few class: Helmet Mounted Display (HMD) shows, projection shows that (Projection display), hand-held display device (Hand Held Display, HHD) show and regular display (Monitor-based Display) shows.Along with updating of PC performance, the augmented reality that regular display shows is subject to applying more and more widely, and AR technology and application and development on mobile device also receive more concern.Early stage augmented reality system adopts HMD to realize 3-D display, but HMD is expensive, wears inconvenience; Regular display can only realize two dimension and show, can not show a candle to the effect of 3-D display in user's experience.In March, 2011, the people such as Tzuan-Ren Jeng of Taiwan industrial technology research institute delivered < < New 3D Image Technologies Developed in Taiwan > >, this article has been introduced a kind of method that ARToolKit of use merges three-dimensional model and real scene, and points out that three-dimensional enhanced reality can be used as the expanded function of following bore hole three-dimensional television.But the system of introducing in article adopts label to detect scene, can not expand to natural scene; Display packing adopts common computer screen to realize two dimension demonstration." a kind of method and system that realize three-dimensional enhanced reality " (patent No. is 200610101229) of Huawei Technologies incorporated company invention discloses a kind of implementation method and system of three-dimensional enhanced reality, the frame of video of obtaining two dimensional visible coding maker thing from true environment has been described in this invention, by the synthetic method of two dimensional visible coding maker thing frame of video in virtual pattern frame and true environment, but do not relate to three-dimensional display apparatus.2009, Shanghai forever spar Information Technology Co., Ltd discloses " the augmented reality method and system based on image characteristic point extraction with random tree classification " (patent No. is 200910048113.8), this invention is trained the camera inner parameter of the method calculation flag object image that obtains training data and proofreaies and correct by selection marker thing front elevation and to mark, based on training data distinguishing mark thing, the marker position matrix that utilization extracts is determined the position of model, draws dummy model.This invention still depends on extraction and the identification of mark, can only on two dimensional image, synthesize three-dimensional model, does not relate to multi-viewpoint three-dimensional and shows.The patent No. be 200810062854.7 disclosure of the invention " a kind of natural three-dimensional television system ", this system is a kind of obtains scene coloured image by multi-eye stereo camera, extract principal phase machine coloured image, depth image and auxiliary camera block information, after layered encoded, transmission and decoding by viewpoint reconstructed module, reconstruct principal phase machine viewpoint, auxiliary camera viewpoint and virtual view, the image showing through the synthetic display of Graphics Processing module.This invention only limits to shooting, content production and the stereo display of natural scene, does not relate to the fusion of dummy object and reality scene.
Summary of the invention
The object of the invention is to overcome camera in existing augmented reality system and follow the trail of the deficiency with display module, a kind of three-dimensional enhanced reality method of multi-view auto stereo display is provided.
The step of the three-dimensional enhanced reality method of multi-view auto stereo display is as follows:
1) binocular camera stereoscopic shooting natural scene, obtains an auxiliary camera image sequence in principal phase machine image sequence He Yi road, road and inputs computing machine;
2) unique point the matching characteristic point of each frame of camera tracing module extraction principal phase machine image sequence in computing machine, produce in real time the three-dimensional point cloud atlas of natural scene and calculate main camera parameter and auxiliary camera parameter;
3) many viewpoints are rebuild module and are calculated according to main camera image, auxiliary camera image, main camera parameter and auxiliary camera parameter the depth map that main camera image is corresponding, according to the virtual camera parameter of main camera image, the depth map calculating and appointment, with DIBR algorithm, calculate multichannel virtual visual point image and the depth map thereof that multi-view auto stereo display needs, for the cavity existing in virtual visual point image and occlusion area, in conjunction with auxiliary camera image sequence, adopt the consistent cavity of the auxiliary space-time of the degree of depth to repair algorithm and repair;
4) by d-making Software on Drawing three dimensional virtual models, actual situation Fusion Module carries out three-dimensional registration according to main camera parameter, three-dimensional point cloud atlas and three dimensional virtual models, main viewpoint depth map and virtual view depth map, realizing actual situation merges, illumination calculation unit calculates actual situation scene illumination, realize actual situation illumination consistance, seamless integrated unit smoothly spreads the otherness on actual situation border;
5) the main viewpoint actual situation fused images that Graphics Processing module forms main camera image, the virtual view actual situation fused images that a plurality of virtual visual point images form is carried out suitable synthesizing, the needs of observing to be applicable to many viewpoints;
6) three dimensional display provides actual situation fused images bore hole, that be applicable to many people multi-angle observation.
Described step 2) be:
1) camera is followed the trail of with unique point mapping and be divided into two threads;
2) by Stereo Matching Algorithm, do the initialization of three-dimensional point cloud, with FAST corner algorithm, extract the unique point of each frame of principal phase machine image sequence, according to the quantity of unique point and matching relationship, select key frame, unique point on key frame is mapped to three-dimensional point cloud atlas, according to the mapping relations between the unique point coordinate on the unique point coordinate in three-dimensional point cloud atlas and each principal phase machine picture frame, calculates current camera position.
Described step 4) be:
1) three-dimensional registration is determined the position of three dimensional virtual models in main visual point image and virtual visual point image according to the characteristic point data in main camera parameter, virtual camera parameter, depth map and three-dimensional point cloud, in conjunction with three dimensional virtual models data, plays up the dummy object in respective viewpoints;
2) illumination calculation unit adopts Cook-Torrance model to calculate actual situation scene illumination model, realizes actual situation illumination consistance;
3) seamless integrated unit adopts Poisson picture editting algorithm that the difference of dummy object edge and natural scene is done to level and smooth diffusion, actual situation is merged and have more the sense of reality.
The present invention adopts binocular camera stereoscopic shooting, adopt the good physical feature of real-time performance to extract and matching algorithm, need in the natural scene of taking, not mark, reduce the restriction that scene is taken, can produce in real time the three-dimensional point cloud atlas of natural scene and obtain camera parameter; Actual situation Fusion Module utilizes main viewpoint depth map and virtual view depth map processing mutual occlusion problem and realizes illumination consistance and the seamless fusion of actual situation scene, and the actual situation fused images of a plurality of viewpoints is provided; 3D stereoscopic display device can provide the bore hole multi-viewpoint three-dimensional display effect of many people multi-angle.
Accompanying drawing explanation
Fig. 1 is the three-dimensional enhanced reality method flow diagram of multi-view auto stereo display;
Fig. 2 is camera trace flow figure of the present invention;
Fig. 3 is three-dimensional feature point mapping process flow diagram of the present invention;
Fig. 4 is real scene three-dimensional point cloud illustrated example of the present invention;
Fig. 5 is that actual situation scene of the present invention merges example.
Embodiment
The step of the three-dimensional enhanced reality method of multi-view auto stereo display is as follows:
1) binocular camera stereoscopic shooting natural scene, obtains an auxiliary camera image sequence in principal phase machine image sequence He Yi road, road and inputs computing machine;
2) unique point the matching characteristic point of each frame of camera tracing module extraction principal phase machine image sequence in computing machine, produce in real time the three-dimensional point cloud atlas of natural scene and calculate main camera parameter and auxiliary camera parameter;
3) many viewpoints are rebuild module and are calculated according to main camera image, auxiliary camera image, main camera parameter and auxiliary camera parameter the depth map that main camera image is corresponding, according to the virtual camera parameter of main camera image, the depth map calculating and appointment, with DIBR algorithm, calculate multichannel virtual visual point image and the depth map thereof that multi-view auto stereo display needs, for the cavity existing in virtual visual point image and occlusion area, in conjunction with auxiliary camera image sequence, adopt the consistent cavity of the auxiliary space-time of the degree of depth to repair algorithm and repair;
4) by d-making Software on Drawing three dimensional virtual models, actual situation Fusion Module carries out three-dimensional registration according to main camera parameter, three-dimensional point cloud atlas and three dimensional virtual models, main viewpoint depth map and virtual view depth map, realizing actual situation merges, illumination calculation unit calculates actual situation scene illumination, realize actual situation illumination consistance, seamless integrated unit smoothly spreads the otherness on actual situation border;
5) the main viewpoint actual situation fused images that Graphics Processing module forms main camera image, the virtual view actual situation fused images that a plurality of virtual visual point images form is carried out suitable synthesizing, the needs of observing to be applicable to many viewpoints;
6) three dimensional display provides actual situation fused images bore hole, that be applicable to many people multi-angle observation.
Described step 2) be:
1) camera is followed the trail of with unique point mapping and be divided into two threads;
2) by Stereo Matching Algorithm, do the initialization of three-dimensional point cloud, with FAST corner algorithm, extract the unique point of each frame of principal phase machine image sequence, according to the quantity of unique point and matching relationship, select key frame, unique point on key frame is mapped to three-dimensional point cloud atlas, according to the mapping relations between the unique point coordinate on the unique point coordinate in three-dimensional point cloud atlas and each principal phase machine picture frame, calculates current camera position.
Described step 4) be:
1) three-dimensional registration is determined the position of three dimensional virtual models in main visual point image and virtual visual point image according to the characteristic point data in main camera parameter, virtual camera parameter, depth map and three-dimensional point cloud, in conjunction with three dimensional virtual models data, plays up the dummy object in respective viewpoints;
2) illumination calculation unit adopts Cook-Torrance model to calculate actual situation scene illumination model, realizes actual situation illumination consistance;
3) seamless integrated unit adopts Poisson picture editting algorithm that the difference of dummy object edge and natural scene is done to level and smooth diffusion, actual situation is merged and have more the sense of reality.
Described camera tracing module adopts the method for tracing based on feature point detection, processes frame by frame the image sequence that principal phase machine obtains, and estimates the main camera parameter of present frame and produces the three-dimensional feature point cloud chart of natural scene.Camera is followed the trail of and unique point mapping is divided into two threads, as shown in Figure 2 and Figure 3.Before tracing process starts, camera need to, through demarcating, be determined camera parameter.At the initial phase of tracing process, manually specify two picture frames for Stereo matching, produce initial three-dimensional point cloud atlas and set up world coordinate system.After initialization, pretreatment unit is according to motion model guestimate camera position.Feature point detection unit detects the unique point of present frame with FAST Corner algorithm, the unique point in three-dimensional point cloud atlas is mapped to present frame plane and detected unique point is done to mate with shining upon the unique point of coming, and according to matching relationship, upgrades camera position.This module is exported camera position parameter and the natural scene three-dimensional point cloud atlas of each frame.Three-dimensional point cloud exemplary plot as shown in Figure 4.
Described many viewpoints are rebuild module and are comprised Stereo matching unit, depth calculation unit, dummy model production unit, DIBR unit and the hole-filling unit ，Cong mono-auxiliary camera image sequence in principal phase machine image sequence ,Yi road, tunnel and principal phase machine calculation of parameter and export main visual point image sequence (principal phase machine image sequence) and depth map sequence, multichannel virtual visual point image sequence and corresponding depth map sequence thereof.Stereo matching unit extracts parallax information from principal phase machine image sequence and auxiliary camera image sequence, and depth calculation unit by using parallax information and camera parameter calculate the depth map that main camera image is corresponding.According to required virtual view position and the relation between main camera position, determine the camera model of virtual visual point image, DIBR unit by using principal phase machine coloured image, corresponding depth map and the camera model of virtual visual point image position render virtual visual point image and corresponding virtual view depth map.On hole-filling unit by using time orientation, obtainable scene information carries out the consistent cavity reparation algorithm of space-time with obtainable texture information on direction in space.On time orientation, before and after utilizing, in frame, visible scene information is repaired the hole region in current time virtual visual point image frame; On direction in space, utilize similar background texture information to repair hole region.Dummy model production unit adopts 3DS MAX Software on Drawing 3D model, has made the rear switching software with Wcvt2pov and the file of .3ds type has been converted to the C type file that meets OpenGk form, in order to Graphics Processing.
Described actual situation Fusion Module comprises three-dimensional registering unit, illumination calculation and seamless integrated unit.Three-dimensional registering unit is calculated the model transferring parameter of each key frame according to camera parameter, determine position and the visual angle of three dimensional virtual models in each visual point image, according to the hiding relation between main viewpoint depth map and virtual view depth map and three dimensional virtual models data analysis three dimensional virtual models and natural scene, and three dimensional virtual models is added on main visual point image and each virtual visual point image, form main viewpoint actual situation fused images and multichannel virtual view actual situation fused images.Illumination consistance refers to that dummy object and background image should reach consistent lighting effect, light and shade and hatching effect that actual situation is consistent.Illumination calculation unit adopts the illumination information of expressing and obtain real scene based on image irradiation technology, adopt Cook-Torrance illumination model to calculate actual situation scene illumination, and during dummy object material properties is considered to real-time lighting calculates, simulate the virtual objects lighting effect of common unlike material, the Realistic rendering arithmetic of the virtual objects of realizing common unlike material under complex illumination condition.Cook-Torrance model is divided into two aspects by illumination and considers: diffuse and specular light, expression formula is as follows:
I＝k
aI
pa+k
dI
pd(L*N)+k
sI
psD
kFG/(N*V) (1)
Wherein, k
ai
pareflection of ambient light component, k
areflection of ambient light coefficient, k
di
pd(L*N) be diffuse reflection component, k
si
paspecular components, k
sspecularity factor, I
psbe incident light mirror-reflection efficient intensity, L is incident light direction, and N is body surface point normal vector direction, D
kbe bit plane slope distribution function, generally with gauss of distribution function, represent, F is Fresnel function, and G is the geonetrical attenuation factor, thus the specular components decay that main consideration can not reflect away part light because micro-interplanar blocks mutually to be caused.Due to the illumination very complex in real scene, be difficult to the virtual light source simulation in computer graphics, the illumination information of brightest area and dark areas in complex environment actinogram real scene based on image irradiation technology, and replace virtual light source " to illuminate " virtual objects with this, virtual object physical efficiency is incorporated in real scene around well.Environment mappings technology can product surface complexity lighting effect, by incident illumination being illustrated in texture, realize complicated illumination model, conventionally can be by calculating the diffuse reflection component of bidirectional reflectance distribution function BRDF and diffuse map and the mirror-reflection pinup picture that specular components obtains original pinup picture.Illumination calculation unit calculates the illumination coefficient of natural scene image according to Cook Torrance illumination model, using OpenGL tinter to complete diffuse reflection calculates, by environment mappings technology, mirror-reflection is partly simulated the Realistic rendering arithmetic of the dummy object of realizing common unlike material under complex illumination condition.
Seamless integrated unit adopts Poisson to edit algorithm and utilizes gradient field to treat integration region to guide interpolation, image co-registration problem is summed up as to the minimization of object function problem (as shown in expression formula 2) of asking.Under the guiding of source images vector field, calculate the interpolating function of target image, source images and the borderline difference of target image are smoothly diffused in fused images piece, reach the object that merges border.
Wherein v represents the vector field of source images, and f represents scalar function to be asked, and Ω represents the closed set in target image.The solution of expression formula (2) is the solution of Poisson equation under Dirichlet boundary condition:
Δ f=dfvv boundary condition
(3)
Described seamless integrated unit is selectively actuatable unit.When dummy object to be that real scene content is relevant, need to incorporate while being smoothly transitted in real scene, select to carry out seamless integrated unit, the difference of dummy object edge and real scene is smoothly diffused in image, strengthen the sense of reality of actual situation fusion.
The many viewpoints of the main viewpoint actual situation fused images that described Graphics Processing module generates actual situation Fusion Module and multirouting are rebuild the virtual view actual situation fused images of module and the generation of actual situation Fusion Module and are carried out suitable synthesizing, the needs of observing to be applicable to many viewpoints.9 viewpoints of take are example, the resolution of supposing synthetic stereo image is H*V, 9 visual point images are sampled, the vertical resolution of each visual point image is sampled into H/h, horizontal resolution is sampled into H/h, wherein, and v*h=9, v gets the round values that approaches 3 most, finally according to the some distance of the LPI value of actual displayed and actual tilt angles and LCD display, obtains needed stereo-picture.Stereoscopic display device is a kind of stereoscopic display device that bore hole can be provided, be applicable to many people multi-angle observation.What this laboratory adopted is the adaptive display of a kind of 2D/3D, is a kind of based on pole mirror grating LCD freedom stereoscopic display device.
Claims (3)
1. a three-dimensional enhanced reality method for multi-view auto stereo display, is characterized in that its step is as follows:
1) binocular camera stereoscopic shooting natural scene, obtains an auxiliary camera image sequence in principal phase machine image sequence He Yi road, road and inputs computing machine;
2) unique point the matching characteristic point of each frame of camera tracing module extraction principal phase machine image sequence in computing machine, initial phase at tracing process, two picture frames of artificial appointment are for Stereo matching, produce initial three-dimensional point cloud atlas and set up world coordinate system, produce in real time the three-dimensional point cloud atlas of natural scene and calculate main camera parameter and auxiliary camera parameter;
3) many viewpoints are rebuild module and are calculated according to main camera image, auxiliary camera image, main camera parameter and auxiliary camera parameter the depth map that main camera image is corresponding, according to the virtual camera parameter of main camera image, the depth map calculating and appointment, with DIBR algorithm, calculate multichannel virtual visual point image and the depth map thereof that multi-view auto stereo display needs, for the cavity existing in virtual visual point image and occlusion area, in conjunction with auxiliary camera image sequence, adopt the consistent cavity of the auxiliary space-time of the degree of depth to repair algorithm and repair;
4) by d-making Software on Drawing three dimensional virtual models, actual situation Fusion Module carries out three-dimensional registration according to main camera parameter, three-dimensional point cloud atlas and three dimensional virtual models, main viewpoint depth map and virtual view depth map, realizing actual situation merges, illumination calculation unit calculates actual situation scene illumination, realize actual situation illumination consistance, seamless integrated unit smoothly spreads the otherness on actual situation border;
5) the main viewpoint actual situation fused images that Graphics Processing module forms main camera image, the virtual view actual situation fused images that a plurality of virtual visual point images form is carried out suitable synthesizing, the needs of observing to be applicable to many viewpoints;
6) three dimensional display provides actual situation fused images bore hole, that be applicable to many people multi-angle observation.
2. the three-dimensional enhanced reality method of multi-view auto stereo display according to claim 1, is characterized in that described step 2) be:
1) camera is followed the trail of with unique point mapping and be divided into two threads;
2) by Stereo Matching Algorithm, do the initialization of three-dimensional point cloud, with FAST corner algorithm, extract the unique point of each frame of principal phase machine image sequence, according to the quantity of unique point and matching relationship, select key frame, unique point on key frame is mapped to three-dimensional point cloud atlas, according to the mapping relations between the unique point coordinate on the unique point coordinate in three-dimensional point cloud atlas and each principal phase machine picture frame, calculates current camera position.
3. the three-dimensional enhanced reality method of multi-view auto stereo display according to claim 1, is characterized in that described step 4) is:
1) three-dimensional registration is determined the position of three dimensional virtual models in main visual point image and virtual visual point image according to the characteristic point data in main camera parameter, virtual camera parameter, depth map and three-dimensional point cloud, in conjunction with three dimensional virtual models data, plays up the dummy object in respective viewpoints;
2) illumination calculation unit adopts Cook-Torrance model to calculate actual situation scene illumination model, realizes actual situation illumination consistance;
3) seamless integrated unit adopts Poisson picture editting algorithm that the difference of dummy object edge and natural scene is done to level and smooth diffusion, actual situation is merged and have more the sense of reality.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201110412061.5A CN102568026B (en) | 2011-12-12 | 2011-12-12 | A 3D augmented reality method for autostereoscopic display of multi-viewpoints |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201110412061.5A CN102568026B (en) | 2011-12-12 | 2011-12-12 | A 3D augmented reality method for autostereoscopic display of multi-viewpoints |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN102568026A CN102568026A (en) | 2012-07-11 |
| CN102568026B true CN102568026B (en) | 2014-01-29 |
Family
ID=46413368
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201110412061.5A Active CN102568026B (en) | 2011-12-12 | 2011-12-12 | A 3D augmented reality method for autostereoscopic display of multi-viewpoints |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN102568026B (en) |
Families Citing this family (86)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102867057B (en) * | 2012-09-17 | 2015-04-29 | 北京航空航天大学 | Virtual wizard establishment method based on visual positioning |
| US20140132595A1 (en) * | 2012-11-14 | 2014-05-15 | Microsoft Corporation | In-scene real-time design of living spaces |
| CN103177468A (en) * | 2013-03-29 | 2013-06-26 | 渤海大学 | Three-dimensional motion object augmented reality registration method based on no marks |
| CN104183011A (en) * | 2013-05-27 | 2014-12-03 | 万克林 | Three-dimensional interactive virtual reality (3D IVR) restoring system |
| CN103489214A (en) * | 2013-09-10 | 2014-01-01 | 北京邮电大学 | Virtual reality occlusion handling method, based on virtual model pretreatment, in augmented reality system |
| CN106304842A (en) * | 2013-10-03 | 2017-01-04 | 舒朗科技公司 | For location and the augmented reality system and method for map building |
| US9147113B2 (en) * | 2013-10-07 | 2015-09-29 | Hong Kong Applied Science and Technology Research Institute Company Limited | Deformable surface tracking in augmented reality applications |
| CN103796064B (en) * | 2014-02-21 | 2017-05-31 | 深圳创维－Rgb电子有限公司 | Player method, player and display device |
| WO2015175730A1 (en) | 2014-05-13 | 2015-11-19 | Nant Vision, Inc. | Augmented reality content rendering via albedo models, systems and methods |
| CN105635707A (en) * | 2014-11-06 | 2016-06-01 | 福州瑞芯微电子股份有限公司 | Image generation method and device |
| CN104504671B (en) * | 2014-12-12 | 2017-04-19 | 浙江大学 | Method for generating virtual-real fusion image for stereo display |
| US9412034B1 (en) * | 2015-01-29 | 2016-08-09 | Qualcomm Incorporated | Occlusion handling for computer vision |
| CN104766270B (en) * | 2015-03-20 | 2017-10-03 | 北京理工大学 | One kind is based on fish-eye actual situation illumination fusion method |
| CN106296801B (en) * | 2015-06-12 | 2019-11-26 | 联想(北京)有限公司 | A kind of method that establishing object three-dimensional image model and electronic equipment |
| CN105005970B (en) * | 2015-06-26 | 2018-02-16 | 广东欧珀移动通信有限公司 | The implementation method and device of a kind of augmented reality |
| EP3128413A1 (en) | 2015-08-04 | 2017-02-08 | Nokia Technologies Oy | Sharing mediated reality content |
| CN108369639B (en) * | 2015-12-11 | 2022-06-21 | 虞晶怡 | Image-based image rendering method and system using multiple cameras and depth camera array |
| CN105701821B (en) * | 2016-01-14 | 2018-07-24 | 福州华鹰重工机械有限公司 | Stereo-picture surface detects matching process and device |
| EP3422955B1 (en) * | 2016-02-29 | 2023-10-18 | Packsize International, LLC | System and method for assisted 3d scanning |
| CN105869160B (en) * | 2016-03-28 | 2019-11-26 | 武汉理工大学 | The method and system of three-dimensional modeling and holographic display are realized using Kinect |
| US10134198B2 (en) | 2016-04-19 | 2018-11-20 | Adobe Systems Incorporated | Image compensation for an occluding direct-view augmented reality system |
| CN107358609B (en) * | 2016-04-29 | 2020-08-04 | 成都理想境界科技有限公司 | Image superposition method and device for augmented reality |
| CN106131536A (en) * | 2016-08-15 | 2016-11-16 | 万象三维视觉科技（北京）有限公司 | A kind of bore hole 3D augmented reality interactive exhibition system and methods of exhibiting thereof |
| CN106355647A (en) * | 2016-08-25 | 2017-01-25 | 北京暴风魔镜科技有限公司 | Augmented reality system and method |
| CN106447643A (en) * | 2016-09-19 | 2017-02-22 | 西安你的主意电子商务有限公司 | AR technology based interactive image processing method |
| TWI651657B (en) * | 2016-10-21 | 2019-02-21 | 財團法人資訊工業策進會 | Augmented reality system and method |
| CN106447705B (en) * | 2016-11-24 | 2019-07-16 | 华南理工大学 | Multi-eye stereo vision system and method applied to indoor scene virtual reality live broadcast |
| CN106815555B (en) * | 2016-12-21 | 2020-02-14 | 深圳增强现实技术有限公司 | Augmented reality method and system for distributed scene target recognition |
| EP3340103A1 (en) * | 2016-12-21 | 2018-06-27 | Axis AB | Method for identifying events in a motion video |
| CN107223270B (en) | 2016-12-28 | 2021-09-03 | 达闼机器人有限公司 | Display data processing method and device |
| US20180192033A1 (en) * | 2016-12-30 | 2018-07-05 | Google Inc. | Multi-view scene flow stitching |
| CN108421252B (en) * | 2017-02-14 | 2023-12-29 | 杭州融梦智能科技有限公司 | Game realization method based on AR equipment and AR equipment |
| CN107016704A (en) * | 2017-03-09 | 2017-08-04 | 杭州电子科技大学 | A kind of virtual reality implementation method based on augmented reality |
| CN106910251A (en) * | 2017-03-22 | 2017-06-30 | 朱海涛 | Model emulation method based on AR and mobile terminal |
| US10706612B2 (en) * | 2017-04-01 | 2020-07-07 | Intel Corporation | Tile-based immediate mode rendering with early hierarchical-z |
| CN107147894B (en) * | 2017-04-10 | 2019-07-30 | 四川大学 | A kind of virtual visual point image generating method in Auto-stereo display |
| CN108876852B (en) * | 2017-05-09 | 2021-06-22 | 中国科学院沈阳自动化研究所 | Online real-time object identification and positioning method based on 3D vision |
| CN107134194A (en) * | 2017-05-18 | 2017-09-05 | 河北中科恒运软件科技股份有限公司 | Immersion vehicle simulator |
| CN107330965B (en) * | 2017-06-12 | 2020-08-04 | 长春理工大学 | Method for realizing hard shadow anti-aliasing by using local conservative rasterization method |
| JP6948175B2 (en) * | 2017-07-06 | 2021-10-13 | キヤノン株式会社 | Image processing device and its control method |
| CN107330964B (en) * | 2017-07-24 | 2020-11-13 | 广东工业大学 | Display method and system of complex three-dimensional object |
| CN107590859A (en) * | 2017-09-01 | 2018-01-16 | 广州励丰文化科技股份有限公司 | A kind of mixed reality picture processing method and service equipment |
| CN107608077B (en) * | 2017-09-08 | 2020-01-03 | 长春理工大学 | Multi-light-source position estimation method |
| CN107767462B (en) * | 2017-10-16 | 2023-08-25 | 北京视据科技有限公司 | Non-wearable augmented reality holographic display method and display system |
| CN108305326A (en) * | 2018-01-22 | 2018-07-20 | 中国人民解放军陆军航空兵学院 | A method of mixing virtual reality |
| CN108320334B (en) * | 2018-01-30 | 2021-08-17 | 公安部物证鉴定中心 | A method for establishing a 3D scene roaming system based on point cloud |
| CN108632538B (en) * | 2018-05-22 | 2020-07-03 | 长沙全度影像科技有限公司 | CG animation and camera array combined bullet time shooting system and method |
| CN110536125A (en) * | 2018-05-25 | 2019-12-03 | 光宝电子(广州)有限公司 | Image processing system and image treatment method |
| KR102530278B1 (en) * | 2018-10-18 | 2023-05-10 | 삼성디스플레이 주식회사 | Electronic device having display module and image display method |
| CN111080704B (en) * | 2018-10-22 | 2023-09-15 | 浙江宇视科技有限公司 | Video augmented reality methods and devices |
| CN109544671B (en) * | 2018-11-12 | 2022-07-19 | 浙江大学 | Projection mapping method of video in three-dimensional scene based on screen space |
| WO2020113423A1 (en) * | 2018-12-04 | 2020-06-11 | 深圳市大疆创新科技有限公司 | Target scene three-dimensional reconstruction method and system, and unmanned aerial vehicle |
| CN109597486B (en) * | 2018-12-05 | 2021-01-15 | 中国科学院长春光学精密机械与物理研究所 | Intelligent home decoration experience equipment and method |
| CN109474816B (en) * | 2018-12-28 | 2024-04-05 | 上海北冕信息科技有限公司 | Virtual-real fusion device for augmented reality and virtual-real fusion method, equipment and medium thereof |
| CN109886121B (en) * | 2019-01-23 | 2021-04-06 | 浙江大学 | Human face key point positioning method for shielding robustness |
| CN109901713B (en) * | 2019-02-25 | 2020-07-17 | 山东大学 | Multi-person cooperative assembly system and method |
| CN109769109A (en) * | 2019-03-05 | 2019-05-17 | 东北大学 | Method and system for synthesizing and rendering 3D objects based on virtual viewpoints |
| CN109883414B (en) * | 2019-03-20 | 2021-08-27 | 百度在线网络技术（北京）有限公司 | Vehicle navigation method and device, electronic equipment and storage medium |
| CN110139028B (en) * | 2019-03-25 | 2020-07-07 | 华为技术有限公司 | Image processing method and head-mounted display device |
| CN110246146B (en) * | 2019-04-29 | 2021-07-30 | 北京邮电大学 | Method and device for generating full parallax light field content based on multiple depth image rendering |
| CN110288657B (en) * | 2019-05-23 | 2021-05-04 | 华中师范大学 | A Kinect-based Augmented Reality 3D Registration Method |
| CN110349246B (en) * | 2019-07-17 | 2023-03-14 | 广西师范大学 | Method for reducing reconstruction distortion degree of viewpoint in light field rendering |
| CN110503710A (en) * | 2019-09-04 | 2019-11-26 | 北京国润视光科技有限公司 | A kind of complex scene recovery training method based on real enhancing technology |
| TWI709062B (en) * | 2019-09-20 | 2020-11-01 | 財團法人資訊工業策進會 | Virtuality reality overlapping method and system |
| CN112738534B (en) * | 2019-10-28 | 2023-06-23 | 阿里巴巴集团控股有限公司 | Data processing method and system, server and storage medium |
| CN111127629A (en) * | 2019-11-28 | 2020-05-08 | 武汉烽火信息集成技术有限公司 | Dynamic three-dimensional visualization system and method |
| CN111199573B (en) * | 2019-12-30 | 2023-07-07 | 成都索贝数码科技股份有限公司 | A virtual-real interreflection method, device, medium and equipment based on augmented reality |
| CN111297501B (en) * | 2020-02-17 | 2021-07-30 | 北京牡丹电子集团有限责任公司 | Augmented reality navigation method and system for oral implantation operation |
| CN111679743A (en) * | 2020-08-11 | 2020-09-18 | 南京瑞巨数码科技有限公司 | Method for realizing posture interaction naked eye three-dimensional mixed virtual reality system |
| CN112330815B (en) * | 2020-11-26 | 2024-05-14 | 北京百度网讯科技有限公司 | Three-dimensional point cloud data processing method, device and equipment based on obstacle fusion |
| CN112633324A (en) * | 2020-11-27 | 2021-04-09 | 中山大学 | System, method and medium for matching stereoscopic vision around the eyes based on neural network |
| CN112618026B (en) * | 2020-12-15 | 2022-05-31 | 清华大学 | Remote surgical data fusion interactive display system and method |
| CN112734914A (en) * | 2021-01-14 | 2021-04-30 | 温州大学 | Image stereo reconstruction method and device for augmented reality vision |
| CN112926676B (en) * | 2021-03-24 | 2022-12-13 | 成都新潮传媒集团有限公司 | False target identification method and device and computer equipment |
| CN113066191B (en) * | 2021-04-10 | 2022-12-02 | 中国人民解放军陆军装甲兵学院 | Virtual-real fusion method and system for holographic stereogram parallax image based on depth map |
| CN113099204B (en) * | 2021-04-13 | 2022-12-13 | 北京航空航天大学青岛研究院 | Remote live-action augmented reality method based on VR head-mounted display equipment |
| CN113674574B (en) * | 2021-07-05 | 2023-10-13 | 河南泊云电子科技股份有限公司 | Augmented reality semi-physical complex electromechanical equipment training system |
| CN116266340B (en) * | 2021-12-16 | 2025-08-22 | 广联达科技股份有限公司 | A method, device, computer equipment and storage medium for enhancing display of graphic elements |
| CN114887321B (en) * | 2022-04-26 | 2024-12-17 | 广州宸境科技有限公司 | Multi-user AR interaction method, device, equipment and storage medium |
| CN114575205A (en) * | 2022-04-28 | 2022-06-03 | 中铁第四勘察设计院集团有限公司 | Water jet steel rail profile intelligent polishing system based on image data processing |
| CN114818992B (en) * | 2022-06-23 | 2022-09-23 | 成都索贝数码科技股份有限公司 | Image data analysis method, scene estimation method, 3D fusion method |
| CN115063562B (en) * | 2022-06-23 | 2024-11-12 | 温州大学大数据与信息技术研究院 | A virtual-real fusion augmented reality presentation method based on multi-view 3D reconstruction |
| CN116828165A (en) * | 2023-05-29 | 2023-09-29 | 京东方科技集团股份有限公司 | Image processing methods and devices, storage media, electronic equipment |
| CN116958332B (en) * | 2023-09-20 | 2023-12-22 | 南京竹影数字科技有限公司 | Method and system for mapping 3D model in real time of paper drawing based on image recognition |
| CN118138741B (en) * | 2024-05-08 | 2024-07-09 | 四川物通科技有限公司 | Naked eye 3D data communication method |
| CN119888093B (en) * | 2025-03-26 | 2025-06-24 | 杭州电子科技大学 | Binocular depth estimation-based three-dimensional road scene generation method |
Family Cites Families (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| KR100450823B1 (en) * | 2001-11-27 | 2004-10-01 | 삼성전자주식회사 | Node structure for representing 3-dimensional objects using depth image |
| CN101277454A (en) * | 2008-04-28 | 2008-10-01 | 清华大学 | A real-time stereoscopic video generation method based on binocular cameras |
| CN101610423B (en) * | 2009-07-13 | 2010-09-08 | 清华大学 | A method and device for rendering an image |
-
2011
- 2011-12-12 CN CN201110412061.5A patent/CN102568026B/en active Active
Also Published As
| Publication number | Publication date |
|---|---|
| CN102568026A (en) | 2012-07-11 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN102568026B (en) | A 3D augmented reality method for autostereoscopic display of multi-viewpoints | |
| US11699263B2 (en) | Apparatus, method and computer program for rendering a visual scene | |
| CN110517355B (en) | Ambient composition for illuminating mixed reality objects | |
| CN107341853B (en) | Virtual-real fusion method and system for super-large virtual scene and dynamic screen shooting | |
| Sugano et al. | The effects of shadow representation of virtual objects in augmented reality | |
| US9288476B2 (en) | System and method for real-time depth modification of stereo images of a virtual reality environment | |
| CN103337095B (en) | The tridimensional virtual display methods of the three-dimensional geographical entity of a kind of real space | |
| CN105704479B (en) | The method and system and display equipment of the measurement human eye interpupillary distance of 3D display system | |
| Wither et al. | Pictorial depth cues for outdoor augmented reality | |
| JP2004537082A (en) | Real-time virtual viewpoint in virtual reality environment | |
| US20130057540A1 (en) | Methods and apparatus for digital stereo drawing | |
| CN103077552B (en) | A kind of three-dimensional display method based on multi-view point video | |
| CN102306088A (en) | Solid projection false or true registration device and method | |
| Verhoeven | Computer graphics meets image fusion: the power of texture baking to simultaneously visualise 3D surface features and colour | |
| CN113989434B (en) | A method and device for three-dimensional reconstruction of human body | |
| CN108881886A (en) | A method of it is realized based on camera Matrix Technology and carries out the lossless interactive application of big data in display end | |
| Park et al. | " DreamHouse" NUI-based Photo-realistic AR Authoring System for Interior Design | |
| Wang et al. | Bidirectional shadow rendering for interactive mixed 360° videos | |
| JP2022093262A (en) | Image processing device, control method and program of image processing device | |
| Li et al. | Augmented reality and virtual reality | |
| Tao | A VR/AR-based display system for arts and crafts museum | |
| Noh et al. | A review of shadow techniques in augmented reality | |
| Fischer et al. | Stylized depiction in mixed reality | |
| CN108898653A (en) | A kind of method for designing three-dimensional cartoon toys based on picture | |
| CN114170409A (en) | A method for automatically judging and displaying annotations of three-dimensional models |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| C14 | Grant of patent or utility model | ||
| GR01 | Patent grant | ||
| C41 | Transfer of patent application or patent right or utility model | ||
| TR01 | Transfer of patent right |
Effective date of registration: 20160616 Address after: 518000 new energy building, Nanhai Road, Shenzhen, Guangdong, Nanshan District A838 Patentee after: Meng Qi media (Shenzhen) Co. Ltd. Address before: 310027 Hangzhou, Zhejiang Province, Xihu District, Zhejiang Road, No. 38, No. Patentee before: Zhejiang University |
|
| C41 | Transfer of patent application or patent right or utility model | ||
| TR01 | Transfer of patent right |
Effective date of registration: 20160920 Address after: 518000, 101, 2, Fengyun technology building, Fifth Industrial Zone, North Ring Road, Shenzhen, Guangdong, Nanshan District Patentee after: World wide technology (Shenzhen) Limited Address before: 518000 new energy building, Nanhai Road, Shenzhen, Guangdong, Nanshan District A838 Patentee before: Meng Qi media (Shenzhen) Co. Ltd. |
|
| EE01 | Entry into force of recordation of patent licensing contract |
Application publication date: 20120711 Assignee: WANWEI YUNSHI (SHANGHAI) DIGITAL TECHNOLOGY CO., LTD. Assignor: World wide technology (Shenzhen) Limited Contract record no.: 2018440020049 Denomination of invention: Three-dimensional enhancing realizing method for multi-viewpoint free stereo display Granted publication date: 20140129 License type: Exclusive License Record date: 20180428 |
|
| EE01 | Entry into force of recordation of patent licensing contract | ||
| TR01 | Transfer of patent right |
Effective date of registration: 20180903 Address after: New 101 building, the wind and cloud mansion of West Road 5 unit B of 518000 Shenzhen City, Guangdong Province Nanshan District Xi Li neighbourhood committees Patentee after: Wan D display technology (Shenzhen) Co., Ltd. Address before: The 2 of 518000 FengYun Science building, No. 5 Industry Zone, Bei Huan Road, NanShan District, Shenzhen City, GuangDong Prov mansion 101 Patentee before: World wide technology (Shenzhen) Limited |
|
| TR01 | Transfer of patent right |