JP2019521387A - Hybrid Photonic VR / AR System - Google Patents

Abstract

A VR / AR system, method, and architecture for simultaneously receiving and processing real world image composition signals while creating composite world image composition signals, and then further processing these signals. Includes augmentor to interleave / enhance. In some implementations, real-world signals (passed with the possibility of being processed by an augmentor) are visualized (to the visible spectrum) to create a set of display image precursors intended for HVS. Conversion), amplitude / bandwidth processing, and continuous processing including output shaping, converted to infrared (eg, using a false colormap) and interleaved with a synthetic world signal (created in infrared) . [Selection] Figure 5

Description

(Cross-reference to related applications)
This application is related to U.S. Patent Application Nos. 15 / 457,967, 15 / 457,980, 15 / 457,991, and 15 / 458,009, all filed March 13, 2017. U.S. Patent Application Nos. 62 / 308,825, 62 / 308,361, 62 / 308,585, and 62/308 claiming the benefit of U.S. patent application Ser. No. 687, and the present application is related to US Patent Application Nos. 12 / 371,461, 62 / 181,143, and 62 / 234,942, the contents of which are incorporated herein by reference. It is expressly incorporated herein by reference in its entirety for all purposes.

  The present invention generally relates to data processing devices and networks that generate, transmit, switch, assign, store and display video and digital images, as well as their data, and non-arrays such as sensing arrays and spatial light modulators. With regard to video and non-pixel data processing, and the application and use of data for these, and in particular (but not exclusively) digital image displays (flat screen, flexible screen, 2D or 3D, or any of the projected images) And non-display data processing with arrays of elements, and the spatial form of organization and arrangement of these processes (including flat-screen TVs and compact devices such as consumer mobile devices), and pixel signals or data signals or these Aggregation of The image capture collection, transmission, assignment, division, organization, storage, delivery, to a data network to provide a display and projection.

  The subject matter discussed in the background section should not be assumed to be prior art only as a result of being pointed out in the background section. Similarly, the issues pointed out in the background section or associated with the subject matter of the background section should not be assumed to have been previously recognized in the prior art. The subject matter in the background section merely represents a different approach that may itself also be the invention.

  The field of the invention is not one, but rather a combination of two related fields, augmented reality and virtual reality, and addresses an integrated mobile device solution that solves the key problems and limitations of the prior art in both fields. And provide this. A brief review of the background of these related areas will clarify the issues and limitations to be solved and will prepare for the proposed solution of the present disclosure.

  The definitions of two standard dictionaries of these terms (Source: Dictionary.com) are as follows.

  Virtual Reality: "Realistic simulation of an environment including 3D graphics by a computer system using interactive software and hardware. Abbreviation: VR"

  Augmented Reality: "an image or environment expanded to be viewed on a screen or other display by overlaying computer generated images, sounds or other data in a real world environment. And:" such an expanded environment System or technology used to create a "abbreviation: AR"

  From the above definition (although it is non-technical), the most important difference for the person skilled in the relevant field is the completely and immersive type in which the simulated elements are completely screened even in real partial dilation. It is clear whether it is a simulation or whether the simulated element is superimposed on a clear and unobstructed real view.

  The Wikipedia entry for this topic provides a slightly more technical definition that is considered to be a good representation of the field, given the depth and extent of its contribution to the compilation of the page.

  Virtual reality (VR) (sometimes referred to as immersive multimedia) is a computer-simulated environment that can simulate physical presence in the real world or a place in an imaginary world. Virtual reality can reproduce perceptual experiences such as virtual taste, vision, smell, hearing, and touch.

  Augmented reality (AR) is a live, direct or indirect view of a physical real-world environment whose elements are augmented (or supplemented) by computer-generated perceptual inputs such as voice, video, graphics or GPS data.

  Being specific to these definitions but simply implicit is a very important attribute of mobile perspectives. A more general class of computer simulation (with or without "real-time" real "direct" imaging (whether local or remote), fusion, synthesis, or integration), virtual or extended It is the simulated or hybrid (enhanced or "mixed") real "simultaneous" images that distinguish reality, and as the viewer moves in the real world, the viewer's viewpoint is also with the viewer It is the point of moving.

  The present disclosure is intended to distinguish this more accurate definition from the immersive displayed and experienced simulated world static navigation (simulator) and mobile world of the simulated world (virtual reality). It proposes that it is necessary. And subcategories of the simulator allow partially "virtually realistic" navigation of the simulated world, stationary users wear immersive HMDs (head mounted displays) and haptic interfaces (eg motion tracking gloves) It will be an "Individual Simulator" or at most a "Partial Virtual Reality".

  On the other hand, the CAVE system will be regarded as generally suitable as a restricted virtual reality system. This is because navigation beyond the size of CAVE will only be possible with movable floors, and once CAVE's own limit has been reached, another form of "partial virtual reality" follows. It becomes ".

  Note the difference between the "mobile" and "mobile" viewpoints. A computer simulation such as a video game is a simulated world or unless the seeker in the simulated world is personally moving or otherwise directing the movement of another person or robot. "Reality" and what can be said there (This achievement among the major achievements of computer graphics in the past 40 years is simply to "build up" a simulated world that is searchable in software ), The simulated world is "navigable".

  Because simulation is either virtual or hybrid (author's favorite term) real, so critical features are mapping of the simulation to real space (whether completely synthetic or hybrid) It is that there is. Such real space may be just as simple as mapping and calibrating at a ratio to the simulated world, as basic as a room in a laboratory or sound stage.

  This distinction is not evaluative, because it does not map to real or non-mobile real reality topography (whether natural, artificial or hybrid or not) in real time natural interface ( Partial VR providing head tracking, tactile, auditory etc) is not of fundamentally lower value than partial VR systems simulating physical interaction and providing perceptual immersion . However, without the foot disease treatment feedback system, or more universally the whole body movement range feedback system, and / or the whole body movement on the terrain (as the rest) simulated (as the feeling of the user) ( The VR system is "partial" by definition, without a dynamically deformable mechanical interface or interaction surface that supports standing, sitting or reclining.

  However, in the absence of such an ideal full-body physical interface / feedback system, restricting the VR to a "full" mobile version means that the topography of the VR world can be found in the real world Or built from scratch). Such limitations will generally severely limit the scope and power of the virtual reality experience.

  However, as will become apparent in the following disclosure, this distinction brings about changes. Because this distinction sets out how existing VR and AR systems differ, and sets a "clear boundary reference" for these limitations, as well as providing a background for informing the teachings of this disclosure. is there.

  Although missing but very important simulation features and requirements have proved to be a complete "virtual reality", the next step is what does the "mobile perspective" realized mean To identify the implicit problem of The answer is to provide a view of the simulation that is mobile, which is two components that are themselves realized by a combination of hardware and software, ie an animation display means (by which the simulation is viewed Requires motion tracking means (capable of tracking the movement of the device including the display in 3-axis motion), which means that the position of the 3D viewing device at least three tracking points at least If mapped and measured, and the third position of the third axis is inferred, it means to measure at two tracking points), and the reference three-axis frame (arbitrary mapped to the real space) A plane in which two axes are the ground for the practical purpose of mechanically performing navigation of space in relation to the 3D coordinate system of Formed, Z-axis is the third axis, perpendicular to the its ground.

  Solutions to practically achieve this positional orientation accurately and frequently as a function of time require a combination of sensors and software, and advances in these solutions include VR and AR hardware / software It represents the main trajectory in the development of the field of both mobile viewing devices and systems.

  These are relatively new areas in terms of time frame between early experiments and current practical technologies and products, and it is sufficient to note the origin and the latest technologies in both categories of mobile visual simulation systems The only exception is that it is important to the development of the present disclosure or it serves to better explain either the current problem in the field or the distinction of the solution of the present disclosure from the prior art. It is a particular innovation in the prior art that relates to differences or similarities.

  The period from 1968 to the late 90's is the period of many innovations in relevant simulation and simulator, VR and AR fields, during this period many of the important issues in achieving partial VR and AR But found the first or partial solution.

  The original and influential experimental and experimental head-mounted display systems of Ivan Sutherland and his assistant Bob Sprouell since 1968 are generally regarded as marking the origin of these related areas, In earlier work, essentially conceptual development precedes this by the first experimental implementation of any form of AR / VR that achieves immersiveness and navigation.

  The birth of the static simulator system dates back to the addition of computer-generated imaging to flight simulators, which is generally recognized to have begun in the mid to late 1960's. This was limited to the use of a CRT, which displays a fully focused image at a distance from the user to the CRT, but by 1972 the Singer-Link Company was able to First introduced a collimated projection system that projects the field of view to about 25-35 degrees for each unit (100 degrees for 3 units used in a single-seat simulator).

  This benchmark was first improved in 1982 with the wide-angle infinity display system that introduced a wide field of view system, and was refined by the Rediffusion Company, which used 150 degrees by using multiple projectors and large curved collimated screens. The FOV, and finally the FOV of 240 degrees was realized. At this stage, the static simulator is important for realistic immersiveness in virtual reality by using the HMD to isolate viewers and eliminate distracting visual cues from the surroundings It may be explained that the degree was finally achieved.

  However, as the Singer-Link Company introduced screen collimation systems for simulators as a stepping stone to VR-type experiences, the first very limited commercial helmet-mounted displays were being developed for military use for the first time, This integrated the reticle based electronic target system with the helmet's motion tracking itself. These initial developments were generally recognized as being achieved in a rudimentary form by the South African Air Force in the 1970s (following, from that point onwards to the mid-'70s by the Israeli Air Force), a rudimentary AR or It may be said that the initiation of an intermediary / hybrid reality system.

  These early, graphically minimalistic, yet inventive and influential helmet mount systems combine positionally adjusted target information superimposed on a reticle with user-actuated motion tracking targeting. There is an invention by Steve Mann of the first generation "EyeTap", the first "intermediate reality" mobile view-through system that implemented restrictive compounding and then superimposed graphics on the glasses.

  Later versions by Mann used a beam splitter / combiner light fusion system based on light fusion reality and processed images. After this study, there is a study by Chunyu Gao and Augmented Vision Inc, which basically proposes a dual-Mann system that optically combines the processed real image and the generated image, where the Mann system is Both electronically processed real images and generated images were achieved. While Mann's system preserves the real view through the images, all of Gao's system processes view-through images, eliminating direct view-through images even as an option. (U.S. Patent Application No. 20140177023 to Chunyu Gao, filed April 13, 2013). The "optical path folding optics" structure and method specified by Gao's system can also be found in other optical HMD systems.

  By 1985, Jaron Lanier and VPL Reseearch were formed to develop HMD and “Data Globe”, and by the 1980s, some of the most significant developments among the very active development areas Three major developments by Simulation, VR and AR by Mann, Lanier and Redefussion Company that have established several basic solution types that have received high praise and, in most cases, have survived to date as current technology A road existed.

  Refinement of computer-generated imaging (CGI), continuous improvement on game consoles (hardware and software) with real-time interactive CG technology, greater system integration among multiple systems, and AR and (more The expansion of both VR's mobility to a limited degree was included in the major development trends of the 1990's.

  Both the restricted form of mobile VR and the new kind of simulator was the CAVE system, which was developed in 1992 by the Electronic Visualization Laboratory at Illinois State University in Chicago and debuted in the world. (Carolina Cruz-Neira, Daniel J. Sandin, Thomas A. DeFanti, Robert V. Kenyon and John C. Hart, "The CAVE: Audio Visual Experience Automatic Virtual Environment (CAVE)", Communications of the ACM, 35 (6), 1992, 64-72)). Instead of the Lanier HMD / Data Grove combination, CAVE combined the WFOV multi-wall simulator "stage" with a haptic interface.

  At the same time, one form of the stationary part AR was developed by Louis Rosenberg at the Armstrong US Air Force Research Institute as the "Virtual Fixtures" system (1992), while Jonathan Waldern's stationary "Virtuality" VR system (1985) It is recognized that the first development was made early in the year from 1990 to 1990), and also made its commercial debut in 1992.

  A mobile AR that combines real and virtual vehicles with “enhanced simulation” (“AUGSIMM”) integrated into a multi-unit mobile vehicle “war game” system, a form of Loral WDL that was demonstrated in the industry in 1993 The next major step forward was seen. Then, in 1999, Peculiar Technologies project participant Jon Barrilleaux commented on the final 1995 SBIR report's findings, “Experiences and Observations in Applying Augmented Reality to Live Training And “Observations” were written, and the following contents, which are the challenges facing mobile VR and (mobile) AR, are described continuously until now.

  AR vs. VR tracking

  In general, the commercial products developed for VR have good resolution but lack the absolute accuracy and wide area coverage needed for AR and are less for use in AUGSIM.

  For VR applications where users immerse themselves in synthetic environments, relative tracking is more concerned than absolute accuracy. The user's world is completely synthetic, and it is more important to turn your head only 0.1 degrees than to know that you are currently pointing to true north, even within 10 degrees There is no contradiction with the fact that

  An AR system such as AUGSIM can not afford this. AR tracking must have good resolution so that virtual elements appear to move smoothly in the real world as the user's head rotates or the vehicle moves, and virtual elements overlap exactly AR tracking must have good accuracy so that it can be obscured by objects in the real world.

  Computational speed and network speed continue to improve during the 90's, and a new project on the outdoor AR system is launched, which is the project of BARS system at the US Naval Research Institute ("BARS: Battlefield Augmented Reality System (BARS: Battlefield) Augmented Reality System), Simon Julier, Yohan Baillot, Marco Lanzagorta, Dennis Brown, Lawrence Rosenblum; 2000 NATO Symposium on Information Processing Technology for Military Systems). The summary has the following description. "This system consists of a wearable computer, a wireless network system, and a tracked see-through head mounted display (HMD). The perception of the environment by the user is enhanced by superimposing graphics in the user's view This graphic is registered in the actual environment.

  Non-military development is also in progress, including ARToolkit, a study by Hirokazu Kato of Nara Institute of Science and Technology Graduate University, which will be announced later and further developed by HITLab, software development suite and viewpoint tracking and virtual object tracking Introduced the protocol for.

  These landmark events are frequently mentioned as being most important in this period, but other researchers and companies were also active in this area.

  Although military funding for large-scale development and testing of the AR for training simulation is well documented and evident that there was a demand for it, other system-level design and system demonstrations are also available. The military funded research effort was proceeding at the same time.

  Among the most important non-military experiments, development started and led by Bruce Thomas at the University of South Australia's Wearable Computers Institute is AR Quake, an AR version of the video game Quake, “ARQ uake: An Outdoor / Indoor Augmented Reality First Person Application (ARQuake: Outdoor / Indoor Augmented Reality First Person Application) ”(The 4th International Symposium on Wearable Computers, pp. 139-146, Atlanta, GA, October 2000; (Thomas, B Close, B., Donoghue, J., Squires, J., De Bondi, P., Morris, M. and Piekarski, W.) The abstract states: "We present a low-cost, appropriately accurate 6-DOF tracking system based on GPS, digital compass and tracking based on reference vision. "

  Another system, whose design development was launched in 1995, was developed by the author of the present disclosure. It was originally intended to realize a hybrid of the "Everquest Live" dubbed by outdoor AR and TV programs, and its design was further developed through the late 90's, the most important element being finished by 1999 At that time, a commercial effort to fund the original video game / TV hybrid was launched, and by that time, another version was included for use in the development of the finest theme resorts . By 2001, this is to companies including Ridley and Tony Scott, and in particular to their joint venture Airtightplanet (other partners including Renny Harlin, Jean Giraud, and the European Heavy Metal) Disclosed on a confidential basis, the authors of the present disclosure serve as executive supervision services for them, and at that time the “Otherworld” and “Otherworld Industries” projects and ventures at the same time, joint investment and collaboration with ATP It was submitted as a venture proposal.

  The following is an overview of the system design and components completed by 1999/2000.

  Excerpt from "OTHERWORLD INDUSTRIES BUSINESS PROPOSAL DOCUMENT (OTHERWORLD INDUSTRIES business proposal)" (archived document version, 2003):

  Technical background description: proprietary integration of state-of-the-art "Open Field" simulation and mobile virtual reality: tools, equipment and technology

  This is simply a partial list and outline of the relevant art that together form the backbone of a proprietary system. Some technology components are proprietary and some are from external vendors. However, a unique system combining proven components will be absolutely proprietary and innovative.

  Interaction with the VR modified world

  1) Mobile military standard VR equipment for immersing guests / participants and actors in OTHERWORLD VR extended landscapes. Their “adventures” (that is, their respective actions as they explore OTHERWORLD around the resort) are captured in real time (with automatic matting technology) by mobile motion capture sensors and digital cameras, guest / player and employee Members / Actors can see each other through their visors, with the superposition of computer simulation images. A visor is either binoculars (semi-transparent flat panel displays) or binoculars (but opaque flat panel displays with a binocular camera mounted on the front).

  These "composite elements" superimposed on the flat panel display in the field of view can include the altered portion of the landscape (or the entire digitally altered landscape). In fact, these "composite" landscape parts, which actually replace what is there, are generated based on "captured" the original 3D photos of each part of the resort. (Please refer to the following paragraph 7). As an accurate, photo-based geometric "virtual space" in a computer, it is possible to somehow digitally alter this virtual space while maintaining the photorealistic quality and geometric / spatial accuracy of the original capture It is. This produces an accurate combination of live digital photos of the same space and modified digital parts.

  Other "synthetic elements" superimposed by the flat panel display include computer-generated or modified people, creatures, atmosphere FX, and "magic". These appear as realistic elements of the field of view through the display (transparent or opaque).

  Positioning data, motion capture data of guests / players and employees / actors, and real-time matting of these data with multiple digital cameras (all of which are calibrated to a pre-captured version of each area of the resort (See Sections 4 and 5 below) allows the compositing element to be matched in real time with absolute accuracy to the real element shown through the display.

  Thus, a graphic computer-generated dragon can come round, fly back and fly up behind the real tree, and land on the resort's real castle, and the dragon It can "burn" with computer-generated fire. In a flat panel display (translucent or opaque), the fire appears to be coming out of the top of the "blackened" castle. Through the visor, this effect is achieved by the upper part of the castle being "over-matted" by a computer-modified version of 3D "capture" of the castle in the file of the system.

  2) Physical electro-optical mechanical gear for combat between real humans and virtual humans, creatures and FX. Motion sensors and other data, as well as "haptic" interfaces that provide vibration and resistance feedback, allow real-time interaction with real people, virtual people, creatures and magic. For example, a haptic device in the form of a "props" sword handle may have data during the swing of the guest / player and the guest / player poke off the virtual demon to achieve a combat illusion. Provide physical feedback when it is considered. All this is combined in real time and displayed through the binoculars flat panel display.

  3) Open field motion capture device. Mobile and fixed motion capture equipment tools (similar to those used in the movie "Matrix") are deployed across the resort's premises. Data points on themed "gears" worn by guests / players and employees / actors are tracked by cameras and / or sensors and virtual elements within the field of view displayed on binoculars flat panel in VR visor Provide motion data to interact with

  The output from the motion capture data is a CGI modified version of the guest / player and employee / actor in line with the character principle of Goram in the second and third parts of The Lord of the Rings. Enable (with sufficient computational rendering capabilities and use of motion editing and motion libraries).

  4) Enhancement of motion capture data with LAAS & GPS data, live laser distance measurement data and triangulation technology (including Moller Aerobot's UAV technology). The additional "positioning data" allows for even more effective (and error corrected) integration of live and synthetic elements.

  From a press release by a UAV manufacturer.

  July 17. One week ago a contract for the first network of Local Area Augmentation System (LAAS) stations was signed with Honeywell, and several test stations are already running. This system allows the aircraft to be guided to the airport (and vertical take-off and landing airports) accurately to the nearest inch accuracy. The LAAS system is expected to be operational by 2006.

  5) Automatic real-time matting of the open field "play". In combination with motion capture data that allows interaction with simulated elements, resort guests / participants are digitally imaged with a P24 (or equivalent) digital camera and interfaced with proprietary Automatte software Then, automatically separate (mat) the appropriate element from the field of view to be integrated with the synthetic element. This technique will be one of the package software used to ensure proper foreground / background separation when superimposing digital elements.

  6) Military standard simulation hardware and technology combined with cutting edge game engine software. Motion capture systems, haptic devices to interact with "synthetic" elements such as props, combining with data from synthetic and live elements (matted or complete), military simulation software and Integration with game engine software.

  These software components may be AI codes or "AIs such as Massive software used to animate the armies in the movie" Lord of the Ring "or" AI code for animating synthetic humans and creatures. "Artificial intelligence" software is provided to generate realistic water, clouds, fire etc. and unify and combine all elements in other ways just as computer games and military simulation software do.

  7) Photograph-based capture of the actual location (the basis of the "Brett Time" FX in "The Matrix") for creating realistic digital virtual sets with image-based technology, developed by Dr. Paul Debevec as a pioneer.

  The "base" virtual locations (inside and outside of the resort) are indistinguishable from the real world as they are derived from the photos and actual lighting of the location when they were "captured". A small set of high quality digital images combined with data from an optical probe and laser ranging data, and appropriate 'image based' graphics software, create a realistic virtual 3D space in the computer that exactly matches the actual one It is all you need to reproduce.

  The 'virtual set' is captured from the actual location inside the castle and the location outside the surrounding countryside, but once digitized these 'base' or default versions (the originally captured Add the elements that do not exist in the real world, modify the elements that exist to create fantasy versions of our world, and "decorate" with the lighting parameters and all other data at that time) Can be modified (including lighting).

  As guests / players and employees / actors pass “gateways” at various points in the resort (“gateways” are effective “passing points” from “our world” to “otherworld” ), Calibration procedure occurs. In order to "lock" the virtual space in the computer to the coordinates of the "gateway", positioning data from the guest / player or employee / actor at the "gateway" is then taken. The computer "knows" the coordinates of the gateway for the entire virtual version of the resort, acquired through the image-based "capture" process described above.

  Thus, the computer can "align" the virtual resort with what the guest / player or employee / actor sees before putting on the VR goggles. Thus, passing through the translucent version of the binocular flat panel display, one will match very closely to the other if the virtual version is superimposed on the actual resort.

  Alternatively, an "opaque" binocular flat panel display goggle or helmet would allow the wearer to walk confidently while wearing the helmet, looking only at the virtual version of the resort in front of him. Because the landscape of the virtual world will be in perfect agreement with the landscape he is actually walking.

  Of course, what can be shown to the wearer through goggles is a red sky that has been altered, a storm cloud that is not really there, and a dragon that has just "fired" just on the battlemented battlements of the castle. It would be the battlements of the castle that stopped at the top.

  And the army of 1000 ores to charge the distant hills!

  8) Resort super computer rendering and simulation equipment. An important resource that will enable the simulation of very high quality, near-future movie quality will be the supercomputer rendering and simulation complex of each resort scene.

  As with computer games for desktop computers, improvements to graphics and gameplay on stand-alone computer game consoles (PlayStation 2, Xbox, Gamecube) are well known.

  However, consider that such improvements in the gaming experience are based on improvements in the processor and support system of a single console or personal computer. And imagine that you put the power of the supercomputing center into the gaming experience. That alone will be a breakthrough in the quality of graphics and gameplay. And this is only one aspect of the mobile VR adventure that will be the Otherworld experience.

  It will be clear by reviewing the above, which should be obvious to those skilled in the relevant fields (more broadly, the fields of VR, AR, and simulation), but to improve the state of the art The individual hardware or software system proposed must take into account wider system parameters and clarify the assumptions about the system parameters to be evaluated properly.

  Thus, the essence of the proposal (the focus is on hardware technology systems that fall into the category of portable AR and VR technologies, and in fact both are fusions, but the most preferred version of that fusion is wearable technology And the preferred wearable version is HMD technology) is a perfect case to be an excellent solution by considering or reconsidering the whole system that is part of it. Thus, there is a need to present this history of larger VR, AR and simulation systems, because new HMD technology proposals and commercial offers have become too narrow, for example, system-level assumptions, requirements and It is because they tend not to consider or consider new possibilities.

  It is not necessary to review the major milestones in the evolution of HMD technology as well as historically. Because it is a help in explaining the limitations of the prior art of HMD and the current state of the prior art, and the reason for the proposed solution, and why the proposed solution solves the identified problem It will be necessary to provide an available framework, as it has a broader history at the system level.

  What is sufficient to understand and identify the limitations of the HMD prior art begins with:

  In the category of head mounted displays (including helmet mounted displays for the purposes of this disclosure), two major subtypes, "VR HMD" and "AR HMD" have been identified to date, which are According to the meaning of these definitions already provided in the specification, and within the category of AR HMDs, two categories are used to distinguish these types, “video see-through” or “optical see-through” (more Often referred to simply as "optical HMD").

  In the VR HMD display, the user looks at one panel or two separate displays. Although the typical shape of such HMDs is goggles or face mask shapes, many VR HMDs have the appearance of welder's helmets with bulky sealed visors. In order to ensure optimal video quality, immersion and no distraction, the system is completely sealed using a light absorbing material at the peripheral edge of the display.

  The author of the present disclosure has previously filed the US Provisional Application “SYSTEM, METHOD AND COMPUTR PROGRAM PRODUCT FOR MAGNETO-OPTIC DEVICE DISPLAY (System for Magneto-optical Device Display, filed in February, 2004, and incorporated herein) , Methods and Computer Program Products) "60 / 544,591 proposed two types of VR HMDs. One of the two types of wafer type embodiments of the main object of the application merely proposes to replace a conventional direct view LCD, which is an embodiment of the present invention. The first practical magneto-optical display, the superior performance features of which are very high frame rates for improved display technology in general, and in this embodiment other advantages for the improved VR HMD. including.

  The second version, in accordance with the teachings of the present disclosure, contemplates a new type of remotely generated image display, which is generated, for example, in the vehicle cockpit, and subsequently transmitted via a fiber optic bundle, and a special fiber optic array. Based on the experience of fiber optic faceplates that are delivered through the structure (the structures and methods disclosed in the present application) and have new approaches and structures for remote image delivery via optical fibers.

  Initially the core MO technology for HMD was not commercialized, but rather for projection systems, these developments pertain to some aspects of the present proposal and are generally known in the art. Not. In particular, the second version discloses methods published prior to others, and more recent proposals are not integrated into the HMD optical body or not close to the HMD optical body Optical fibers are used to transmit video images from the imaging engine.

  An extremely important consideration for the practicality of a fully enclosed VR HMD for mobility beyond a tightly controlled stage environment with even floors is that the mobility is safe and the virtual world to be navigated Should perform 1: 1 mapping (within the safety deviation for human movement) on the real surface topography or motion trajectory.

  However, consistently by researchers such as Barrilleaux of Loral WDL, who is the developer of BARS, and by other researchers in the field over the last quarter century development about AR systems as a system that should be practical As observed and concluded, virtual (including synthetic CG generated images), including the shape of the vehicle in motion (not surprisingly from the development of a military system for urban warfare) A very close correspondence must be obtained between the world and the real world topography and construction environment.

  Thus, in order for either VR or AR to be possible in mobile form, there must be a 1: 1 positional correspondence between the "virtual" or combining elements and any real-world elements. Is the more common case.

  In the AR HMD category, the distinction between "video see-through" and "optical see-through" means that the user views directly as a transparent or translucent pixel array and display placed directly in front of the viewer as part of the optical glasses. , A translucent projection image on an optical element generated from a microdisplay (typically directly adjacent), transmitted through the form of an optical relay to the facing optical piece, also arranged in front of the viewer It is the distinction between looking through.

  A transparent or translucent display system, which is the main and perhaps only partially practical type of direct view-through display, has been (historically) an LCD configured without a lighting backplane. Thus, in particular, AR video view through glasses hold an optical screen that includes a transparent light substrate on which an LCD light modulator pixel array is fabricated.

  For applications similar to the original Mann's "EyeTap" (text / data is directly displayed or projected onto facing optics), calibration to real world terrain and objects is not required, but to some extent Positional correlation helps to contextually "tag" items in the field of view with informational text. This is the main purpose written for Google Glass products, but when drafting this disclosure, an AR type that a large number of developers superimpose more than text on live scenes Concentrated on the development of applications.

  The main problem with such "calibration" to terrain or objects in the field of view of users of either video or optical see-through systems, other than loose approximate position correlation in a roughly 2D plane or coarse view cone, is in the viewer environment. It is the determination of the relative position of an object. Calculations of significant discordant perspectives and relative sizes can not be made without reference and / or near real time spatial positioning data and 3D mapping of the local environment.

  In addition to relative sizes, an important aspect of perspectives from all perspectives is realistic lighting / shading (including drop shadows) depending on the direction of lighting. And finally, the shielding of objects from a given viewing location is an important optical feature of perceived perspective and relative distance and location.

  There is no video see-through or optical see-through HMD, or the perimeter of the wearer, which is essential for safe movement or path finding for either video or optical view-through type systems or indeed for mobile VR type systems It is not possible to design a video see-through or optical see-through HMD isolated from the problem of how such data is provided to enable dimensional viewing of the. Will such data be provided externally, locally, or in combination of multiple sources? If partially local and part of the HMD, would this affect the overall design and performance of the HMD system? Among other implications and affected parameters in the choice between video see-through and optical see-through, if this problem is to be influenced, a given weight, balance, bulk, data processing requirements, among components What effect does it have on the details of the lag, and the choice between display and optical components?

  Among the technical parameters and problems to be solved as the VR HMD evolves and advances, mainly the increase of the field of view, the reduction of latency (change of lag between motion tracking sensors and change of virtual perspective), resolution, frame rate There has been an increase in dynamic range / contrast, as well as other common display quality features, as well as weight, balance, bulk and general ergonomics issues. Image collimation and other display optics details have been refined to effectively address the "simulator disease" problem that has been a major issue from the outset.

  The weight and bulk of displays, optics and other electronics tended to decrease over time, with improvements in these general technology categories, and weight, size / bulk and balance.

  While stationary VR gears have been commonly used for night vision systems in vehicles (including aircraft), mobile night vision goggles can be considered as a form of mediating viewing similar to mobile VR. This is because basically what the wearer is looking at is the real-time real scene (infra-red imaged) (but through the video screen, not in the form of a "view through").

  This subtype is similar to what Barrilleaux defined as "indirect vision display", retrospectively to 1999 also mentioned. He proposes that there is no actual "view through" but what is visible is an exclusively fused / processed real image / virtual image on the display, perhaps included as any VR type or night vision system The definition is presented for the selected AR HMD.

  However, the night vision system actually, but rather directly transmits, the infrared sensor data as interpreted through video signal processing, as a virtual composite landscape and as a monochromatic image of different intensity depending on the intensity of the infrared signature. It is not a fusion or mixing with a video image. As a video image, this is real-time text / graphics, in the same simple form as Eyetap was originally conceived, and, as Google stated, the main purpose intended for Glass products Suitable for overlays.

  How and what data will be extracted live, or provided from reference to either mobile VR or mobile AR system, or both, or consistently queue out to provide a combined view The question is whether to include now the "indirect view" of this hybrid live processed video feed that has similarities with both categories to enable effective integration of the virtual landscape and the actual landscape. Regardless of type, design parameter, and type, is a problem that must be considered in designing any new and improved mobile HMD system.

  Software and data processing for AR have been advanced to address these issues, based on the earlier work of the system developers mentioned above. An example of this is a pending U.S. patent application entitled "Mixed reality space image generation method and mixed reality system" (U.S. patent application filed on September 29, 2004). There are studies by Matsui and Suzuki of Canon Inc., which are disclosed in 10 / 951,684 (US Patent Application Publication No. 20050179617, now US Patent 7,589, 747). The summary is described as follows.

  “A mixed reality space image generating apparatus for generating a mixed reality space image formed by superimposing a virtual space image on a real space image obtained by capturing a real space An image combining unit (109) for superimposing a virtual space image displayed in consideration of occlusion by an object in space into a real space image, and an image displayed without considering the occlusion of the virtual space image In this way, a mixed reality space image can be generated that can achieve both natural and convenient displays. "

  The purpose of this system is intended to enable the combination of fully rendered industrial products (such as cameras) to be superimposed on the mock-up (optical props), and the optical view through Both HMD glasses and mock-ups are equipped with position sensors. A real-time method for matting pixels from the mockup so that CG-generated virtual models can be superimposed on composite video feeds (buffered to allow layering with slight lag) A pixel-by-pixel search comparison process is used. Annotation graphics are also added by the system. Computer graphic. The most important sources of data to determine the matting, and thus ensure accurate and not false shielding in the composition, are the motion sensors on the mockup, and for drawing the handmat and the mockup mat It is a pre-determined lookup table that compares pixels.

  This system does not help to generalize mobile AR, VR, or any hybrid, but it is simple to analyze real 3D space and position virtual objects properly in perspective view It is an example of an attempt to provide a system (although not fully automatic).

  In the area of video or optical see-through HMDs, even with the ideal calculated mixed reality perspective view delivered to the HMD, satisfying real and accurate fused perspective views (handling of the proper order of perspectives, in real space There has been little progress in the design of displays or optics and display systems that can implement the appropriate shielding of the fusion element from the given viewer position.

  One system that is claimed to be the most effective (even partial) solution to this problem, and perhaps only one integrated HMD system (independent of the HMD, some generic aspect (In contrast to software / photogrammetry / data processing and delivery systems, which was designed to solve these problems in the US), which has already been described (US Patent Application No. 20140177023). ) “APPARATS FOR OPTICAL SEE-THROUGH HEAD MOUNTED DISPLAY WITH MUTUAL OCCLUSION AND OPAQUENESS CONTROL CAPABILITY (for optical see-through head mount display having mutual shielding and opacity control capability) It has been mentioned in the proposal of Chunyu Gao in location) ".

  Gao begins his overview on the field of view-through HMDs for AR from the following point of view.

  There are two types of ST-HMD, optical and video (J. Rolland and H. Fuchs, "Optical versus video see-through head mounted, displays", In Fundamentals of Wearable Computers and Augmented Reality (on the basis of wearable computers and augmented reality), 113-157, 2001). The main drawbacks of the video see-through approach include the degradation of the image quality of the see-through view, the image lag due to the processing of the incoming video stream, the possibility of the loss of see-through view due to hardware / software malfunction. Also, in contrast, optical see-through HMD (OST-HMD) provides real-world direct view through beam splitters, thus the impact on real-world views is minimal. It is highly suitable when requesting an application where the user's awareness of the live environment is paramount.

  However, Gao's view of the video see-through problem is not qualified in the first case by exclusively identifying the prior art video see-through as an LCD, and he also And there is no proof of claim that it must degrade the see-through image (which is omitted on any basis). Those skilled in the art will recognize that this poor quality image view is derived from the results achieved with the early view-through LCD system prior to the recent accelerated progress in the field. I will. Optical see-through as compared to state-of-the-art LCD or other video view-through display technologies, using many optics as a comparison and the influence of other display technologies on "real" "see-through image" reprocessing or mediation It is neither virtually true nor obvious that the system relatively deteriorates the final result or is inferior to the proposal such as Gao.

  Another problem with this groundless generalization is the estimation of lag in the see-through category as compared to other systems that must also process the input live image. In this case, the speed comparison is the result of a detailed analysis of the competing system's overall components and their performance. And finally, the speculation of "the possibility of the loss of see-through view to hardware / software" is between the video and the optical see-through scheme in general, or between and any particular version of its component technology and system design A rigorous analysis of the robustness or stability of the comparison system, either in principle, is basically unfounded, arbitrary, and not effective.

  Beyond the first issue of comparative flawed and biased assertions in the field concerned, the solutions they have proposed themselves are qualitative, together with the data acquisition, analysis and distribution issues already mentioned and addressed. There is a problem (including omission and omission of considering the proposed HMD system as a complete HMD system (also included as a component in a wider AR system)). HMD is not permitted to handle data or processing power of a particular level and quality for the generation of modified or mixed images as "known fact", in which case HMD itself is And important questions and problems that can assist or impede the design, and can not be simply presented as known facts.

  Furthermore, omitted from the task and solution detailing is the full range of real and virtual visual integration problems on the mobile platform.

  The disclosure and the system that the disclosure teaches are particularly as follows.

  As mentioned above in this context, Gao's proposal is to use two display-type devices, because the specifications of spatial light modulators that selectively reflect or transmit live images are fundamental Indeed, this is because they are specifications of the SLM for the same purpose as being operable in any display application.

  And while the output images from the two devices are lined up on a pixel-by-pixel basis, they are combined within the beam splitter combiner (assumed) without any specific explanation other than the description of the accuracy of these devices Be

  However, in order to achieve this fusion of two pixelated arrays, Gao specifies an overlap of what he calls "folding optics" but basically two "folding optics" in total Elements (eg, grating planes / HOE or other compact prisms or "flat" optics), one for each source, and two objective lenses (one for the wavefront from the real scene and one for the combined image) And at the other end of the focal point of the beam splitter combiner, nothing is required other than the double version of the Mann Eyetap scheme.

  Thus, a large number of optical elements, for which he presents a variety of conventional optical variations, 1) a first reflective / folded optical body (planar grating / mirror, HOE, TIR prism, or Collect the light of the actual scene through the other "flat" optics and from there to the objective lens and pass the light to the next grating / mirror, HOE, TIR prism or other "flat" optics To re-fold the light path, all of which are to ensure that the entire optical system is relatively compact and is included in the approximate set of two rectangular optical relay areas. It is a thing. From the folding optics this beam is passed through a beam splitter / combiner to the SLM. The SLM is then reflected or transmitted based on pixelation (sampling), and thus variably modulated (changes from real image contrast and intensity to correct gray scale etc), and now passes through the pixelated real image Let it go back to the beam splitter / combiner. While the display synchronously generates virtual or composite / CG images, such images are also possibly calibrated to ensure easy integration with the corrected pixelated / sampled real wavefront, in multiple steps The beam splitter is passed through to integrate pixel-by-pixel with the samples of the corrected and pixelated real scene, from there through the eyepiece objective, and back to another "folding optics" element, the viewer Is reflected out of the optical system to the eye.

  As a whole, for the real image wavefront of the corrected, pixelated and sampled part, it passes through seven optical elements (not including the SLM) before reaching the viewer's eye. The composite image generated by the display simply passes through two optical elements.

  The problem of accurate alignment of the optical image combiner, ie, at the pixel level, whether it is reflected light collected from the image sample queried by the laser, or the combination of the images produces a small-featured SLM / display The question of whether or not it is considered that maintaining alignment, especially under conditions of mechanical vibration and thermal stress, is not considered trivial in the art.

  Digital projection free space light beam mixing systems, which combine the output of high resolution (2k or 4k) red, green and blue image engines (typically images generated by DMD or LCoS SLMs) are expensive and these It is not easy to achieve and maintain alignment. There are also simpler designs than in the case of the 7 element obstacles of the Gao scheme.

  Furthermore, these complex multi-engine multi-element optical combiner systems are not nearly as compact as needed for HMDs.

  Monolithic prisms such as the T-Rhomboid combiner, developed and marketed by Agilent for the life sciences market, have been developed specifically to address the problems that free space combiners have shown in existing applications.

  And although Microvision and other companies have successfully deployed HMD platforms originally developed for microprojection technology based on SLM, these optical setups are typically more substantial than Gao's proposal Less complicated.

  In addition, what is the basic rationale for the two image processing steps and calculation iterations on the two platforms, and combined to achieve smoothing and integration of the real and virtual wavefront inputs It is difficult to determine why it is needed to perform proper occlusion / opacity of scene elements. The greatest concern of Gao and the problem to be solved is the problem of synthetic images that compete with difficulty with the brightness of the real image, so the main task of the SLM is the real scene part or the real scene It seems to be to selectively reduce the overall brightness. In general, for example, while reducing the intensity of the shielded actual scene element by minimizing the duration of the DMD mirror at the reflecting position in the time division multiplexing system, the shielded pixels are simply It is also speculated that it will be left as it is, but this has not been specified by Gao, nor has it specified the details of how SLM achieves its associated image modification function .

  Among the many parameters that should both be calculated, calibrated, and registered, it is included to determine which pixel from the real field is the one that has been calibrated for the composite pixel. Without perfect agreement, ghost overlap and misalignment and occlusion especially doubles in moving scenes. The position of the reflective optics which causes the wavefront part of the actual scene to pass through the objective has an actual perspective position with respect to the scene which is not initially identical to the perspective position of the viewer in that scene. It is not a plane, it is not located in the middle, and it is merely a wavefront sample, and not a position. Furthermore, if it is mobile, even moving, and not previously known to the composite imaging device. The number of variables in this system is very large, just for the reasons given above.

  If so, and if the purpose of this solution is made more specific, a second display (a total of two displays in the binocular system, ie the specified SLM is added to achieve this) It may become clear that there may be a simpler way than using).

  Second, either approach durability of such composite systems with multiple accumulated alignment resistances, accumulation of defects from multi-element path original parts and aging wear, non-alignment of fused beams The reason is that if it causes accumulated thermal and mechanical vibration effects, seven other elements and additionally other problems arising from the complexity of the optical system, it is likely that the external live image wavefront is essentially degraded It is this system that brings about (especially with age).

  Additionally, as discussed in greater detail above, the problem of computing the spatial relationship between real and virtual elements is not a trivial problem. The design of a system that must drive a four-display type device that may be two (and in a binocular system) possibly different types (and thus different color gamuts, frame rates etc.) based on this calculation has already required It adds complexity to stringent system design parameters.

  In addition, high frame rates are essential to deliver high-performance images without ghosting or lag and without inducing eyestrain or visual system fatigue. However, in the Gao system, the system design is slightly more simplified with the use of only a view-through rather than a reflective SLM, but even with faster FeLCoS microdisplays, the frame rate and image speed are still TT DLP Substantially lower than the frame rate and image rate of MEMS devices such as (DMD).

  However, as higher resolutions are desired for HMDs (to achieve at least a wider FOV), resources for high resolution DMDs such as TT 2k or 4k devices imply very expensive solution resources . Because their feature size and number of DMDs are lower yield, higher defect rates that can typically be accepted for mass consumer or business production and cost, the systems in which they are currently used (TI OEM It is known to have very high price points for digital cinema projectors (commercially marketed by Barco, Christie, and NEC).

  Flat optical projection technology for optical see-through HMDs like Lumus, BAE and others (where shielding is not a design goal, nor is it possible within the scope and capabilities of these approaches), basically It is an instinctively easy step to proceed to duplicate this approach and adjust the real image, and then combine the two images using the conventional optical setup as proposed by Gao. It relies on a large number of flat optics to provide a combination and to do so in a relatively compact space.

  Returning to the current leaders in the two general categories of HMDs, optical see-through HMD and classical VR HMD, to conclude the outline of the background, the outline of the current state-of-the-art is as follows: The point is that the other variants, optical see-through HMD and VR HMD, are both commercially available and are the subject of intensive research and development, and are essentially Google, Glass, and Oculus VR HMD, the Rift A great deal of commercial and academic research has been carried out, including product announcements, publications and patent applications, which have since grown to a breakthrough.

  At the time of writing this document with Glass, a commercially major mobile AR optical HMD, Google has established a breakthrough general awareness and major marketing position in the optical see-through HMD category.

  However, they followed others on the market (including Lumus and BAE (including Q-Sight holographic waveguide technology)) which has already developed and launched products mainly in the defense / industrial sector. Other companies that have entered the recent market and research stages include TruLife Optics, a study from the UK National Physical Reality that commercialized research in the area of holographic waveguides, and they are relatively Insist on superiority.

  For many military helmet-mounted display applications, and for the official main application case for Google's Glass (again, as analyzed above), text to view space and only rough position correlation is needed Superimposition of symbolic graphic elements may be sufficient for many early simple mobile AR applications.

  However, even in the case of information display applications, the higher the density of information tagged with items and terrain in the view space facing the viewer (and ultimately surrounding the viewer), the spatial order of the tags / It is clear that the need for layering to match the perspective / relative position of the tagged element increases.

  Thus, overlap, ie partial shielding of the tag by real elements in the field of view (not just the overlap of the tag itself), necessarily “basic” information display to manage visual clutter Even the desired optical view-through system is the requirement.

  Tags are also not only relative position of tagged elements in the perspective view of real space, but also automated priorities (based on pre-determined or software-calculated priorities) and real-time users Tag size and transparency (specifying the only two major visual cues used by the graphics system to reflect the information hierarchy), as it must reflect both degrees of priority assigned. Must be managed and implemented.

  The question that arises immediately is the basic optical see-through HMD (monocular reticle type or binocular all-glass type), after carefully examining the issues of translucency and overlap / masking of tags and superimposed graphic elements. The relative brightness of the live elements and superimposed generated video display elements (in particular with brightly lit outdoor lighting conditions and very dimly) How to deal with the problem of the illuminated outdoor condition). In order to fully extend the usefulness of these display types, nighttime use is clearly an extreme case of the low light problem.

  Thus, moving beyond the conditions of the most restricted use cases of passive optical see-through HMD types, as the information density increases (such systems are commercially successful and usually dense urban or suburban areas This is expected to happen if the tag gets information from a for-profit business), and "passive" optical see-through HMD mobiles as the light and dim use parameters add to the constraints It is apparent that neither the problems nor the needs of the realistic and practical implementations of AR HMDs can be addressed or addressed.

  So we have to consider passive optical pass-through HMD as an imperfect model to realize mobile AR HMD, and it will only be seen as a temporary footing to an active system when looking back at it later It will be.

  Oculus Rift VR (Facebook) HMD: somewhat parallel to the impact of the Google Glass product marketing campaign, but Oculus solves some of the barriers at the critical starting point of practical VR HMDs and / or substantive There is a difference that it was actually going ahead in the field (in the case of Google, rather than following Lumus and BAE), but the Oculus Rift VR HMD At the time of writing, it is the major pre-mass release VR HMD product that enters and builds the market for widely recognized consumer and business / industrial VR.

  An overview of the advancements at the basic starting point of the Oculus Rift VR HMD can be summarized in the following list of product features.

  o Achieved using a single 7 pw display at 1080p resolution, positioned a few inches from the user's eye and divided into binocular perspective areas on a single display, significantly An expanded field of view. The current FOV is 100 degrees in this document (compared to its original 90 degrees), compared to 45 degrees, which is a general specification of the HMD previously present. Separate binocular optics provide stereo vision effects.

  o Significantly improved head tracking and resulting low lag. This is an advance on improved motion sensors / software, among other handheld and handheld device products with built-in motion sensors (accelerometers, MEMS gyroscopes etc) for 3D position tracking, Nintendo Wii, Apple and It uses miniature motion sensor technology that has moved away from other mobile phone sensor technologies, Playstation PSP (now Vita), Nintendo DS (now 3DS), and the Xbox Kinect system. Current head tracking implements multipoint infrared optical systems using coordinated external sensors.

  o Low latency. This is the result of a combination of improved head tracking and a fast software processor that updates the interactive game software system, but with the display technology used (originally an LCD, with slightly faster OLEDs Limited by its inherent response time).

  o Low sustainability. This is a form of buffering to help keep the video stream smooth, working in combination with faster switching speed OLED displays.

  o Lighter weight, reduced bulk, better balance, and totally improved ergonomics by using ski goggle form factor / material and machine platform.

  Summarizing the net benefits of combining these improvements, such a system may not be structurally or operationally new as a pattern, but with improved components and a particularly effective US Design Patent No. D 701,206, as well as the net effect of any proprietary software, resulted in breakthrough levels of performance and verification of mass market VR HMDs.

  In many cases following these precedents and adopting that approach, and in the case of others who have modified their design based on the success of the Oculus VR Rift configuration, a few have occurred at the same time Product programs, and there are many VR HMD product developers (both branded companies and start-up companies), they are the original 2012 Electronic Expo demonstrations and kickstarter financing campaigns by Oculus VR After that, we made a product plan announcement.

  Among those who followed quickly and those who apparently modified their strategies according to the Oculus VR template were Samsung. Samsung demonstrated the design of the Oculus VR Rift and a development model very similar to Sony's Morpheus at the time of this writing. Emerging companies that have emerged in this area include Vrvana (formerly True Gear Player), GameFace, InfiniteEye, and Avgant.

  None of these system configurations appear to be exactly the same as the Oculus VR, but some use two panels, others use four panels, and the InfiniteEye, according to its claim, is FOV up to 200+ degrees. Used a four panel system to expand the Some use LCD and others use OLED. Optical sensors have been used to improve the accuracy and update rate of head tracking systems.

  All of these systems are implemented because of their essentially fixed location or highly constrained mobility. These make use of on-board and active optical marker based motion tracking systems designed to be used in a living room or enclosed space such as a surgical classroom or simulator stage.

  The systems that differ the most from the Oculus VR scheme are Avent's Glyph and Vrvana Totem.

  The Glyph actually implements a previously established optical view-through HMD solution and structure-based display solution using the Texas Instruments DLP DMD to generate a micro image projected onto a reflective planar optical element. And the same configuration and operation as the planar optical elements of the existing optical view-through HMD, except that a high contrast light absorbing backplane structure is used to realize the reflective / indirect micro-projector display type. Video images belong to the general category of opaque, non-transparent display images.

  However, as demonstrated here above in the discussion of the Gao disclosure, limitations when enhancing display resolution and other system performance beyond 1080p / 2k, when using DLP DMD or other MEMS components Is the cost, manufacturing yield and defect rate, durability and reliability of the system.

  Furthermore, the limitation to image size / FOV from limited expansion / magnification of planar optics (grating structure, HOE or other) (which extends SLM image size but human vision system (HVS) (especially focusing system) Interactions / burdens)) present limitations on viewer safety and comfort. The user's reaction to using similar sized but lower resolution images in the Google Glass test is brighter at higher resolutions, but the additional burden on HVS with equally small image areas poses a challenge for HVS Is suggested. Eli Peli, a doctor at Ophthalmologist, Google's official consultant, told Google Glass users in an interview with the online site BetaBeat a previous warning (May 19, 2014) that predicted eyestrain and discomfort It was followed up and called for a revised warning (May 29, 2014) to limit the cases and scope of potential use. The demarcation is about the eye muscles used in an unintended manner or used for a long time, and this near cause in the revised statement has forced the user to look up It was a small display image position. Other experts

  However, assume that the specific combination of use of the eye muscles necessary for focus use on small parts of the actual FOV is identical to that required for eye movement throughout the actual FOV. I can not do it. In fact, the small tweaking of the focal muscles is more limited / constrained than the range of motion associated with a natural FOV scan. Thus, the repetitive movement of the contractile ROM is not limited only to the focus direction as is known in the art, but the nature of the HVS adds excessive tension beyond normal use, Still further, it is expected to add motion range constraints and the need to make very small controlled fine adjustments.

  The added complexity is that eye strain quickly outpaces precision tooling as the level of detail in the area of constrained eye movement increases in resolution with scenes with complex detailed movement It is. No exacting treatment for this problem has been reported by any of the developers of the optical view-through system, and these problems and over many years with Steve Mann's use of his EyeTap system Reported eye strain, headache and dizziness problems (The problem has been reported to have been partially improved in the latest Digital EyeTap update by moving the image to the center of the field of view but systematic No research has been done), with only limited comments focusing on only a small fraction of the problems and issues of eye strain and “computer vision disease” that can result from precision work.

  However, the limited public comments that Google has made available to Dr. Peli generally suggest that Glass as an optical view-through system is intended for occasional use rather than long-term or high frequency viewing And repeatedly insist.

  Another way to understand the Glyph scheme is to use, at the highest level, a variation of the embodiment for light-split VR operation according to the Mann Digital EyeTap system and the structural arrangement, and the latest optical view through system Side projection deflection optical setup.

  In Vrvana Totem, departure from the Oculus VR Rift is a binocular to allow switching between forward captured image captured on the same optically covered OLED display panel and generated simulation. By adding a conventional video camera, it is to adopt the "indirect view display" scheme of Jon Barrilleaux. Vrvana has shown marketing the material to realize this very basic "indirect viewing display" for AR, just according to Barrilleaux's specified schemes and patterns. While at least affecting the weight and balance of the HMD, it is clear that virtually any other VR HMD of the current Oculus VR generation can be attached to such conventional cameras.

  From the above it is clear from the above content that in the category of "Video See-through HMD" or generally in the field of "Indirect Vision Display" there is little or no progress over the category of night vision goggles I will. Night vision goggles are well developed as a subtype, but within the scope of video processor methods known in the art, any other than providing adding text or other simple graphics to live images It lacks AR features.

  Furthermore, with respect to the existing limitations to VR HMDs, all such systems using OLED and LCD panels have relatively low frame rates, which are motion lag and latency, and this is a broad category of "simulators" Contribute to negative physiological effects on some users who belong to Also, with digital stereo projection systems for movies using such commercially available stereo systems such as RealD systems realized for projectors based on Texas Instruments DLP DMD or projectors based on Sony LCoS, the frame rate too high is also insufficient. Also note that a small percentage of the audience (about 10% in some studies) has also been reported to contribute to experiencing headaches and related symptoms. Some of these are specific to the relevant individual, but a significant percentage of this can be attributed to frame rate limitations.

  And further, as noted, Oculus VR's "buffering system in the pad to compensate for the still inadequately high pixel switching / frame rate of the OLEDs used at the time of this writing" Is realized.

  A further impact on the performance of existing VR HMDs is due to the resolution limitations of existing OLED and LCD panel displays, this resolution limitation allows for a 5-7 inch diagonal display to achieve a sufficiently effective resolution. Which is part of the requirement to attach them using and slightly away from the viewing optics (and the viewer's eyes) and are considerably larger, bulkier and heavier than most other optical headwear products Contribute to the bulk, size and balance of existing and planned offerings.

  Potential partial improvements are expected from adopting curved OLED displays that can be expected to further improve the FOV without adding bulk. However, the cost to market a sufficient quantity that requires significant additional investment in the capacity of a production plant with acceptable yields makes this outlook impractical in the short run. Do. Therefore, the problem of bulk and size is only partially addressed.

  For completeness, point out a video HMD employed to view video content without the ability to navigate the virtual or hybrid (mixed reality / AR) world without being interactive or with any motion sensing capabilities You also need to Such video HMDs have been fundamentally improved over the past 15 years to enhance effective FOV and resolution and viewing comfort / ergonomics, and to be utilized and developed by the latest VR HMDs Provided development paths and advances that have become possible. However, these too have been limited by the core performance of the adopted display technology, with patterns according to the limitations observed for OLED / LCD / DMD based reflection / deflection optical systems.

  Other important variations on projection images in the transparent eyewear optical paradigm include those by Osterhoudt Design Group, Magic Leap, and Microsoft (Hololens).

  These variations bring some relative advantages or disadvantages to each other and to the other prior art discussed in detail above, all of which hold the limitations of the basic approach.

  Commonly more fundamental and even universal, as a frame rate / refresh of existing core display technology, whether high speed LC, OLED or MEMS, and also transferring display images to viewing optics These are also limited by the basic type of display / pixel technology employed, whether or not they employ mechanical scanning fiber inputs or other optical systems disclosed for High quality, eye-friendly (HVS), low power, high resolution, high dynamic range and other display performance parameter requirements are yet to be met, contributing separately and jointly to the realization of AR and VR enjoyable in quality It is enough.

  A summary of the state of the art with the details described above is as follows.

  "High vision" VR has improved in substantially many points: FOV, latency, head / motion tracking, lightness, size and bulk.

  However, the frame rate / latency and resolution, and to the extent that result as a matter of course, the weight, size and bulk are limited by the limitations of core display technology.

  • And modern VR is constrained to stationary or very constrained and restricted mobile applications in small controlled spaces.

  VRs configured as a side projection deflection system based on an enclosed version of an optical view-through system but where the SLM projects the image onto the eye through a series of three optical elements are limited in the performance of the size of the reflected image, This reflected image is magnified but not significantly larger than the output of the SLM (DLP DMD, other MEMS, or FeLCoS / LCoS) compared to the total area of a standard spectacle lens. The very intense version of extended viewing of the "close-up work" and the risk of eyestrain from this which is required of the eye muscles is an additional limitation on practical acceptance. And, SLM type and size displays also limit the practical path to improved resolution and overall performance due to the scaling cost of higher resolution SLMs of the mentioned technology.

  Optical view-through systems generally have the same potential for eyestrain due to the limitation of use to relatively small areas of the eye muscle, and are relatively fine for these constraints and for cases exceeding a short period of use And often require adjustments to track the eye. Google Glass was designed to reflect the prospect of using a limited duration by positioning the optics above and outside the direct rest position of the eye looking straight ahead. However, the user has nevertheless reported eyestrain, as it has been extensively documented in the publication by texts and interviews from Google Glass Explorers.

  Optical view-through systems are limited in the density of superimposed translucency information due to the need to organize tags to real-world objects in perspective views. The need for mobility and information density makes passive optical view-through limited even for graphic information display applications.

  -Aspects of "indirect vision display" have been realized in the form of night vision goggles, and Vrvana, a competitor of Oculus VR, only suggested the adaptation of Totem with a binocular video camera for AR. is there.

  ・ Gao's proposal. It is claimed to be an optical view-through display, but in fact it has a pseudo-view-through aspect due to the use of an SLM device, rather a "indirect view display", which is part of the actual wavefront Are modified for the projection display to function as such, in order to sample and modify portions of the wavefront digitally.

  The number of optical elements involved in the light routing of the first wave front portion (and the point to add here is much smaller (7 or nearly so) than the optical area of the conventional lens of the conventional glasses) ) Provides the opportunity for both screen aberration, image distortion, and loss, but complex free space alignment of many such elements is not common and they are needed, expensive and expensive If it is difficult and not robust, a complex system of optical alignment is required. Neither the method used when SLM is expected to manage the real scene modifications nor the identification or verification of its specific requirements. The problem of coordinating signal processing between two to four types of displays (depending on the monocular in a binocular system) has not been identified or verified, and the problem is that real and synthetic elements in a perspective view Especially in situations where it is already a very demanding requirement to make calculations to create a proper relationship between them, especially when the individual is moving in a densely populated, geographically complex environment It involves determining which pixels from the real field are specifically calibrated pixels for the proper composition. Mounting on vehicles merely exacerbates this problem.

  There are numerous additional problems in developing a complete system, even with the task of constructing an optical set proposed by Gao, or even reducing it to a relatively compact form factor. The size, balance, and weight are just one of the various processing and number of optical body array units and many results that will be (implicitly) required, but other issues and limitations mentioned It is relatively minor compared to. However, it is serious to actually deploy such a system for field use (either for military use or for ruggedized industrial use or consumer use).

  100% "indirect vision display" will have similar needs at key points regarding Gao's proposal (details of the number of display type units, alignment, optical systems, pixel and system matching, and perspective issues) Except that all important parameters of such a system require a "round-robin" calculation of synthetic CG3D mapped space stored in concert with individual real-time perspective real-time view-through images Throw a question about the degree of The problem is that for video images captured by forward video cameras, with the basic Barrilleaux, and now with the possible Vrvana design, all the calculations are done, non-local (HMD and / or Or to the wearer himself) to a degree that must be relayed to the processor.

  What is needed (whether VR or AR) for a truly mobile system that achieves both immersiveness and calibration in a real environment is as follows.

  Ergonomics and viewing systems that minimize non-regular demands on human vision systems. This is to allow for more extended use, which is implied by mobile use.

  -A wide FOV of 120 to 150 degrees (ideally including the peripheral vision).

  High frame rates (ideally 60 fps / eye) to minimize latency and other image artifacts typically caused by the display.

  -High effective resolution at a comfortable distance from the face to the unit. The effective resolution standard that can be used to measure the maximum value is the effective 8k or "retinal display". This distance should be similar to that of conventional glasses, which generally use the bridge bridge as the balance point. Collimation and optical path optics are needed to establish a proper virtual focal plane that also achieves this effective display resolution and the actual distance of the optical element to the eye.

  High dynamic range that matches as closely as possible the dynamic range of the live scene.

  On-board motion tracking to determine both head and body orientation on known terrain (whether known in advance or just in time within the wearer's field of view). This may be supplemented by an external system in a hybrid scheme.

  A display optics system that enables a fast synthesis process within the framework of the human vision system between the wavefront of the real scene and any synthesis element. Many passive means should be used to minimize the burden on any of the on-board (HMD and wearer) and / or external processing systems as much as possible.

  • A relatively simple and crude device with only a few optical elements, only a few active device elements, both minimum weight and thickness, and a simple active device design that is robust under mechanical and thermal stresses Display optics system.

  -Suitable for design configurations known to be acceptable to both professional users, such as lightweight, low bulk, balanced in gravity, and military and durable environmental industrial users, and sport applications, as well as general consumption and A form factor that makes business applications more durable. This is recognized by the factor range from eyeglass manufacturers such as Oakley, Wiley, Nike, and Adidas to Oakley, Adidas, Smith, Zeal and some other more specialized sports goggle manufacturers.

  -A system that is switchable between a VR experience while maintaining full mobility and a variable shielding, perspective integrated hybrid viewing AR system.

  -A system capable of managing the wavelength to be received for the HVS and acquiring valid information from the target wavelength (via the sensor) and these hybrids. Infrared, visible and UV are typical wavelengths of interest.

  In a manner that frees the device and system design from the impaired functionality of the non-optimized operation stage of the capture, delivery, organization, transmission, storage and presentation process to the human vision system or for non-display data array output functionality And, alternatively, a system for imagining the above process in a manner that breaks down the photonic signal processing and array signal processing stages into an operating stage that enables the optimization function of the device most suitable for each stage And methods, which in practice design and operate the devices at the frequency at which these devices and processes operate most efficiently, and more efficient all-optical signal processing (local and long distance Of the “usual frequency” as well as the net effect of Which means that work on specific frequency / wavelength modulation / shift stage.

  The following summary of the invention is provided to facilitate an understanding of some of the technical features related to signal processing and is not intended to describe the invention in full. By taking the entire specification, claims, drawings and abstract as a whole, various aspects of the invention can be fully appreciated.

  Embodiments of the present invention may involve disassembling the components of the integrated pixel signal "modulator" into separate signal processing stages, and thus a telecommunications type network (which may be compact or spatially remote). . The most basic version to be operable is pixel logic "state" coding, typically achieved with an integrated pixel modulator, a color modulation stage separated from this, and then again We propose a three-step "pixel signal processing" sequence, which includes an intensity modulation stage to be separated. A more detailed pixel signal processing system is described in more detail, the pixel signal processing system including substages and options, and detailed and specifically tailored to the efficient implementation of the magnetic photonic system, 1 2.) an efficient illumination light source stage, and 2) a pixel / logic processing and encoding stage it supplies, where bulk light (preferably invisible near infrared) is converted to the appropriate mode and incorporated into a channelized array; This is followed by 3) optional invisible energy filter and recovery stage, 4) optional signal modification stage to improve / modify attributes such as signal splitting and mode modification, 5) frequency / wavelength modulation / shifting and addition Bandwidth and peak intensity management, 6) optional signal amplification / gain, 7) And optional analyzer to complete the switching MO type light valve, 8) consists of an optional configuration for the particular wireless pixel signal processing and delivery (Stage). Furthermore, a DWDM-type configuration of this system is proposed, which provides a version to the all-optical network and a route to the all-optical network, the main costs and efficiencies obtained thereby being particularly motivated And more efficient handling of image information (both live and recorded). And finally, new hybrid magnetic photonic devices and structures have been proposed, enabling others not previously practical for the system of the present disclosure to take full advantage of the pixel signal processing system, Around that, such systems are optimally configured (new basic switches and new hybrid 2D and 3D photonic crystal structures that improve many, if not most, MPC-type devices for all applications A new and / or improved version of the device based on the hybridization of magneto-optical and non-magneto-optical effects (such as slow light and reverse magneto-optical effects)

  In a co-pending application by the inventors of the present disclosure, a new class of display system is proposed that breaks down the components of a typical integrated pixel signal "modulator" into separate signal processing stages. Thus, the basic logic "states" commonly achieved in integrated pixel modulators are separated from the color modulation stage, which is separated from the intensity modulation stage. This may be considered as a telecom signal processing architecture that applies to the problem of visible image pixel modulation. In general, three signal processing stages and three separate device components and operations are proposed, but operations affecting additional signals may be added and are contemplated (polarization characteristics, conventional Conversion of signals to other forms such as polaritons and surface plasmons, superposition of signals (including base pixel on / off states superimposed on other signal data etc) etc.). Highly distributed video / signal processing architecture across a broadband network, providing a relatively "dam" display device that is substantially configured at a later stage of passive material, in series, in the same device or separate The main result is with multiple devices in intimate contact between the devices and compact photonic integrated circuit devices performing individual signal processing steps in a large array.

  The present disclosure of the improved and detailed version of the hybrid telecommunications type, in particular in combination with other pixel signal processing stages / devices, including frequency / wavelength modulation / shifting stages and devices (which may be realized in a robust range of embodiments). Also included are pixel signal processing display systems using magneto-optical / magnetic photonic stages / devices, and the improved and novel hybrid magneto-optical / photonic components are not restricted to the classical or non-linear Faraday effect MO effects, More broadly encompasses irreversible MO effects and phenomena and combinations thereof, and also includes devices based on hybrid Faraday / slowlight and Kerr effects, and hybrid and other MO effects based on Faraday and MO Kerr effects, and Reduce overall device shape size Also includes an improved "light baffle" structure in which the path of the modulation signal is folded in the same plane as the surface of the device, and pseudo-2D and 3D photonic crystal structures and multilayer films PC and surface gratings / polarization PC And hybrids of MO and Mach-Zehnder interferometer devices.

  Thus, including both early MO-based devices and the improved devices disclosed herein, the present disclosure is directed to a pixel signal processing (or, similarly, PIC, sensor, or telecom signal processing) stage. We propose a telecommunications type or telecommunications structured pixel signal processing system of the following process flow, and thus an architecture (and variants thereof) characterized by the system of the present disclosure.

  Any of the embodiments described herein may be used alone or in any combination with each other. The inventions included herein may also include embodiments that are only mentioned or suggested, or not at all, in this summary or in the abstract. The various embodiments of the present invention may be motivated by the various shortcomings of the prior art (which may be discussed or suggested at one or more locations herein). Embodiments of the present invention do not necessarily address any of these drawbacks. In other words, different embodiments of the present invention may address different drawbacks that may be discussed herein. Some embodiments may only partially address some or only one shortcoming that may be discussed herein, and some may not address any of these shortcoming.

  Other features, benefits, and advantages of the present invention will become apparent upon review of the present disclosure, including the specification, drawings and claims.

  BRIEF DESCRIPTION OF THE DRAWINGS The accompanying drawings (like reference numerals in these drawings refer to the same or functionally similar elements throughout the different views, and are drawings that are incorporated herein and form a part of this specification) The foregoing will further illustrate the invention and, together with the detailed description thereof, serve to explain the principles of the invention.

FIG. 1 shows an imaging architecture that may be used to implement an embodiment of the present invention.

FIG. 2 illustrates one embodiment of a photonic converter that implements one version of the imaging architecture of FIG. 1 using a photonic converter as a signal processor.

FIG. 3 shows the general structure for the photonic converter of FIG.

FIG. 4 shows a specific embodiment for a photonic converter.

FIG. 5 shows a generalized architecture for a hybrid photonic VR / AR system.

FIG. 6 shows an embodiment architecture for a hybrid photonic VR / AR system.

  Embodiments of the present invention are devices and methods from the impaired functionality of the non-optimized operation stages of the capture, delivery, organization, transmission, storage, and presentation processes for human vision systems or for non-display data array output functionality. In the manner of releasing the system design, and in the alternative, the above process is renewed in the manner of decomposing the pixel signal processing and array signal processing stages into operation stages that enable the optimization function of the device most suitable for each stage. Provide a system and method for imagining which, in practice, design and operate the devices at the frequency at which these devices and processes operate most efficiently, and a more efficient all-optical signal. Going between such "conventional frequencies" with the net effect of further enabling processing (both local and long distance) As to means to address the efficient frequency / wavelength modulation / shift stage. The following description is presented to enable any person skilled in the art to make and use the invention, and is provided in the context of a patent application and its requirements.

  Various modifications to the preferred embodiments and the general principles and features described herein will be readily apparent to those skilled in the art. Thus, the present invention is not intended to be limited to the embodiments shown, but is to be accorded the widest scope consistent with the principles and features described herein.

  Definition

  Unless otherwise defined, all terms (including technical and scientific terms) used herein are generally understood by one of ordinary skill in the art to which this general inventive concept belongs. It has the same meaning. Furthermore, terms such as those defined in commonly used dictionaries should be construed as having a meaning consistent with the meaning in the relevant technical field and context of this disclosure, and It is understood that unless explicitly defined herein, it is not to be interpreted in an idealized or overly formal sense.

  The following definitions apply to some of the aspects described in relation to some embodiments of the present invention. These definitions may be described in more detail herein as well.

  As used herein, the term "or" includes "and / or" and the term "and / or" includes any and all combinations of one or more of the associated listed items. When an expression such as "at least one of" follows a list of a plurality of elements, it does not qualify the entire list of the elements and does not qualify individual elements of the list.

  As used herein, the singular terms "a", "an" and "the" include plural referents unless the context clearly dictates otherwise. Thus, for example, reference to a singular object may encompass more than one object, unless the context clearly indicates otherwise.

  Also, as used herein in the description and throughout the claims, the meaning of "in" is intended to mean "within" unless the context clearly dictates otherwise. Including (in) and "on". Where an element is referred to as being "on" the other element, it is understood that the element may be directly on the other or there may be intervening elements present between the elements. It will On the other hand, when an element is referred to as being "directly on" another element, there are no intervening elements between the two elements.

  As used herein, the term "set" refers to a collection of one or more objects. Thus, for example, the set of objects may comprise a single object or a plurality of objects. A set of objects may also refer to the elements of the set. One set of objects may be the same or different. In some cases, a set of objects may share one or more common characteristics.

  As used herein, the term "adjacent" refers to near or next to. Adjacent objects may be separated from one another or may actually or directly touch each other. Adjacent objects may be coupled to one another or integrally formed with one another.

  As used herein, the terms "connect," "connected," and "connected" refer to a direct attachment or link. Connected objects, as the context indicates, have no or substantially no intervening objects in one or a set of intervening objects.

  As used herein, the terms "coupled," "coupled," and "coupled" refer to an operative connection or link. The combined objects may be connected directly to each other or indirectly to each other, such as through an intervening set of objects.

  As used herein, the terms "substantially" and "substantially" refer to a significant degree or range. When used in conjunction with an event or situation, the term refers to the case where the event or situation has just occurred, or the case where the event or situation is likely to occur (typical acceptable levels of the embodiments described herein or To explain variability etc.).

  As used herein, the terms "optional" and "optionally" mean that the event or condition described below may or may not occur, and that the description indicates that the event or condition It is meant to include cases that occur and cases in which the event or situation does not occur.

  As used herein, the term "functional element" broadly refers to a decelerating structure that receives energy from the energy providing structure. The term functional element encompasses unidirectional structures and bidirectional structures. In some embodiments, the functional element may be a component or element of a display.

  As used herein, the terms "display" and "display" refer broadly to structures or methods for making display composition elements. The display component is a collection of display image components created from the processed image composition signal created from the display image primitive precursor. This image primitive precursor may also be referred to as a pixel or sub-pixel in other contexts. Unfortunately, the term "pixel" has developed a number of different meanings, including output from pixels / sub-pixels and components of the displayed image. Some embodiments of the invention include embodiments that separate these elements to form additional intermediate structures and elements (some of which are for independent processing), which further include all these elements. As confusion can be caused by calling the structure / pixel as a pixel, various terms are used herein to specifically refer to particular components / elements. An image composition signal is emitted that may be received by the intermediate processing system to create a set of display image primitives from the image composition signal. A collection of display image primitives that create an image when presented to a human vision system (in direct view through a display or projected by a projection system) under intended viewing conditions. A signal in this context means the output of a signal generator which is or corresponds to a display image primitive precursor. Importantly, as long as processing is desired, these signals are stored as signals in various signal storage propagation channels without being sent to free space, in which the signals may be transmitted to other sources. Create an extended wavefront that combines with the other extended wavefronts (which are also propagating in free space). The signal has no handedness and no mirror image (that is, there is no signal that is inverted, upside down, upside down, or inverted, but the image and image portions have different mirror images). In addition, image parts are not added directly (it is difficult to predict the result (it would be difficult if it were possible to overlay one image part on another image part)) and to process the image part It can be very difficult to do. There are many different technologies that can be used as signal generators, and different technologies provide signals with different features or benefits, and different drawbacks. Some embodiments of the present invention enable hybrid assemblies / systems that can take advantage of combinations of multiple technologies while minimizing the disadvantages of certain technologies. The incorporated US patent application Ser. No. 12 / 371,461 describes systems and methods that can advantageously combine such technologies, thus the term display image primitive precursor is a pixel structure for pixel technology. And sub-pixel structures for sub-pixel technology.

  As used herein, the term "signal" refers to the output from a signal generator, such as a display image primitive precursor, which conveys information about the state of the signal generator at the time the signal is generated. In an imaging system, each signal is part of a display image primitive that produces an image or image portion when recognized by the human vision system in the intended conditions. In this sense, a signal is a codified message, that is, a sequence of states of display image primitives in a channel that encodes the message. A collection of synchronization signals from a set precursor of display image primitive precursors may define a frame (or a portion of a frame) of the image. Each signal may have features (color, frequency, amplitude, timing, but not handedness) that may be combined with one or more features from one or more other signals.

  As used herein, the term "human vision system" (HVS) is a biological and psychological process that involves the recognition and visualization of an image from a plurality of discrete display image primitives, either directly or projected. Point to Thus, HVS implies the human eye, optic nerve and human brain in receiving a composite of display image primitive propagation and formation of an image concept based on these primitives received and processed. Do. The HVS is not exactly the same for every person, and there is a general similarity of a significant percentage of the artificial.

FIG. 1 shows an imaging architecture 100 that may be used to implement an embodiment of the present invention. Some embodiments of the present invention contemplate that the formation of a human perceptible image using a human vision system (HVS) from a large set of signal generating structures includes an architecture 100. The architecture 100 includes an image engine 105 including a plurality of display image primitive precursors (DIPP) 110 _i , i = 1 to N (N can be any integer from 1 to dozens, hundreds, or thousands of DIPPs). Including. Each DIPP110 _i, a plurality of image composition signal ₁₁₅ i, i = 1 to N are suitably operated and modulated to produce a (individual image composition signal 115 _i from the DIPP110 _i). These image composition signals 115 _i are processed to form a plurality of display image primitives (DIPs) 120 _j , j = 1 to M (where M is an integer less than, equal to or greater than N). An aggregation / collection of DIPs 120 _j (one or more image composition signals 115 _i occupying the same space and cross-sectional area) that form a display image 125 (or a series of display images, for example, for animation / motion effects) when recognized by the HVS ). HVS is or arrays on a display screen, such as the projected image on the wall or other surface, when presented in a suitable format to reconstruct the display image 125 from DIP120 _j. This is familiar with HVS that recognizes images from arrays of small differently colored or small shapes (such as "dots") of grayscale shading that are small enough in relation to the distance to the viewer (and HVS) It is a phenomenon. Therefore, the display image primitive precursor 110 _i corresponds to the structure, commonly referred to as "pixels" in referring an apparatus for generating an image composition signals from the non-composite color system, therefore, the image composition signals from the composite color system Will correspond to a structure generally referred to as a "sub-pixel" when referring to the device that produces. Many familiar systems employ synthetic color systems, such as RGB image composition signals, which are one image composition signal from each RGB element (e.g., an LCD cell, etc.). Unfortunately, the terms pixel and sub-pixel refer to the hardware LCD cell (sub-pixel), the light emitted from the cell (sub-pixel), and the signal as perceived by the HVS (generally, in an imaging system). Such sub-pixels are mixed and used to refer to many different concepts such as being configured to be imperceptible to the user in a set of conditions intended for viewing. Architecture 100 distinguishes between various "pixels or sub-pixels", and thus different terms are employed when referring to different compositional elements.

  Architecture 100 may include a hybrid structure, where image engine 105 includes different technologies for one or more subsets of DIPPs 110. That is, the first subset of DIPPs may use a first color technology (eg, mixed color technology) to create a first subset of image composition signals, and the second subset of DIPPs may A second color technology different from the first color technology (eg, different mixed color technology or non-synthetic color technology) may be used to create the second subset of image composition signals. This allows the use of a variety of techniques to create a set of display image primitives and the combination with display image 125, which is better than if it were created from any one technology obtain.

Architecture 100 further receives an image composition signal 115 _i as an input, and a signal processing matrix 130 to create a display image primitive 120 _j as an output. There are many possible arrangements of the matrix 130 (some embodiments including single dimensional arrays), depending on the fit and purpose of a particular implementation of one of the embodiments of the present invention. In general, matrix 130 includes a plurality of signal channels (e.g., channels 135 through 160). There are many different possible arrangements for each channel of matrix 130. Since each channel is sufficiently separated from the other channels, such as optical separation resulting from individual fiber optic channels, the signals in one channel have the crosstalk threshold for that embodiment / embodiment. Do not interfere with other signals beyond. Each channel includes one or more inputs and one or more outputs. Each input receives an image composition signal 115 from DIPP 110. Each output creates a display image primitive 120. From input to output, each channel directs pure signal information, and the pure signal information at any point in a channel is the original image composition signal 115 and one or more processed original images The deconvolution of the set of composition signals and / or the aggregation of the set of one or more processed original image composition signals, each "processing" may be one or more aggregations of one or more signals or It may have included disaggregation.

In this context, aggregation refers to the number of S _A (S _A > 1) channels (these aggregated signals may themselves be the original image composition signals, processed signals, or a combination) T _A coupled signal to a channel with _A number (1 ≦ T _A <S _A ), and disaggregation is a channel with signal channel S _D number (S _D 11) (it is itself the original image composition signal, processed It refers to the division of signals into T _D numbers (S _D <T _D ) from channels, which may be signals or combinations. S _A may be greater than N, such as with early deaggregation without any aggregation, and S _D may be greater than M with subsequent aggregation. In some embodiments, S _A = 2, S _D = 1 and T _D = 2. However, the architecture 100 allows many signals to be aggregated, which can create a sufficiently strong signal, which can be broken into many channels, each channel being It is strong enough to be used in the embodiment. Signal aggregation enables joining, fusing, combining, etc., of the signals propagated by the adjacent channels, following aggregation (eg, joining, merging, combining, etc.) of the channels or other arrangements of adjacent channels. Signal disaggregation enables splitting, splitting, splitting, etc. of the signal propagated by the channel, subsequent to splitting (eg, splitting, splitting, splitting, etc.) of the channel or other channel arrangement. For aggregating two or more signals in multiple channels while preserving the signal states of content propagating through matrix 130 (or for separating one signal in one channel into multiple signals of multiple channels) In some embodiments, there are particular structures or elements of the channel).

There are many representative signal channels shown in FIG. Channel 135 represents a channel having a single input and a single input. Channel 135 receives an image composition signal 115 _k single original, and creates a single display image primitive 120 _k. This is not to say that channel 135 does not do anything. For example, this process may include transformation of physical features. The physical size dimension of the input of channel 135 is adapted to match / supplement the active area of its corresponding / related DIPP 110 that produces image composition signal 115k. The physical size of the output is not required to match the physical size dimension of the input. That is, the output may be relatively tapered or expanded, or the circumferential input may be a straight line peripheral output. Other transformations include repositioning of the signal, on the other hand, the image composition signal 115 _i is started in the vicinity of the image composition signals 115 _2, the display image primitives 1201 created by the channel 135, "Remote before The image composition signal 115 _x may be located next to a display image primitive 120 _x created from the signal _x . This allows great flexibility in interleaving signals / primitives separated from the technology used to create them. The possibility of individual or collective physical transformations is an option for each channel of matrix 130.

Channel 140 shows a channel having a pair of inputs and a single output (aggregating the pair of inputs). Channel 140, two original images composition signal, to receive, for example, signals 115 ₃ and the signal 115 _4, for example, to create a single display image primitive 120 _2. Channel 140, like the primitive 120 ₂ has a greater amplitude than either of the constituent signals, two amplitude allowing it to be added. Channel 140 also allows for improved timing by interleaving / multiplexing composition signals, each composition signal may operate at 30 Hz, but the resulting primitive may be operated at 60 Hz, for example.

Channel 145 represents a channel having a single input and a pair of outputs (allowing the inputs to be disjointed). Channel 140 is a single original image composition signal, for example, receives the signal 115 _5, a pair of display image primitives, namely to create a primitive 120 ₃ and primitive 120 _4. Channel 145 allows a single signal to be reformed, such as by being branched into two parallel channels with many of the features of the disjointed signal (possibly except for the amplitude). If the amplitude is not desired as described above, the amplitude may be increased by aggregation, and disaggregation results in a sufficiently strong signal as demonstrated in the other of the representative channels shown in FIG. Can be generated.

  Channel 150 represents a channel having three inputs and a single output. Channel 150 is included to emphasize that virtually any number of independent inputs may be aggregated into the processed signal, eg, in a single channel for creation of a single primitive 120.

Channel 155 shows a channel with a single input and three outputs. Channel 150 is included to emphasize that a single channel (and the signals therein) can be virtually deconstructed into any number of independent but related outputs and primitives. ing. Channel 155 is different from channel 145 in another manner. That is, the amplitude of primitive 120 is created from the output. In channel 145, each amplitude may be branched to equal amplitudes (although some disaggregation structures allow for variable amplitude branching). In the channel 155, primitive 120 ₆ may not be equal to the amplitude of the primitives 120 ₇ and 120 ₈ (e.g., primitives 120 _6, approximately twice the amplitude of the respective amplitudes of the primitive 120 ₇ and primitive 120 ₈ It may be because all the signals need not be disjointed at the same node). First division results in one signal-half to create a primitive 120 ₆ As a result, the first signal of 2 minutes which is obtained as a result is divided into more half, each primitive 120 ₇ and primitive 120 It may be _eight .

  Channel 160 represents a channel that contains both an aggregation of three inputs and a disaggregation into a pair of outputs. Channel 160 is included to emphasize that a single channel can include both signal aggregation and signal disaggregation. Thus, one channel may have multiple regions of aggregation and multiple regions of disaggregation, as needed or desired.

  Thus, the matrix 130 is a signal processor with manipulation of the physical and signal characteristics of the processing stage 170 including aggregation and disaggregation.

  In some embodiments, the matrix 130 is a precise physical structure that defines the channels, such as a jacquard process for a set of optical fibers that collectively define thousands or millions of channels. It may be created by the weaving process.

Broadly speaking, embodiments of the present invention may include an image generation stage (e.g., image engine 105) coupled to a primitive generation system (e.g., matrix 130). The image generation stage includes a number N of display image primitive precursors 110. Each of the display image primitive precursor 110 _i generates a corresponding image composition signal 115 _i. These images composition signal 115 _i is input to the primitive generating system. The primitive generation system includes M input channels (M may be equal to N, but need not match. In FIG. 1, for example, some signals are not input to matrix 130), and input stage 165 Including. The input of the input channel receives an image composition signal 115 _x from a single display image primitive precursor 110 _x . In FIG. 1, each input channel has an input and an output, and each input channel directs its single original image composition signal from its input to its output, and the input stage 165 has M inputs and M Output. The primitive generation system also includes a delivery stage 170 having P delivery channels, each delivery channel including an input and an output. In general, M = N, and P can vary depending on the embodiment. In some embodiments, P is less than N, eg, P = N / 2. In these embodiments, each input of the distribution channel is coupled to a unique pair of outputs from the input channel. In some embodiments, P is greater than N, eg, P = N * 2. In these embodiments, each output of the input channel is coupled to an input of a unique pair of distribution channels. Thus, the primitive generation system scales the image composition signal from the display image primitive precursor, i.e. multiple image composition signals may be combined as a signal in the delivery channel, and in other cases only a single one. One image composition signal is split and presented to multiple delivery channels. There are many possible variations of matrix 130, input stage 165, and delivery stage 170.

  FIG. 2 illustrates an embodiment of an imaging system 200 that implements a version of the imaging architecture of FIG. The system 200 preferably includes a plurality of image composition signals (IR / A) provided to a photonic signal converter 215 that produces a set 220 of digital image primitives 225 at visible frequencies, and more specifically at real world visible imaging frequencies. A set of coded signals 205, such as at near infrared frequencies) is included.

  FIG. 3 shows the general structure of the photonic signal converter 215 of FIG. The converter 215 receives one or more input photonic signals and produces one or more output photonic signals. The converter 215 adjusts various features of the input photonic signal such as signal logic states (eg, ON / OFF), signal color states (visible from infrared), and / or signal strength states.

  FIG. 4 shows a particular embodiment of photonic converter 400. Converter 405 includes an efficient light source 405. The light source 405 may include, for example, an infrared and / or near infrared source (e.g., an infrared and / or near infrared emitting LED array) for implementation of the optimal modulator in a subsequent stage. Converter 400 includes an optional bulk light energy source homogenizer 410. Homogenizer 410 provides a structure that homogenizes the polarization of the light from light source 405 if necessary or desired. Homogenizer 410 may be arranged for active and / or passive homogenization.

  The converter 400 then includes an encoder 415 in the order of propagation of light from the light source 405. The encoder 415 provides logical encoding of the light (which may be homogenized) from the light source 405 to create an encoded signal. The encoder 405 may include a hybrid magnetic photonic crystal (MPC), a Mach Zehnder, a transmission valve, and the like. The encoder 415 may include an array or matrix of modulators to set the state of the set of image composition signals. In this regard, individual encoder structures may operate equivalently to display image primitive precursors (eg, pixels and / or sub-pixels and / or other display light energy signal generators).

  Converter 400 includes an optional filter 420 such as a polarization filter / analyzer (eg, a photonic crystal derivative mirror) combined with a planar deflection mechanism (eg, a prism array / grating structure).

  Converter 400 includes an optional energy recapturer 425 that recaptures energy from light source 405 (eg, infrared / near infrared polarized energy) deflected by elements of filter 420.

  Converter 400 includes an adjuster 430 that modulates / shifts the wavelength or frequency of the encoded signal (which may be filtered by filter 420) generated from encoder 415. The adjuster 430 may include fluorescent material, periodically polarized material, shocked quartz, or the like. The adjuster 430 receives the generated / switched infrared / near infrared frequencies and converts them to one or more desired frequencies (eg, visible frequencies). The adjuster 430 need not shift / modulate all input frequencies to the same frequency, and may shift / modulate different infrared / near infrared input frequencies to the same output frequency. Other adjustments are also possible.

  Converter 400 optionally includes, for example, a second filter 435 for infrared / near infrared energy, and may optionally include a second energy recapturer 440. Filter 435 may include a photonic crystal derivative mirror combined with a planar deflection structure (eg, a prism array / grating structure).

  Converter 400 may also include an optional amplifier / gain adjustment 445 for adjusting one or more parameters (eg, increasing the signal amplitude of the encoded, optionally filtered, and frequency shifted signals, etc.) . Other or additional signal parameters may be adjusted by adjustment 445.

  FIG. 5 shows a generalized architecture 500 for hybrid photonic VR / AR system 505. The architecture 500 exposes the system 505 to the surrounding real world synthetic electromagnetic surface and creates a set 510 of display image primitives for the Human Vision System (HVS). The set of display image primitives 510 includes or may use information from the real world (AR mode), or this set of display image primitives is completely created by the synthetic world (VR mode) Information may be included. System 505 may be configured to be selectively operable in either mode or in both modes. Further, system 500 may be configured such that the amount of real world information used in AR mode may be selectively changed. System 505 is robust and versatile.

  System 505 may be implemented in different manners. One embodiment creates an image composition signal from the composite world, and interleaves the composite signal in AR mode with an image composition signal ("real world signal") created from the real world. These signals can be channelized, processed and distributed as described in the incorporated patent application 12 / 371,461, using the signal processing matrix of the separated optical channels. System 505 includes a signal processing matrix that may incorporate various passive and active signal manipulation structures in addition to distribution, aggregation, disaggregation, and / or formation of physical features.

  These signal manipulation structures may also change based on the goals of the particular arrangement and design of system 505. For example, these manipulation structures may include real world interface 515, augmenter 520, visualizer 525, and / or output constructor 530.

  Interface 515 is similar in function to the image primitive precursor function in converting the complex synthetic electromagnetic surface of the real world into a set of real world image composition signals 535 that are channelized, distributed and presented to the augmenter 520. Including.

  As described herein, system 505 is quite versatile, and there are many different embodiments. The features and functions of the manipulation structure can be influenced by a wide range of considerations and design goals. Not all of this can be explicitly described in detail herein, but some representative embodiments are presented. As described in the incorporated patent applications and herein, each technology may be produced using a combination of technologies (e.g., hybrids), which may be particularly advantageous for some of the creation of the set 510 of DIPs. The architecture 500 is effective in order to provide better overall results than relying on a single technology for all parts of.

  For example, the complex synthetic electromagnetic wave surface of the real world includes both visible and invisible wavelengths. Since the set of DIPs 510 also includes visible wavelengths, it may be considered that the signal 535 should be visible as well. As described herein, not all embodiments can achieve excellent results when the signal 535 is in the visible spectrum.

  System 505 may be configured for use including visible signal 535. There are advantages to some embodiments that provide the HVS with signals 535 that use wavelengths that are not visible. In the present specification, the range to which the electromagnetic spectrum relates is defined as follows.

  a) Visible radiation (light) is electromagnetic radiation having a wavelength of 380 nm to 760 nm (400 to 790 terahertz) that is detected by HVS and recognized as visible light.

  b) Infrared (IR) radiation is invisible (for HVS) electromagnetic radiation having a wavelength of 1 mm to 760 nm (300 GHz to 400 THz) and is far infrared (1 mm to 10 μm), mid infrared (10 to 2) .5 μm) and near infrared (2.5 μm to 750 nm).

  c) Ultraviolet (UV) radiation is invisible (for HVS) electromagnetic radiation having a wavelength of 380 nm to 10 nm (790 THz to 30 PHz).

  Interface 515 of the invisible real world signal embodiment produces a signal 535 in the infrared / near infrared spectrum. In some embodiments, invisible signal 535 is created using a spectral map that maps specific wavelengths or bands of visible spectrum wavelengths to predetermined specific wavelengths or bands of infrared spectrum wavelengths. It is desirable to be done. This has the advantage of allowing signal 535 to be efficiently processed in system 505 as an infrared wavelength, and allows system 505 to restore signal 535 to real world colors. Including the benefits.

  Interface 515 may include other functional and / or structural elements such as filters to remove infrared and / or UV components from the received real world radiation. In some applications, such as for night vision modes that use infrared, interface 515 excludes the infrared filter or allows some of the infrared of the received real world radiation to be sampled and processed It has an infrared filter.

  Interface 515 includes a real world sampling structure for converting the filtered received real world radiation into a matrix of processed real world image composition signals (similar to the matrix of display image primitive precursors), and The processed real world image composition signals are channelized into a signal delivery and processing matrix.

  The signal distribution and processing matrix may also include frequency / wavelength conversion structures to provide processed real world image composition signals in the infrared spectrum (if desired). Depending on which additional signal operations are to be performed later in the system 505 and which encoding / switching technology is implemented, the interface 515 may, for example, polarize the selected features of the filtered real world image composition signal. Pre-processing may be performed including filtering functions (eg, polarization filtering of infrared / UV filtered real world image composition signals, or polarization filtering, sorting, polarization homogenizing, etc.).

  For example, in a system 505 that includes a structure or process for modifying signal amplitude based on polarization, interface 515 may properly prepare signal 535. In some implementations it is desirable to have the default signal amplitude at the highest value (e.g., default "ON"), but in other embodiments it is preferable to have the default signal amplitude at the lowest value (e.g., default "OFF") It may be desirable, and in other implementations, it may have several signal channels that provide defaults under different conditions, and not all default ON or default OFF. Setting the polarization state (visible or not) of the signal 535 is a role of the interface 515. All signals 535 or other signal characteristics for a selected subset of signals 535 may be set by interface 515 as determined by design goals, technology, and implementation details.

  The real world channelized image composition signal 535 is input to the augmentor 520. Augmenter 520 is a special structure in system 505 for further signal processing. This signal processing may be multifunctional operating on signal 535, some or all of which may be considered "pass-through" signals based on how augmenter 520 operates on them. These multiple functions include: a) Manipulation of signal 535, b) desirable characteristics, eg, independent amplitude control of each individual real world image composition signal, frequency / wavelength setting / modification, and / or logic state Creation of a set of independent combined world image composition signals, and c) interleaving of some or all of the "pass-through" real world image composition signals and the set of created world image composition signals in a desired ratio Then, it may include creating a set of interleaved image composition signals 540.

  Augmenter 520 is a generator of a set of synthetic world image composition signals in addition to a processor of the received image composition signals (eg, the real world). System 505 is configured such that all signals are processed by augmenter 520. There may be many different ways to implement the augmenter 520, for example, with multilayer optical element composites (each gate is associated with one signal) where the augmenter 520 defines gates to valve multiple radiations In some cases, some gates configured for possible pass-through individually receive some of the real-world signals for controllable pass-through, and how many were configured for creation of a combined world signal The gate receives background radiation separated from the pass-through signal for creation of a synthetic world image composition signal. Thus, these gates creating a synthetic world in such an embodiment create a synthetic world signal from background radiation.

  As shown, the architecture 500 includes multiple (eg, two) independent sets of display image primitive precursors that are selectively and controllably processed and fused. The interface 515 functions as one set of display image primitive precursors, and the augmenter 520 functions as a second set of display image primitive precursors. This first set produces image composition signals from the real world, and the second set produces image composition signals from the synthetic world. In particular, architecture 500 enables system 505 to have an additional set of display image primitive precursors available (one or more to be a total of three or more display image primitive precursors). An additional channelized set of image composition signals can be made available to the augmenter 520 for processing.

  As one way to consider architecture 500, augmenter 520 defines a master set of display image primitive precursors that produce interleaved signal 540, where one of the interleaved signals is initially one of the display image precursors. Some are created by the above preliminary set (e.g., interface 515 creates real world image composition signals) and others are created directly by augmenter 520. Architecture 500 does not require that all display image primitive precursors use the same or complementary technology. By providing all composition signals in a organized and predetermined format (eg, in a common channel range compatible with signal manipulations such as signal amplitude modulation by the augmenter 520 and in independent channels) Architecture 500 may provide a powerful, robust and versatile solution for one or more of the disadvantages, limitations and disadvantages of modern AR / VR systems.

  As described herein, channelized signal processing and distribution arrangements aggregate, separate and / or otherwise process individual image composition signals as the signals propagate through system 505. It can. As a result, the number of signal channels in signal 540 may be different from the sum of the number of pass-through signals and the number of signals generated. Augmenter 520 interleaves the first amount of real world pass-through signal and the second amount of combined signal (in the pure VR mode of system 505, the first amount is zero). Interleaved in this context roughly includes the presence of both types of signals, and each real-world pass-through signal is present in a channel physically adjacent to another channel that contains a combined world signal. It is not intended to require. The routing may be independently controlled by the channel delivery characteristics of system 505.

  The visualizer 525 receives the interleaved signal 520 and outputs a set 545 of visible signals. In system 505, the synthetic world image composition signal of signal 540 was created in the invisible range of the electromagnetic spectrum (eg, infrared or near infrared). In some embodiments, part or all of the real world signal 535 passed through by the augmenter 520 has been converted into the invisible range of the electromagnetic spectrum (also may overlap, or all or part) May be included in the range of synthetic world signals). The visualizer 525 performs frequency / wavelength modulation and / or conversion of an invisible signal. If the signal (synthetic and real world) is defined and created using an invisible false color map, then the appropriate colors are restored to the frequency corrected real world signal, and synthetic in terms of real world colors. The world can be visualized.

  The output constructor 530 creates a set 510 of display image primitives from the visible signal 545 for recognition by the HVS (e.g., by direct view or by projection). The output constructor 530 may include, among other possible functions, consolidation, aggregation, disaggregation, channel relocation / relocation, physical feature definition, ray formation, and the like. The constructor 530 may amplify some or all of the visible signal 545, bandwidth correction (eg, aggregation and time multiplexing of multiple channels with the signal in a pre-configured timing relationship, ie, they are created with different phases) And may be combined as a signal to create a stream of signals at any of the many frequencies of the stream) and other image composition signal manipulations. Two streams in a 180 degree out-of-phase relationship may double the frequency of each stream. Three streams with a 120 degree phase relationship may be tripled in frequency, and so on for the other N = 1 or more multiplexed streams. The fused streams, which are then in phase with one another, may increase the signal amplitude (eg, two in-phase streams may double signal amplitude etc.).

  FIG. 6 shows a hybrid photonic VR / AR system 600 implementing an embodiment of the system 500. System 600 includes hatched boxes that map the corresponding structures between system 600 and system 505 of FIG.

  The system 600 includes an optional filter 605, a “signalizer” 610, a real world signal processor 615, a radiation diffuser 620 powered by a radiation source 625 (eg, infrared), a magnetic photonic encoder 630, a frequency / wavelength converter 635, A signal processor 640, signal consolidator 645, and output shaper optics 650 are included. As described herein, there are many different embodiments and embodiments, some of which include different technologies with different requirements. For example, some embodiments use radiation in the visible spectrum and do not require an element for wavelength / frequency conversion. For pure VR implementations, structures that handle real world signals are not required. Minimal post-visualization integration and formation may be required or desirable. Architecture 500 is very flexible and can be adapted to a preferred set of technologies.

  Filter 605 removes unwanted wavelengths from ambient real world illumination incident on interface 515. What is undesirable depends on the application and design goals (eg, night vision goggles may require some or all of the infrared, but other AR systems may remove UV / infrared) May be desirable).

  The signalizer 610 acts as a display image primitive precursor to convert the filtered incident real world radiation into a real world image composition signal and insert the individual signals into the optically separated channels of the signal distributor stage. Function. These signals may be based on synthetic or non-synthetic imaging models.

  Processor 615 filters polarization and / or filters, sorts, and homogenizes polarization if some or all of the real world pass-through image composition signal is to be converted to a different frequency (eg, infrared) It may include a wavelength / frequency converter, which is a structure.

  Diffuser 620 receives radiation from the radiation source and sets a background radiation environment for encoder 630 to generate a synthetic world image composition signal. Diffuser 620 maintains the background radiation separated from the real world pass-through signal channel

  The encoder 630 simultaneously receives and processes real world pass-through signals (e.g., it can modulate these signals, among others) to create a combined world signal. The encoder 630 interleaves / alternates signals from the real world and from the synthetic world and maintains these signals in optically separated channels. In FIG. 6, the real world signals are marked by solid arrows and the synthetic world signals are marked as open arrows to illustrate interleaving / alternate. FIG. 6 is not meant to imply that the encoder 630 needs to reject significant portions of the real world signal. The encoder 630 may include a matrix of many display image primitive precursor type structures to process all real world signals and all composite world signals.

  If a converter 635 is present, it converts the invisible signal into a visible signal. Thus, converter 635 may process synthetic world signals, real world signals, or both. In other words, this conversion may be enabled for the individual channels of the signal distribution channel.

  If a signal processor 640 is present, this may correct for signal amplitude / gain, bandwidth, or other signal modification / modulation.

  If signal consolidator 645 is present, it organizes the signals from visualizer 525 (eg, consolidate, declassify, route, group, cluster, duplicate, etc.).

  If the output shaper optic 650 is present, it performs any necessary or desirable signal formation or other signal manipulation to create the desired display image primitives that the HVS should recognize. This may include direct view, projection, reflection, combinations and the like. Routing / grouping may allow for 3D imaging or other visual effects.

  System 600 is (and, if so, included) in the display image precursor to be propagated to the HVS as part of the other signals in the other display image precursors from the time the signal is generated. It can be realized as a stack of functional photonic assemblies that receive, process and transmit the signal in separate optically separated signal channels, sometimes as an integrated functional photonic assembly.

  The field of the invention is not one, but rather a combination of two related fields, augmented reality and virtual reality, and addresses an integrated mobile device solution that solves the key problems and limitations of the prior art in both fields. And provide this. A brief review of the background of these related areas will clarify the issues and limitations to be solved and will prepare for the proposed solution of the present disclosure.

  The definitions of two standard dictionaries of these terms (Source: Dictionary.com) are as follows.

  Virtual Reality: "Realistic simulation of an environment including 3D graphics by a computer system using interactive software and hardware. Abbreviation: VR"

  Augmented Reality: "an image or environment expanded to be viewed on a screen or other display by overlaying computer generated images, sounds or other data in a real world environment. And:" such an expanded environment System or technology used to create a "abbreviation: AR"

  From the above definition (although it is non-technical), the most important difference for the person skilled in the relevant field is the completely and immersive type in which the simulated elements are completely screened even in real partial dilation. It is clear whether it is a simulation or whether the simulated element is superimposed on a clear and unobstructed real view.

  The Wikipedia entry for this topic provides a slightly more technical definition that is considered to be a good representation of the field, given the depth and extent of its contribution to the compilation of the page.

  Virtual reality (VR) (sometimes referred to as immersive multimedia) is a computer-simulated environment that can simulate physical presence in the real world or a place in an imaginary world. Virtual reality can reproduce perceptual experiences such as virtual taste, vision, smell, hearing, and touch.

  Augmented reality (AR) is a live, direct or indirect view of a physical real-world environment whose elements are augmented (or supplemented) by computer-generated perceptual inputs such as voice, video, graphics or GPS data.

  Being specific to these definitions but simply implicit is a very important attribute of mobile perspectives. A more general class of computer simulation (with or without "real-time" real "direct" imaging (whether local or remote), fusion, synthesis, or integration), virtual or extended It is the simulated or hybrid (enhanced or "mixed") real "simultaneous" images that distinguish reality, and as the viewer moves in the real world, the viewer's viewpoint is also with the viewer It is the point of moving.

  The present disclosure is intended to distinguish this more accurate definition from the immersive displayed and experienced simulated world static navigation (simulator) and mobile world of the simulated world (virtual reality). It proposes that it is necessary. And subcategories of the simulator allow partially "virtually realistic" navigation of the simulated world, stationary users wear immersive HMDs (head mounted displays) and haptic interfaces (eg motion tracking gloves) It will be an "Individual Simulator" or at most a "Partial Virtual Reality".

  On the other hand, the CAVE system will be regarded as generally suitable as a restricted virtual reality system. This is because navigation beyond the size of CAVE will only be possible with movable floors, and once CAVE's own limit has been reached, another form of "partial virtual reality" follows. It becomes ".

  Note the difference between the "mobile" and "mobile" viewpoints. A computer simulation such as a video game is a simulated world or unless the seeker in the simulated world is personally moving or otherwise directing the movement of another person or robot. "Reality" and what can be said there (This achievement among the major achievements of computer graphics in the past 40 years is simply to "build up" a simulated world that is searchable in software ), The simulated world is "navigable".

  Because simulation is either virtual or hybrid (author's favorite term) real, so critical features are mapping of the simulation to real space (whether completely synthetic or hybrid) It is that there is. Such real space may be just as simple as mapping and calibrating at a ratio to the simulated world, as basic as a room in a laboratory or sound stage.

  This distinction is not evaluative, because it does not map to real or non-mobile real reality topography (whether natural, artificial or hybrid or not) in real time natural interface ( Partial VR providing head tracking, tactile, auditory etc) is not of fundamentally lower value than partial VR systems simulating physical interaction and providing perceptual immersion . However, without the foot disease treatment feedback system, or more universally the whole body movement range feedback system, and / or the whole body movement on the terrain (as the rest) simulated (as the feeling of the user) ( The VR system is "partial" by definition, without a dynamically deformable mechanical interface or interaction surface that supports standing, sitting or reclining.

  However, in the absence of such an ideal full-body physical interface / feedback system, restricting the VR to a "full" mobile version means that the topography of the VR world can be found in the real world Or built from scratch). Such limitations will generally severely limit the scope and power of the virtual reality experience.

  However, as will become apparent in the following disclosure, this distinction brings about changes. Because this distinction sets out how existing VR and AR systems differ, and sets a "clear boundary reference" for these limitations, as well as providing a background for informing the teachings of this disclosure. is there.

  Although missing but very important simulation features and requirements have proved to be a complete "virtual reality", the next step is what does the "mobile perspective" realized mean To identify the implicit problem of The answer is to provide a view of the simulation that is mobile, which is two components that are themselves realized by a combination of hardware and software, ie an animation display means (by which the simulation is viewed Requires motion tracking means (capable of tracking the movement of the device including the display in 3-axis motion), which means that the position of the 3D viewing device at least three tracking points at least If mapped and measured, and the third position of the third axis is inferred, it means to measure at two tracking points), and the reference three-axis frame (arbitrary mapped to the real space) A plane in which two axes are the ground for the practical purpose of mechanically performing navigation of space in relation to the 3D coordinate system of Formed, Z-axis is the third axis, perpendicular to the its ground.

  Solutions to practically achieve this positional orientation accurately and frequently as a function of time require a combination of sensors and software, and advances in these solutions include VR and AR hardware / software It represents the main trajectory in the development of the field of both mobile viewing devices and systems.

  These are relatively new areas in terms of time frame between early experiments and current practical technologies and products, and it is sufficient to note the origin and the latest technologies in both categories of mobile visual simulation systems The only exception is that it is important to the development of the present disclosure or it serves to better explain either the current problem in the field or the distinction of the solution of the present disclosure from the prior art. It is a particular innovation in the prior art that relates to differences or similarities.

  The period from 1968 to the late 90's is the period of many innovations in relevant simulation and simulator, VR and AR fields, during this period many of the important issues in achieving partial VR and AR But found the first or partial solution.

  The original and influential experimental and experimental head-mounted display systems of Ivan Sutherland and his assistant Bob Sprouell since 1968 are generally regarded as marking the origin of these related areas, In earlier work, essentially conceptual development precedes this by the first experimental implementation of any form of AR / VR that achieves immersiveness and navigation.

  The birth of the static simulator system dates back to the addition of computer-generated imaging to flight simulators, which is generally recognized to have begun in the mid to late 1960's. This was limited to the use of a CRT, which displays a fully focused image at a distance from the user to the CRT, but by 1972 the Singer-Link Company was able to First introduced a collimated projection system that projects the field of view to about 25-35 degrees for each unit (100 degrees for 3 units used in a single-seat simulator).

  This benchmark was first improved in 1982 with the wide-angle infinity display system that introduced a wide field of view system, and was refined by the Rediffusion Company, which used 150 degrees by using multiple projectors and large curved collimated screens. The FOV, and finally the FOV of 240 degrees was realized. At this stage, the static simulator is important for realistic immersiveness in virtual reality by using the HMD to isolate viewers and eliminate distracting visual cues from the surroundings It may be explained that the degree was finally achieved.

  However, as the Singer-Link Company introduced screen collimation systems for simulators as a stepping stone to VR-type experiences, the first very limited commercial helmet-mounted displays were being developed for military use for the first time, This integrated the reticle based electronic target system with the helmet's motion tracking itself. These initial developments were generally recognized as being achieved in a rudimentary form by the South African Air Force in the 1970s (following, from that point onwards to the mid-'70s by the Israeli Air Force), a rudimentary AR or It may be said that the initiation of an intermediary / hybrid reality system.

  These early, graphically minimalistic, yet inventive and influential helmet mount systems combine positionally adjusted target information superimposed on a reticle with user-actuated motion tracking targeting. There is an invention by Steve Mann of the first generation "EyeTap", the first "intermediate reality" mobile view-through system that implemented restrictive compounding and then superimposed graphics on the glasses.

  Later versions by Mann used a beam splitter / combiner light fusion system based on light fusion reality and processed images. After this study, there is a study by Chunyu Gao and Augmented Vision Inc, which basically proposes a dual-Mann system that optically combines the processed real image and the generated image, where the Mann system is Both electronically processed real images and generated images were achieved. While Mann's system preserves the real view through the images, all of Gao's system processes view-through images, eliminating direct view-through images even as an option. (U.S. Patent Application No. 20140177023 to Chunyu Gao, filed April 13, 2013). The "optical path folding optics" structure and method specified by Gao's system can also be found in other optical HMD systems.

  By 1985, Jaron Lanier and VPL Reseearch were formed to develop HMD and “Data Globe”, and by the 1980s, some of the most significant developments among the very active development areas Three major developments by Simulation, VR and AR by Mann, Lanier and Redefussion Company that have established several basic solution types that have received high praise and, in most cases, have survived to date as current technology A road existed.

  Refinement of computer-generated imaging (CGI), continuous improvement on game consoles (hardware and software) with real-time interactive CG technology, greater system integration among multiple systems, and AR and (more The expansion of both VR's mobility to a limited degree was included in the major development trends of the 1990's.

  Both the restricted form of mobile VR and the new kind of simulator was the CAVE system, which was developed in 1992 by the Electronic Visualization Laboratory at Illinois State University in Chicago and debuted in the world. (Carolina Cruz-Neira, Daniel J. Sandin, Thomas A. DeFanti, Robert V. Kenyon and John C. Hart, "The CAVE: Audio Visual Experience Automatic Virtual Environment (CAVE)", Communications of the ACM, 35 (6), 1992, 64-72)). Instead of the Lanier HMD / Data Grove combination, CAVE combined the WFOV multi-wall simulator "stage" with a haptic interface.

  At the same time, one form of the stationary part AR was developed by Louis Rosenberg at the Armstrong US Air Force Research Institute as the "Virtual Fixtures" system (1992), while Jonathan Waldern's stationary "Virtuality" VR system (1985) It is recognized that the first development was made early in the year from 1990 to 1990), and also made its commercial debut in 1992.

  A mobile AR that combines real and virtual vehicles with “enhanced simulation” (“AUGSIMM”) integrated into a multi-unit mobile vehicle “war game” system, a form of Loral WDL that was demonstrated in the industry in 1993 The next major step forward was seen. Then, in 1999, Peculiar Technologies project participant Jon Barrilleaux commented on the final 1995 SBIR report's findings, “Experiences and Observations in Applying Augmented Reality to Live Training And “Observations” were written, and the following contents, which are the challenges facing mobile VR and (mobile) AR, are described continuously until now.

  AR vs. VR tracking

  In general, the commercial products developed for VR have good resolution but lack the absolute accuracy and wide area coverage needed for AR and are less for use in AUGSIM.

  For VR applications where users immerse themselves in synthetic environments, relative tracking is more concerned than absolute accuracy. The user's world is completely synthetic, and it is more important to turn your head only 0.1 degrees than to know that you are currently pointing to true north, even within 10 degrees There is no contradiction with the fact that

  An AR system such as AUGSIM can not afford this. AR tracking must have good resolution so that virtual elements appear to move smoothly in the real world as the user's head rotates or the vehicle moves, and virtual elements overlap exactly AR tracking must have good accuracy so that it can be obscured by objects in the real world.

  Computational speed and network speed continue to improve during the 90's, and a new project on the outdoor AR system is launched, which is the project of BARS system at the US Naval Research Institute ("BARS: Battlefield Augmented Reality System (BARS: Battlefield) Augmented Reality System), Simon Julier, Yohan Baillot, Marco Lanzagorta, Dennis Brown, Lawrence Rosenblum; 2000 NATO Symposium on Information Processing Technology for Military Systems). The summary has the following description. "This system consists of a wearable computer, a wireless network system, and a tracked see-through head mounted display (HMD). The perception of the environment by the user is enhanced by superimposing graphics in the user's view This graphic is registered in the actual environment.

  Non-military development is also in progress, including ARToolkit, a study by Hirokazu Kato of Nara Institute of Science and Technology Graduate University, which will be announced later and further developed by HITLab, software development suite and viewpoint tracking and virtual object tracking Introduced the protocol for.

  These landmark events are frequently mentioned as being most important in this period, but other researchers and companies were also active in this area.

  Although military funding for large-scale development and testing of the AR for training simulation is well documented and evident that there was a demand for it, other system-level design and system demonstrations are also available. The military funded research effort was proceeding at the same time.

  Among the most important non-military experiments, development started and led by Bruce Thomas at the University of South Australia's Wearable Computers Institute is AR Quake, an AR version of the video game Quake, “ARQ uake: An Outdoor / Indoor Augmented Reality First Person Application (ARQuake: Outdoor / Indoor Augmented Reality First Person Application) ”(The 4th International Symposium on Wearable Computers, pp. 139-146, Atlanta, GA, October 2000; (Thomas, B Close, B., Donoghue, J., Squires, J., De Bondi, P., Morris, M. and Piekarski, W.) The abstract states: "We present a low-cost, appropriately accurate 6-DOF tracking system based on GPS, digital compass and tracking based on reference vision. "

  Another system, whose design development was launched in 1995, was developed by the author of the present disclosure. It was originally intended to realize a hybrid of the "Everquest Live" dubbed by outdoor AR and TV programs, and its design was further developed through the late 90's, the most important element being finished by 1999 At that time, a commercial effort to fund the original video game / TV hybrid was launched, and by that time, another version was included for use in the development of the finest theme resorts . By 2001, this is to companies including Ridley and Tony Scott, and in particular to their joint venture Airtightplanet (other partners including Renny Harlin, Jean Giraud, and the European Heavy Metal) Disclosed on a confidential basis, the authors of the present disclosure serve as executive supervision services for them, and at that time the “Otherworld” and “Otherworld Industries” projects and ventures at the same time, joint investment and collaboration with ATP It was submitted as a venture proposal.

  The following is an overview of the system design and components completed by 1999/2000.

  Excerpt from "OTHERWORLD INDUSTRIES BUSINESS PROPOSAL DOCUMENT" (Archive document version, 2003): Technical background explanation: State-of-the-art "Open Field" simulation and exclusive integration of mobile virtual reality: Tools, equipment And technology

  This is simply a partial list and outline of the relevant art that together form the backbone of a proprietary system. Some technology components are proprietary and some are from external vendors. However, a unique system combining proven components will be absolutely proprietary and innovative.

  Interaction with the VR modified world

  1) Mobile military standard VR equipment for immersing guests / participants and actors in OTHERWORLD VR extended landscapes. Their “adventures” (that is, their respective actions as they explore OTHERWORLD around the resort) are captured in real time (with automatic matting technology) by mobile motion capture sensors and digital cameras, guest / player and employee Members / Actors can see each other through their visors, with the superposition of computer simulation images. A visor is either binoculars (semi-transparent flat panel displays) or binoculars (but opaque flat panel displays with a binocular camera mounted on the front).

  These "composite elements" superimposed on the flat panel display in the field of view can include the altered portion of the landscape (or the entire digitally altered landscape). In fact, these "composite" landscape parts, which actually replace what is there, are generated based on "captured" the original 3D photos of each part of the resort. (Please refer to the following paragraph 7). As an accurate, photo-based geometric "virtual space" in a computer, it is possible to somehow digitally alter this virtual space while maintaining the photorealistic quality and geometric / spatial accuracy of the original capture It is. This produces an accurate combination of live digital photos of the same space and modified digital parts.

  Other "synthetic elements" superimposed by the flat panel display include computer-generated or modified people, creatures, atmosphere FX, and "magic". These appear as realistic elements of the field of view through the display (transparent or opaque).

  Positioning data, motion capture data of guests / players and employees / actors, and real-time matting of these data with multiple digital cameras (all of which are calibrated to a pre-captured version of each area of the resort (See Sections 4 and 5 below) allows the compositing element to be matched in real time with absolute accuracy to the real element shown through the display.

  Thus, a graphic computer-generated dragon can come round, fly back and fly up behind the real tree, and land on the resort's real castle, and the dragon It can "burn" with computer-generated fire. In a flat panel display (translucent or opaque), the fire appears to be coming out of the top of the "blackened" castle. Through the visor, this effect is achieved by the upper part of the castle being "over-matted" by a computer-modified version of 3D "capture" of the castle in the file of the system.

  2) Physical electro-optical mechanical gear for combat between real humans and virtual humans, creatures and FX. Motion sensors and other data, as well as "haptic" interfaces that provide vibration and resistance feedback, allow real-time interaction with real people, virtual people, creatures and magic. For example, a haptic device in the form of a "props" sword handle may have data during the swing of the guest / player and the guest / player poke off the virtual demon to achieve a combat illusion. Provide physical feedback when it is considered. All this is combined in real time and displayed through the binoculars flat panel display.

  3) Open field motion capture device. Mobile and fixed motion capture equipment tools (similar to those used in the movie "Matrix") are deployed across the resort's premises. Data points on themed "gears" worn by guests / players and employees / actors are tracked by cameras and / or sensors and virtual elements within the field of view displayed on binoculars flat panel in VR visor Provide motion data to interact with

  The output from the motion capture data is a CGI modified version of the guest / player and employee / actor in line with the character principle of Goram in the second and third parts of The Lord of the Rings. Enable (with sufficient computational rendering capabilities and use of motion editing and motion libraries).

  4) Enhancement of motion capture data with LAAS & GPS data, live laser distance measurement data and triangulation technology (including Moller Aerobot's UAV technology). The additional "positioning data" allows for even more effective (and error corrected) integration of live and synthetic elements.

  From a press release by a UAV manufacturer.

  July 17. One week ago a contract for the first network of Local Area Augmentation System (LAAS) stations was signed with Honeywell, and several test stations are already running. This system allows the aircraft to be guided to the airport (and vertical take-off and landing airports) accurately to the nearest inch accuracy. The LAAS system is expected to be operational by 2006.

  5) Automatic real-time matting of the open field "play". In combination with motion capture data that allows interaction with simulated elements, resort guests / participants are digitally imaged with a P24 (or equivalent) digital camera and interfaced with proprietary Automatte software Then, automatically separate (mat) the appropriate element from the field of view to be integrated with the synthetic element. This technique will be one of the package software used to ensure proper foreground / background separation when superimposing digital elements.

  6) Military standard simulation hardware and technology combined with cutting edge game engine software. Motion capture systems, haptic devices to interact with "synthetic" elements such as props, combining with data from synthetic and live elements (matted or complete), military simulation software and Integration with game engine software.

  These software components may be AI codes or "AIs such as Massive software used to animate the armies in the movie" Lord of the Ring "or" AI code for animating synthetic humans and creatures. "Artificial intelligence" software is provided to generate realistic water, clouds, fire etc. and unify and combine all elements in other ways just as computer games and military simulation software do.

  7) Photograph-based capture of the actual location (the basis of the "Brett Time" FX in "The Matrix") for creating realistic digital virtual sets with image-based technology, developed by Dr. Paul Debevec as a pioneer.

  The "base" virtual locations (inside and outside of the resort) are indistinguishable from the real world as they are derived from the photos and actual lighting of the location when they were "captured". A small set of high quality digital images combined with data from an optical probe and laser ranging data, and appropriate 'image based' graphics software, create a realistic virtual 3D space in the computer that exactly matches the actual one It is all you need to reproduce.

  The 'virtual set' is captured from the actual location inside the castle and the location outside the surrounding countryside, but once digitized these 'base' or default versions (the originally captured Add the elements that do not exist in the real world, modify the elements that exist to create fantasy versions of our world, and "decorate" with the lighting parameters and all other data at that time) Can be modified (including lighting).

  As guests / players and employees / actors pass “gateways” at various points in the resort (“gateways” are effective “passing points” from “our world” to “otherworld” ), Calibration procedure occurs. In order to "lock" the virtual space in the computer to the coordinates of the "gateway", positioning data from the guest / player or employee / actor at the "gateway" is then taken. The computer "knows" the coordinates of the gateway for the entire virtual version of the resort, acquired through the image-based "capture" process described above.

  Thus, the computer can "align" the virtual resort with what the guest / player or employee / actor sees before putting on the VR goggles. Thus, passing through the translucent version of the binocular flat panel display, one will match very closely to the other if the virtual version is superimposed on the actual resort.

  Alternatively, an "opaque" binocular flat panel display goggle or helmet would allow the wearer to walk confidently while wearing the helmet, looking only at the virtual version of the resort in front of him. Because the landscape of the virtual world will be in perfect agreement with the landscape he is actually walking.

  Of course, what can be shown to the wearer through goggles is a red sky that has been altered, a storm cloud that is not really there, and a dragon that has just "fired" just on the battlemented battlements of the castle. It would be the battlements of the castle that stopped at the top.

  And the army of 1000 ores to charge the distant hills!

  8) Resort super computer rendering and simulation equipment. An important resource that will enable the simulation of very high quality, near-future movie quality will be the supercomputer rendering and simulation complex of each resort scene.

  As with computer games for desktop computers, improvements to graphics and gameplay on stand-alone computer game consoles (PlayStation 2, Xbox, Gamecube) are well known.

  However, consider that such improvements in the gaming experience are based on improvements in the processor and support system of a single console or personal computer. And imagine that you put the power of the supercomputing center into the gaming experience. That alone will be a breakthrough in the quality of graphics and gameplay. And this is only one aspect of the mobile VR adventure that will be the Otherworld experience.

  It will be clear by reviewing the above, which should be obvious to those skilled in the relevant fields (more broadly, the fields of VR, AR, and simulation), but to improve the state of the art The individual hardware or software system proposed must take into account wider system parameters and clarify the assumptions about the system parameters to be evaluated properly.

  Thus, the essence of the proposal (the focus is on hardware technology systems that fall into the category of portable AR and VR technologies, and in fact both are fusions, but the most preferred version of that fusion is wearable technology And the preferred wearable version is HMD technology) is a perfect case to be an excellent solution by considering or reconsidering the whole system that is part of it. Thus, there is a need to present this history of larger VR, AR and simulation systems, because new HMD technology proposals and commercial offers have become too narrow, for example, system-level assumptions, requirements and It is because they tend not to consider or consider new possibilities.

  It is not necessary to review the major milestones in the evolution of HMD technology as well as historically. Because it is a help in explaining the limitations of the prior art of HMD and the current state of the prior art, and the reason for the proposed solution, and why the proposed solution solves the identified problem It will be necessary to provide an available framework, as it has a broader history at the system level.

  What is sufficient to understand and identify the limitations of the HMD prior art begins with:

  In the category of head mounted displays (including helmet mounted displays for the purposes of this disclosure), two major subtypes, "VR HMD" and "AR HMD" have been identified to date, which are According to the meaning of these definitions already provided in the specification, and within the category of AR HMDs, two categories are used to distinguish these types, “video see-through” or “optical see-through” (more Often referred to simply as "optical HMD").

  In the VR HMD display, the user looks at one panel or two separate displays. Although the typical shape of such HMDs is goggles or face mask shapes, many VR HMDs have the appearance of welder's helmets with bulky sealed visors. In order to ensure optimal video quality, immersion and no distraction, the system is completely sealed using a light absorbing material at the peripheral edge of the display.

  The author of the present disclosure has previously incorporated two types of US provisional application "SYSTEM, METHOD AND COMPUTR PROGRAM PRODUCT FOR MAGNETO-OPTIC DEVICE DISPLAY (System, Method and Computer Program Product for Magneto-Optic Device Display)". We proposed VR HMD. One of the two types of wafer type embodiments of the main object of the application merely proposes to replace a conventional direct view LCD, which is an embodiment of the present invention. The first practical magneto-optical display, the superior performance features of which are very high frame rates for improved display technology in general, and in this embodiment other advantages for the improved VR HMD. including.

  The second version, in accordance with the teachings of the present disclosure, contemplates a new type of remotely generated image display, which is generated, for example, in the vehicle cockpit, and subsequently transmitted via a fiber optic bundle, and a special fiber optic array. Based on the experience of fiber optic faceplates that are delivered through the structure (the structures and methods disclosed in the present application) and have new approaches and structures for remote image delivery via optical fibers.

  Initially the core MO technology for HMD was not commercialized, but rather for projection systems, these developments pertain to some aspects of the present proposal and are generally known in the art. Not. In particular, the second version discloses methods published prior to others, and more recent proposals are not integrated into the HMD optical body or not close to the HMD optical body Optical fibers are used to transmit video images from the imaging engine.

  An extremely important consideration for the practicality of a fully enclosed VR HMD for mobility beyond a tightly controlled stage environment with even floors is that the mobility is safe and the virtual world to be navigated Should perform 1: 1 mapping (within the safety deviation for human movement) on the real surface topography or motion trajectory.

  However, consistently by researchers such as Barrilleaux of Loral WDL, who is the developer of BARS, and by other researchers in the field over the last quarter century development about AR systems as a system that should be practical As observed and concluded, virtual (including synthetic CG generated images), including the shape of the vehicle in motion (not surprisingly from the development of a military system for urban warfare) A very close correspondence must be obtained between the world and the real world topography and construction environment.

  Thus, in order for either VR or AR to be possible in mobile form, there must be a 1: 1 positional correspondence between the "virtual" or combining elements and any real-world elements. Is the more common case.

  In the AR HMD category, the distinction between "video see-through" and "optical see-through" means that the user views directly as a transparent or translucent pixel array and display placed directly in front of the viewer as part of the optical glasses. , A translucent projection image on an optical element generated from a microdisplay (typically directly adjacent), transmitted through the form of an optical relay to the facing optical piece, also arranged in front of the viewer It is the distinction between looking through.

  A transparent or translucent display system, which is the main and perhaps only partially practical type of direct view-through display, has been (historically) an LCD configured without a lighting backplane. Thus, in particular, AR video view through glasses hold an optical screen that includes a transparent light substrate on which an LCD light modulator pixel array is fabricated.

  For applications similar to the original Mann's "EyeTap" (text / data is directly displayed or projected onto facing optics), calibration to real world terrain and objects is not required, but to some extent Positional correlation helps to contextually "tag" items in the field of view with informational text. This is the main purpose written for Google Glass products, but when drafting this disclosure, an AR type that a large number of developers superimpose more than text on live scenes Concentrated on the development of applications.

  The main problem with such "calibration" to terrain or objects in the field of view of users of either video or optical see-through systems, other than loose approximate position correlation in a roughly 2D plane or coarse view cone, is in the viewer environment. It is the determination of the relative position of an object. Calculations of significant discordant perspectives and relative sizes can not be made without reference and / or near real time spatial positioning data and 3D mapping of the local environment.

  In addition to relative sizes, an important aspect of perspectives from all perspectives is realistic lighting / shading (including drop shadows) depending on the direction of lighting. And finally, occlusion of the object from a given viewing location is an important optical feature of perceived perspective and relative distance and location.

  There is no video see-through or optical see-through HMD, or the perimeter of the wearer, which is essential for safe movement or path finding for either video or optical view-through type systems or indeed for mobile VR type systems It is not possible to design a video see-through or optical see-through HMD isolated from the problem of how such data is provided to enable dimensional viewing of the. Will such data be provided externally, locally, or in combination of multiple sources? If partially local and part of the HMD, would this affect the overall design and performance of the HMD system? Among other implications and affected parameters in the choice between video see-through and optical see-through, if this problem is to be influenced, a given weight, balance, bulk, data processing requirements, among components What effect does it have on the details of the lag, and the choice between display and optical components?

  Among the technical parameters and problems to be solved as the VR HMD evolves and advances, mainly the increase of the field of view, the reduction of latency (change of lag between motion tracking sensors and change of virtual perspective), resolution, frame rate There has been an increase in dynamic range / contrast, as well as other common display quality features, as well as weight, balance, bulk and general ergonomics issues. Image collimation and other display optics details have been refined to effectively address the "simulator disease" problem that has been a major issue from the outset.

  The weight and bulk of displays, optics and other electronics tended to decrease over time, with improvements in these general technology categories, and weight, size / bulk and balance.

  While stationary VR gears have been commonly used for night vision systems in vehicles (including aircraft), mobile night vision goggles can be considered as a form of mediating viewing similar to mobile VR. This is because basically what the wearer is looking at is the real-time real scene (infra-red imaged) (but through the video screen, not in the form of a "view through").

  This subtype is similar to what Barrilleaux defined as "indirect vision display", retrospectively to 1999 also mentioned. He proposes that there is no actual "view through" but what is visible is an exclusively fused / processed real image / virtual image on the display, perhaps included as any VR type or night vision system The definition is presented for the selected AR HMD.

  However, the night vision system actually, but rather directly transmits, the infrared sensor data as interpreted through video signal processing, as a virtual composite landscape and as a monochromatic image of different intensity depending on the intensity of the infrared signature. It is not a fusion or mixing with a video image. As a video image, this is real-time text / graphics, in the same simple form as Eyetap was originally conceived, and, as Google stated, the main purpose intended for Glass products Suitable for overlays.

  How and what data will be extracted live, or provided from reference to either mobile VR or mobile AR system, or both, or consistently queue out to provide a combined view The question is whether to include now the "indirect view" of this hybrid live processed video feed that has similarities with both categories to enable effective integration of the virtual landscape and the actual landscape. Regardless of type, design parameter, and type, is a problem that must be considered in designing any new and improved mobile HMD system.

  Software and data processing for AR have been advanced to address these issues, based on the earlier work of the system developers mentioned above. As an example of this, Matsui and Suzuki of Canon Inc. disclosed in the pending US patent application “Mixed reality space image generation method and mixed reality system” (Mixed reality space image generation method and mixed reality system). There is a study of

  (U.S. Patent Application Publication No. 20050179617, filed on September 29, 2004) The abstract includes the following.

  “A mixed reality space image generating apparatus for generating a mixed reality space image formed by superimposing a virtual space image on a real space image obtained by capturing a real space An image combining unit (109) for superimposing a virtual space image displayed in consideration of occlusion by an object in space into a real space image, and an image displayed without considering the occlusion of the virtual space image In this way, a mixed reality space image can be generated that can achieve both natural and convenient displays. "

  The purpose of this system is intended to enable the combination of fully rendered industrial products (such as cameras) to be superimposed on the mock-up (optical props), and the optical view through Both HMD glasses and mock-ups are equipped with position sensors. A real-time method for matting pixels from the mockup so that CG-generated virtual models can be superimposed on composite video feeds (buffered to allow layering with slight lag) A pixel-by-pixel search comparison process is used. Annotation graphics are also added by the system. Computer graphic. The most important sources of data to determine the matting, and thus ensure accurate and not false shielding in the composition, are the motion sensors on the mockup, and for drawing the handmat and the mockup mat It is a pre-determined lookup table that compares pixels.

  This system does not help to generalize mobile AR, VR, or any hybrid, but it is simple to analyze real 3D space and position virtual objects properly in perspective view It is an example of an attempt to provide a system (although not fully automatic).

  In the area of video or optical see-through HMDs, even with the ideal calculated mixed reality perspective view delivered to the HMD, satisfying real and accurate fused perspective views (handling of the proper order of perspectives, in real space There has been little progress in the design of displays or optics and display systems that can implement the appropriate shielding of the fusion element from the given viewer position.

  One system that is claimed to be the most effective (even partial) solution to this problem, and perhaps only one integrated HMD system (independent of the HMD, some generic aspect (In contrast to software / photogrammetry / data processing and delivery systems, which were designed to solve these problems), but in the previously described US patent application No. 20140177023 "APARATUS FOR OPTICAL SEE-THROUGH HEAD MOUNTED DISPLAY Chunyu Gao in WITH MUTUAL OCCLUSION AND OPAQUENESS CONTROL CAPABILITY (A device for see-through head-mounted display with mutual shielding and opacity control capability) It has been mentioned in the proposal.

  Gao begins his overview on the field of view-through HMDs for AR from the following point of view.

  There are two types of ST-HMD, optical and video (J. Rolland and H. Fuchs, "Optical versus video see-through head mounted, displays", In Fundamentals of Wearable Computers and Augmented Reality (on the basis of wearable computers and augmented reality), 113-157, 2001). The main drawbacks of the video see-through approach include the degradation of the image quality of the see-through view, the image lag due to the processing of the incoming video stream, the possibility of the loss of see-through view due to hardware / software malfunction. Also, in contrast, optical see-through HMD (OST-HMD) provides real-world direct view through beam splitters, thus the impact on real-world views is minimal. It is highly suitable when requesting an application where the user's awareness of the live environment is paramount.

  However, Gao's view of the video see-through problem is not qualified in the first case by exclusively identifying the prior art video see-through as an LCD, and he also And there is no proof of claim that it must degrade the see-through image (which is omitted on any basis). One skilled in the art may recognize that this poor quality image view is derived from the results achieved with earlier view-through LCD systems prior to the recent accelerated progress in the field. . Optical see-through as compared to state-of-the-art LCD or other video view-through display technologies, using many optics as a comparison and the influence of other display technologies on "real" "see-through image" reprocessing or mediation It is neither virtually true nor obvious that the system relatively deteriorates the final result or is inferior to the proposal such as Gao.

  Another problem with this groundless generalization is the estimation of lag in the see-through category as compared to other systems that must also process the input live image. In this case, the speed comparison is the result of a detailed analysis of the competing system's overall components and their performance. And finally, the speculation of "the possibility of the loss of see-through view to hardware / software" is between the video and the optical see-through scheme in general, or between and any particular version of its component technology and system design A rigorous analysis of the robustness or stability of the comparison system, either in principle, is basically unfounded, arbitrary, and not effective.

  Beyond the first issue of comparative flawed and biased assertions in the field concerned, the solutions they have proposed themselves are qualitative, together with the data acquisition, analysis and distribution issues already mentioned and addressed. There is a problem (including omission and omission of considering the proposed HMD system as a complete HMD system (also included as a component in a wider AR system)). HMD is not permitted to handle data or processing power of a particular level and quality for the generation of modified or mixed images as "known fact", in which case HMD itself is And important questions and problems that can assist or impede the design, and can not be simply presented as known facts.

  Furthermore, omitted from the task and solution detailing is the full range of real and virtual visual integration problems on the mobile platform.

  The disclosure and the system that the disclosure teaches are particularly as follows.

  As mentioned above in this context, Gao's proposal is to use two display-type devices, because the specifications of spatial light modulators that selectively reflect or transmit live images are fundamental Indeed, this is because they are specifications of the SLM for the same purpose as being operable in any display application.

  And while the output images from the two devices are lined up on a pixel-by-pixel basis, they are combined within the beam splitter combiner (assumed) without any specific explanation other than the description of the accuracy of these devices Be

  However, in order to achieve this fusion of two pixelated arrays, Gao specifies an overlap of what he calls "folding optics" but basically two "folding optics" in total Elements (eg, grating planes / HOE or other compact prisms or "flat" optics), one for each source, and two objective lenses (one for the wavefront from the real scene and one for the combined image) And at the other end of the focal point of the beam splitter combiner, nothing is required other than the double version of the Mann Eyetap scheme.

  Thus, a large number of optical elements, for which he presents a variety of conventional optical variations, 1) a first reflective / folded optical body (planar grating / mirror, HOE, TIR prism, or Collect the light of the actual scene through the other "flat" optics and from there to the objective lens and pass the light to the next grating / mirror, HOE, TIR prism or other "flat" optics To re-fold the light path, all of which are to ensure that the entire optical system is relatively compact and is included in the approximate set of two rectangular optical relay areas. It is a thing. From the folding optics this beam is passed through a beam splitter / combiner to the SLM. The SLM is then reflected or transmitted based on pixelation (sampling), and thus variably modulated (changes from real image contrast and intensity to correct gray scale etc), and now passes through the pixelated real image Let it go back to the beam splitter / combiner. While the display synchronously generates virtual or composite / CG images, such images are also possibly calibrated to ensure easy integration with the corrected pixelated / sampled real wavefront, in multiple steps The beam splitter is passed through to integrate pixel-by-pixel with the samples of the corrected and pixelated real scene, from there through the eyepiece objective, and back to another "folding optics" element, the viewer Is reflected out of the optical system to the eye.

  As a whole, for the real image wavefront of the corrected, pixelated and sampled part, it passes through seven optical elements (not including the SLM) before reaching the viewer's eye. The composite image generated by the display simply passes through two optical elements.

  The problem of accurate alignment of the optical image combiner, ie, at the pixel level, whether it is reflected light collected from the image sample queried by the laser, or the combination of the images produces a small-featured SLM / display The question of whether or not it is considered that maintaining alignment, especially under conditions of mechanical vibration and thermal stress, is not considered trivial in the art.

  Digital projection free space light beam mixing systems, which combine the output of high resolution (2k or 4k) red, green and blue image engines (typically images generated by DMD or LCoS SLMs) are expensive and these It is not easy to achieve and maintain alignment. There are also simpler designs than in the case of the 7 element obstacles of the Gao scheme.

  Furthermore, these complex multi-engine multi-element optical combiner systems are not nearly as compact as needed for HMDs.

  Monolithic prisms such as the T-Rhomboid combiner, developed and marketed by Agilent for the life sciences market, have been developed specifically to address the problems that free space combiners have shown in existing applications.

  And although Microvision and other companies have successfully deployed HMD platforms originally developed for microprojection technology based on SLM, these optical setups are typically more substantial than Gao's proposal Less complicated.

  In addition, what is the basic rationale for the two image processing steps and calculation iterations on the two platforms, and combined to achieve smoothing and integration of the real and virtual wavefront inputs It is difficult to determine why it is needed to perform proper occlusion / opacity of scene elements. The greatest concern of Gao and the problem to be solved is the problem of synthetic images that compete with difficulty with the brightness of the real image, so the main task of the SLM is the real scene part or the real scene It seems to be to selectively reduce the overall brightness. In general, for example, while reducing the intensity of the shielded actual scene element by minimizing the duration of the DMD mirror at the reflecting position in the time division multiplexing system, the shielded pixels are simply It is also speculated that it will be left as it is, but this has not been specified by Gao, nor has it specified the details of how SLM achieves its associated image modification function .

  Among the many parameters that should both be calculated, calibrated, and registered, it is included to determine which pixel from the real field is the one that has been calibrated for the composite pixel. Without perfect agreement, ghost overlap and misalignment and occlusion especially doubles in moving scenes. The position of the reflective optics which causes the wavefront part of the actual scene to pass through the objective has an actual perspective position with respect to the scene which is not initially identical to the perspective position of the viewer in that scene. It is not a plane, it is not located in the middle, and it is merely a wavefront sample, and not a position. Furthermore, if it is mobile, even moving, and not previously known to the composite imaging device. The number of variables in this system is very large, just for the reasons given above.

  If so, and if the purpose of this solution is made more specific, a second display (a total of two displays in the binocular system, ie the specified SLM is added to achieve this) It may become clear that there may be a simpler way than using).

  Second, either approach durability of such composite systems with multiple accumulated alignment resistances, accumulation of defects from multi-element path original parts and aging wear, non-alignment of fused beams The reason is that if it causes accumulated thermal and mechanical vibration effects, seven other elements and additionally other problems arising from the complexity of the optical system, it is likely that the external live image wavefront is essentially degraded It is this system that brings about (especially with age).

  Additionally, as discussed in greater detail above, the problem of computing the spatial relationship between real and virtual elements is not a trivial problem. The design of a system that must drive a four-display type device that may be two (and in a binocular system) possibly different types (and thus different color gamuts, frame rates etc.) based on this calculation has already required It adds complexity to stringent system design parameters.

  In addition, high frame rates are essential to deliver high-performance images without ghosting or lag and without inducing eyestrain or visual system fatigue. However, in the Gao system, the system design is slightly more simplified with the use of only a view-through rather than a reflective SLM, but even with faster FeLCoS microdisplays, the frame rate and image speed are still TT DLP Substantially lower than the frame rate and image rate of MEMS devices such as (DMD).

  However, as higher resolutions are desired for HMDs (to achieve at least a wider FOV), resources for high resolution DMDs such as TT 2k or 4k devices imply very expensive solution resources . Because their feature size and number of DMDs are lower yield, higher defect rates that can typically be accepted for mass consumer or business production and cost, the systems in which they are currently used (TI OEM It is known to have very high price points for digital cinema projectors (commercially marketed by Barco, Christie, and NEC).

  Flat optical projection technology for optical see-through HMDs like Lumus, BAE and others (where shielding is not a design goal, nor is it possible within the scope and capabilities of these approaches), basically It is an instinctively easy step to proceed to duplicate this approach and adjust the real image, and then combine the two images using the conventional optical setup as proposed by Gao. It relies on a large number of flat optics to provide a combination and to do so in a relatively compact space.

  Returning to the current leaders in the two general categories of HMDs, optical see-through HMD and classical VR HMD, to conclude the outline of the background, the outline of the current state-of-the-art is as follows: The point is that the other variants, optical see-through HMD and VR HMD, are both commercially available and are the subject of intensive research and development, and are essentially Google, Glass, and Oculus VR HMD, the Rift A great deal of commercial and academic research has been carried out, including product announcements, publications and patent applications, which have since grown to a breakthrough.

  At the time of writing this document with Glass, a commercially major mobile AR optical HMD, Google has established a breakthrough general awareness and major marketing position in the optical see-through HMD category.

  However, they followed others on the market (including Lumus and BAE (including Q-Sight holographic waveguide technology)) which has already developed and launched products mainly in the defense / industrial sector. Other companies that have entered the recent market and research stages include TruLife Optics, a study from the UK National Physical Reality that commercialized research in the area of holographic waveguides, and they are relatively Insist on superiority.

  For many military helmet-mounted display applications, and for the official main application case for Google's Glass (again, as analyzed above), text to view space and only rough position correlation is needed Superimposition of symbolic graphic elements may be sufficient for many early simple mobile AR applications.

  However, even in the case of information display applications, the higher the density of information tagged with items and terrain in the view space facing the viewer (and ultimately surrounding the viewer), the spatial order of the tags / It is clear that the need for layering to match the perspective / relative position of the tagged element increases.

  Thus, overlap, ie partial shielding of the tag by real elements in the field of view (not just the overlap of the tag itself), necessarily “basic” information display to manage visual clutter Even the desired optical view-through system is the requirement.

  Tags are also not only relative position of tagged elements in the perspective view of real space, but also automated priorities (based on pre-determined or software-calculated priorities) and real-time users Tag size and transparency (specifying the only two major visual cues used by the graphics system to reflect the information hierarchy), as it must reflect both degrees of priority assigned. Must be managed and implemented.

  The question that arises immediately is the basic optical see-through HMD (monocular reticle type or binocular all-glass type), after carefully examining the issues of translucency and overlap / masking of tags and superimposed graphic elements. The relative brightness of the live elements and superimposed generated video display elements (in particular with brightly lit outdoor lighting conditions and very dimly) How to deal with the problem of the illuminated outdoor condition). In order to fully extend the usefulness of these display types, nighttime use is clearly an extreme case of the low light problem.

  Thus, moving beyond the conditions of the most restricted use cases of passive optical see-through HMD types, as the information density increases (such systems are commercially successful and usually dense urban or suburban areas This is expected to happen if the tag gets information from a for-profit business), and "passive" optical see-through HMD mobiles as the light and dim use parameters add to the constraints It is apparent that neither the problems nor the needs of the realistic and practical implementations of AR HMDs can be addressed or addressed.

  So we have to consider passive optical pass-through HMD as an imperfect model to realize mobile AR HMD, and it will only be seen as a temporary footing to an active system when looking back at it later It will be.

  Oculus Rift VR (Facebook) HMD: somewhat parallel to the impact of the Google Glass product marketing campaign, but Oculus solves some of the barriers at the critical starting point of practical VR HMDs and / or substantive There is a difference that it was actually going ahead in the field (in the case of Google, rather than following Lumus and BAE), but the Oculus Rift VR HMD At the time of writing, it is the major pre-mass release VR HMD product that enters and builds the market for widely recognized consumer and business / industrial VR.

  An overview of the advancements at the basic starting point of the Oculus Rift VR HMD can be summarized in the following list of product features.

  o Achieved using a single 7 pw display at 1080p resolution, positioned a few inches from the user's eye and divided into binocular perspective areas on a single display, significantly An expanded field of view. The current FOV is 100 degrees in this document (compared to its original 90 degrees), compared to 45 degrees, which is a general specification of the HMD previously present. Separate binocular optics provide stereo vision effects.

  o Significantly improved head tracking and resulting low lag. This is an advance on improved motion sensors / software, among other handheld and handheld device products with built-in motion sensors (accelerometers, MEMS gyroscopes etc) for 3D position tracking, Nintendo Wii, Apple and It uses miniature motion sensor technology that has moved away from other mobile phone sensor technologies, Playstation PSP (now Vita), Nintendo DS (now 3DS), and the Xbox Kinect system. Current head tracking implements multipoint infrared optical systems using coordinated external sensors.

  o Low latency. This is the result of a combination of improved head tracking and a fast software processor that updates the interactive game software system, but with the display technology used (originally an LCD, with slightly faster OLEDs Limited by its inherent response time).

  o Low sustainability. This is a form of buffering to help keep the video stream smooth, working in combination with faster switching speed OLED displays.

  o Lighter weight, reduced bulk, better balance, and totally improved ergonomics by using ski goggle form factor / material and machine platform.

  Summarizing the net benefits of combining these improvements, such a system may not be structurally or operationally new as a pattern, but with improved components and a particularly effective US Design Patent No. D 701,206, as well as the net effect of any proprietary software, resulted in breakthrough levels of performance and verification of mass market VR HMDs.

  In many cases following these precedents and adopting that approach, and in the case of others who have modified their design based on the success of the Oculus VR Rift configuration, a few have occurred at the same time Product programs, and there are many VR HMD product developers (both branded companies and start-up companies), they are the original 2012 Electronic Expo demonstrations and kickstarter financing campaigns by Oculus VR After that, we made a product plan announcement.

  Among those who followed quickly and those who apparently modified their strategies according to the Oculus VR template were Samsung. Samsung demonstrated the design of the Oculus VR Rift and a development model very similar to Sony's Morpheus at the time of this writing. Emerging companies that have emerged in this area include Vrvana (formerly True Gear Player), GameFace, InfiniteEye, and Avgant.

  None of these system configurations appear to be exactly the same as the Oculus VR, but some use two panels, others use four panels, and the InfiniteEye, according to its claim, is FOV up to 200+ degrees. Used a four panel system to expand the Some use LCD and others use OLED. Optical sensors have been used to improve the accuracy and update rate of head tracking systems.

  All of these systems are implemented because of their essentially fixed location or highly constrained mobility. These make use of on-board and active optical marker based motion tracking systems designed to be used in a living room or enclosed space such as a surgical classroom or simulator stage.

  The systems that differ the most from the Oculus VR scheme are Avent's Glyph and Vrvana Totem.

  The Glyph actually implements a previously established optical view-through HMD solution and structure-based display solution using the Texas Instruments DLP DMD to generate a micro image projected onto a reflective planar optical element. And the same configuration and operation as the planar optical elements of the existing optical view-through HMD, except that a high contrast light absorbing backplane structure is used to realize the reflective / indirect micro-projector display type. Video images belong to the general category of opaque, non-transparent display images.

  However, as demonstrated here above in the discussion of the Gao disclosure, limitations when enhancing display resolution and other system performance beyond 1080p / 2k, when using DLP DMD or other MEMS components Is the cost, manufacturing yield and defect rate, durability and reliability of the system.

  Furthermore, the limitation to image size / FOV from limited expansion / magnification of planar optics (grating structure, HOE or other) (which extends SLM image size but human vision system (HVS) (especially focusing system) Interactions / burdens)) present limitations on viewer safety and comfort. The user's reaction to using similar sized but lower resolution images in the Google Glass test is brighter at higher resolutions, but the additional burden on HVS with equally small image areas poses a challenge for HVS Is suggested. Eli Peli, a doctor at Ophthalmologist, Google's official consultant, told Google Glass users in an interview with the online site BetaBeat a previous warning (May 19, 2014) that predicted eyestrain and discomfort It was followed up and called for a revised warning (May 29, 2014) to limit the cases and scope of potential use. The demarcation is about the eye muscles used in an unintended manner or used for a long time, and this near cause in the revised statement has forced the user to look up It was a small display image position. Other experts

  However, assume that the specific combination of use of the eye muscles necessary for focus use on small parts of the actual FOV is identical to that required for eye movement throughout the actual FOV. I can not do it. In fact, the small tweaking of the focal muscles is more limited / constrained than the range of motion associated with a natural FOV scan. Thus, the repetitive movement of the contractile ROM is not limited only to the focus direction as is known in the art, but the nature of the HVS adds excessive tension beyond normal use, Still further, it is expected to add motion range constraints and the need to make very small controlled fine adjustments.

  The added complexity is that eye strain quickly outpaces precision tooling as the level of detail in the area of constrained eye movement increases in resolution with scenes with complex detailed movement It is. No exacting treatment for this problem has been reported by any of the developers of the optical view-through system, and these problems and over many years with Steve Mann's use of his EyeTap system Reported eye strain, headache and dizziness problems (The problem has been reported to have been partially improved in the latest Digital EyeTap update by moving the image to the center of the field of view but systematic No research has been done), with only limited comments focusing on only a small fraction of the problems and issues of eye strain and “computer vision disease” that can result from precision work.

  However, the limited public comments that Google has made available to Dr. Peli generally suggest that Glass as an optical view-through system is intended for occasional use rather than long-term or high frequency viewing And repeatedly insist.

  Another way to understand the Glyph scheme is to use, at the highest level, a variation of the embodiment for light-split VR operation according to the Mann Digital EyeTap system and the structural arrangement, and the latest optical view through system Side projection deflection optical setup.

  In Vrvana Totem, departure from the Oculus VR Rift is a binocular to allow switching between forward captured image captured on the same optically covered OLED display panel and generated simulation. By adding a conventional video camera, it is to adopt the "indirect view display" scheme of Jon Barrilleaux. Vrvana has shown marketing the material to realize this very basic "indirect viewing display" for AR, just according to Barrilleaux's specified schemes and patterns. While at least affecting the weight and balance of the HMD, it is clear that virtually any other VR HMD of the current Oculus VR generation can be attached to such conventional cameras.

  From the above it is clear from the above content that in the category of "Video See-through HMD" or generally in the field of "Indirect Vision Display" there is little or no progress over the category of night vision goggles I will. Night vision goggles are well developed as a subtype, but within the scope of video processor methods known in the art, any other than providing adding text or other simple graphics to live images It lacks AR features.

  Furthermore, with respect to the existing limitations to VR HMDs, all such systems using OLED and LCD panels have relatively low frame rates, which are motion lag and latency, and this is a broad category of "simulators" Contribute to negative physiological effects on some users who belong to Also, with digital stereo projection systems for movies using such commercially available stereo systems such as RealD systems realized for projectors based on Texas Instruments DLP DMD or projectors based on Sony LCoS, the frame rate too high is also insufficient. Also note that a small percentage of the audience (about 10% in some studies) has also been reported to contribute to experiencing headaches and related symptoms. Some of these are specific to the relevant individual, but a significant percentage of this can be attributed to frame rate limitations.

  And further, as noted, Oculus VR's "buffering system in the pad to compensate for the still inadequately high pixel switching / frame rate of the OLEDs used at the time of this writing" Is realized.

  A further impact on the performance of existing VR HMDs is due to the resolution limitations of existing OLED and LCD panel displays, this resolution limitation allows for a 5-7 inch diagonal display to achieve a sufficiently effective resolution. Which is part of the requirement to attach them using and slightly away from the viewing optics (and the viewer's eyes) and are considerably larger, bulkier and heavier than most other optical headwear products Contribute to the bulk, size and balance of existing and planned offerings.

  Potential partial improvements are expected from adopting curved OLED displays that can be expected to further improve the FOV without adding bulk. However, the cost to market a sufficient quantity that requires significant additional investment in the capacity of a production plant with acceptable yields makes this outlook impractical in the short run. Do. Therefore, the problem of bulk and size is only partially addressed.

  For completeness, point out a video HMD employed to view video content without the ability to navigate the virtual or hybrid (mixed reality / AR) world without being interactive or with any motion sensing capabilities You also need to Such video HMDs have been fundamentally improved over the past 15 years to enhance effective FOV and resolution and viewing comfort / ergonomics, and to be utilized and developed by the latest VR HMDs Provided development paths and advances that have become possible. However, these too have been limited by the core performance of the adopted display technology, with patterns according to the limitations observed for OLED / LCD / DMD based reflection / deflection optical systems.

  Other important variations on projection images in the transparent eyewear optical paradigm include those by Osterhoudt Design Group, Magic Leap, and Microsoft (Hololens).

  These variations bring some relative advantages or disadvantages to each other and to the other prior art discussed in detail above, all of which hold the limitations of the basic approach.

  Commonly more fundamental and even universal, as a frame rate / refresh of existing core display technology, whether high speed LC, OLED or MEMS, and also transferring display images to viewing optics These are also limited by the basic type of display / pixel technology employed, whether or not they employ mechanical scanning fiber inputs or other optical systems disclosed for High quality, eye-friendly (HVS), low power, high resolution, high dynamic range and other display performance parameter requirements are yet to be met, contributing separately and jointly to the realization of AR and VR enjoyable in quality It is enough.

  A summary of the state of the art with the details described above is as follows.

  "High vision" VR has improved in substantially many points: FOV, latency, head / motion tracking, lightness, size and bulk.

  However, the frame rate / latency and resolution, and to the extent that result as a matter of course, the weight, size and bulk are limited by the limitations of core display technology.

  • And modern VR is constrained to stationary or very constrained and restricted mobile applications in small controlled spaces.

  VRs configured as a side projection deflection system based on an enclosed version of an optical view-through system but where the SLM projects the image onto the eye through a series of three optical elements are limited in the performance of the size of the reflected image, This reflected image is magnified but not significantly larger than the output of the SLM (DLP DMD, other MEMS, or FeLCoS / LCoS) compared to the total area of a standard spectacle lens. The very intense version of extended viewing of the "close-up work" and the risk of eyestrain from this which is required of the eye muscles is an additional limitation on practical acceptance. And, SLM type and size displays also limit the practical path to improved resolution and overall performance due to the scaling cost of higher resolution SLMs of the mentioned technology.

  Optical view-through systems generally have the same potential for eyestrain due to the limitation of use to relatively small areas of the eye muscle, and are relatively fine for these constraints and for cases exceeding a short period of use And often require adjustments to track the eye. Google Glass was designed to reflect the prospect of using a limited duration by positioning the optics above and outside the direct rest position of the eye looking straight ahead. However, the user has nevertheless reported eyestrain, as it has been extensively documented in the publication by texts and interviews from Google Glass Explorers.

  Optical view-through systems are limited in the density of superimposed translucency information due to the need to organize tags to real-world objects in perspective views. The need for mobility and information density makes passive optical view-through limited even for graphic information display applications.

  -Aspects of "indirect vision display" have been realized in the form of night vision goggles, and Vrvana, a competitor of Oculus VR, only suggested the adaptation of Totem with a binocular video camera for AR. is there.

  ・ Gao's proposal. It is claimed to be an optical view-through display, but in fact it has a pseudo-view-through aspect due to the use of an SLM device, rather a "indirect view display", which is part of the actual wavefront Are modified for the projection display to function as such, in order to sample and modify portions of the wavefront digitally.

  The number of optical elements involved in the light routing of the first wave front portion (and the point to add here is much smaller (7 or nearly so) than the optical area of the conventional lens of the conventional glasses) ) Provides the opportunity for both screen aberration, image distortion, and loss, but complex free space alignment of many such elements is not common and they are needed, expensive and expensive If it is difficult and not robust, a complex system of optical alignment is required. Neither the method used when SLM is expected to manage the real scene modifications nor the identification or verification of its specific requirements. The problem of coordinating signal processing between two to four types of displays (depending on the monocular in a binocular system) has not been identified or verified, and the problem is that real and synthetic elements in a perspective view Especially in situations where it is already a very demanding requirement to make calculations to create a proper relationship between them, especially when the individual is moving in a densely populated, geographically complex environment It involves determining which pixels from the real field are specifically calibrated pixels for the proper composition. Mounting on vehicles merely exacerbates this problem.

  There are numerous additional problems in developing a complete system, even with the task of constructing an optical set proposed by Gao, or even reducing it to a relatively compact form factor. The size, balance, and weight are just one of the various processing and number of optical body array units and many results that will be (implicitly) required, but other issues and limitations mentioned It is relatively minor compared to. However, it is serious to actually deploy such a system for field use (either for military use or for ruggedized industrial use or consumer use).

  100% "indirect vision display" will have similar needs at key points regarding Gao's proposal (details of the number of display type units, alignment, optical systems, pixel and system matching, and perspective issues) Except that all important parameters of such a system require a "round-robin" calculation of synthetic CG3D mapped space stored in concert with individual real-time perspective real-time view-through images Throw a question about the degree of The problem is that for video images captured by forward video cameras, with the basic Barrilleaux, and now with the possible Vrvana design, all the calculations are done, non-local (HMD and / or Or to the wearer himself) to a degree that must be relayed to the processor.

  What is needed (whether VR or AR) for a truly mobile system that achieves both immersiveness and calibration in a real environment is as follows.

  Ergonomics and viewing systems that minimize non-regular demands on human vision systems. This is to allow for more extended use, which is implied by mobile use.

  -A wide FOV of 120 to 150 degrees (ideally including the peripheral vision).

  High frame rates (ideally 60 fps / eye) to minimize latency and other image artifacts typically caused by the display.

  -High effective resolution at a comfortable distance from the face to the unit. The effective resolution standard that can be used to measure the maximum value is the effective 8k or "retinal display". This distance should be similar to that of conventional glasses, which generally use the bridge bridge as the balance point. Collimation and optical path optics are needed to establish a proper virtual focal plane that also achieves this effective display resolution and the actual distance of the optical element to the eye.

  High dynamic range that matches as closely as possible the dynamic range of the live scene.

  On-board motion tracking to determine both head and body orientation on known terrain (whether known in advance or just in time within the wearer's field of view). This may be supplemented by an external system in a hybrid scheme.

  A display optics system that enables a fast synthesis process within the framework of the human vision system between the wavefront of the real scene and any synthesis element. Many passive means should be used to minimize the burden on any of the on-board (HMD and wearer) and / or external processing systems as much as possible.

  • A relatively simple and crude device with only a few optical elements, only a few active device elements, both minimum weight and thickness, and a simple active device design that is robust under mechanical and thermal stresses Display optics system.

  -Suitable for design configurations known to be acceptable to both professional users, such as lightweight, low bulk, balanced in gravity, and military and durable environmental industrial users, and sport applications, as well as general consumption and A form factor that makes business applications more durable. This is recognized by the factor range from eyeglass manufacturers such as Oakley, Wiley, Nike, and Adidas to Oakley, Adidas, Smith, Zeal and some other more specialized sports goggle manufacturers.

  -A system that is switchable between a VR experience while maintaining full mobility and a variable shielding, perspective integrated hybrid viewing AR system.

  -A system capable of managing the wavelength to be received for the HVS and acquiring valid information from the target wavelength (via the sensor) and these hybrids. Infrared, visible and UV are typical wavelengths of interest.

  The system proposed by the present disclosure solves the problems of the functionality in both augmented and virtual reality, tasks and criteria that the prior art was fundamentally limited and inadequate, and the ultimate goal for these Achieve.

The present disclosure is directed to telecommunications structured and pixel signal processing systems and hybrid magnetic photonics (pending US patent application [2008] by the same inventor. And photonic encoders Of the same inventor pending US patent application Incorporate and implement with preferred pixel signal processing subtypes of “Hybrid MPC Pixel Signal Processing, Display and Network”. Addressing and powering (especially of the array) of the device is a pending US patent application Addressing and powering of "Wireless Addressing and Powering of Arrays", a preferred embodiment of the hybrid MPC type system is a pending US patent application It can also be found in "3D fab and systems therefrom".

  This application is incorporated by reference in its entirety.

  However, while establishing system type categories and important subsystem categories, and preferred versions and embodiments of the subsystem, it is included in the application to which all the details of the present proposal are referred to, and the present application Is not simply stated as a combination of these systems, structures and methods.

  Rather, the proposal proposes new and improved systems and subsystems that fall into most or often these mentioned (and generally new) categories and classes, their components, systems, subsystems , Together with the detailed disclosure of structure, process, and method, while the unique combination of these and other component classes also enables unique new types of mobile AR and VR systems The preferred embodiment of the wearable system as a head mounted wearable system is the most preferred.

  The description of the proposed system starts with taking out the major subsystems (in order) and organizing the overall structure and manipulation structure, and then provides details of these subsystems in a hierarchical overview form Is the best.

  The main subsystems are as follows.

  I. Telecom system type architecture for displays with pixel signal processing platform and suitable hybrid MPC pixel signal processing (including photonic coding system and apparatus).

  II. Sensor system for mobile AR and VR

  III. Structural and substrate systems

  Implemented by these major subsystems is a novel integrated dual "generative" and variably directly transmittable direct view hybrid display system.

  I. Telecom system type architecture for display with pixel signal processing platform and suitable hybrid MPC pixel signal processing (including photonic coding system and apparatus)

  The purpose of the present disclosure is to process sensor data (especially in real time) and to compute computer generated images and integrate 3D perspective views of real and synthetic / digital or stored digital image information The use of passive optical systems and components to the greatest extent possible to help minimize the need for active device systems for computing.

  The following analysis of the structural / manipulation structural stages, subsystems, components, and elements of the image processing and pixel image display generation system includes the rifles of how this object is achieved. The structure, components and operation stages of the system are shown in order, from interception of the external image wavefront to transmission of the final intermediation image to the HVS (order is arbitrary from left to right for simplicity) It is set (see Figure 1).

  A. General Case-Key Elements of the System

  1. Infrared / near infrared and UV filtering stages and structures (infrared and near infrared filtering are omitted in the version of the system implemented for night vision systems).

  2. Polarization filtering to reduce incident pass-through illumination intensity (this option has some advantages and benefits), or polarization filtering / sorting into channels to hold the maximum input or pass-through illumination stage, polarization Rotation, and re-combination of channels (this option has other benefits and benefits).

  3. Pixelation or sub-pixelation of real world pass-through illumination and a channel to implement it.

  4. Integration of pass-through signal channels with internally generated sub-pixels (combination in integrated array) to achieve optimal enhanced / hybrid / mixed real or virtual reality image display presentation.

  i. Two preferred overall schemes and structural / architectures for handling and processing pass-through (real-world) lighting: Other permutations and versions are enabled by the general features of the present disclosure, but two preferred implementations The main difference in morphology is basically in the construction of the optical system, which processes incident natural light and transmits the light via the subsequent processing stage to the surface of the composite optical body facing the inward / viewer. The point is that the channel is different. In one case, all real-world pass-through lighting is downconverted to infrared and / or near-infrared "fake color" for efficient processing, and in another case real-world pass-through visible frequency illumination is / Processing / control directly without wavelength shifting.

  ii. Generated / "Artificial" Sub-Pixels in Integrated Array: This is preferably a hybrid magnetic photonic, pixel signal processing and photonic coding system. The same general method, sequence and process apply to the version and case pass-through optical signal channel where all pass-through light is downconverted to infrared and / or near infrared.

  B. Detailed disclosure

  1. Infrared / Near Infrared and UV Filtering Stages and Structures: Wearable HMD "glasses" or "visors" have a first optical element, which in a preferred form is a separate element on the left or right or A binocular element that is any one of a visor-like connected element and that intercepts the real world wavefront through the view of rays emanating from the outside world relatively forward of the viewer / wearer.

  This first element is a composite or structure (e.g. there is a deposited layer of material / film thereon, or itself is a periodic or aperiodic but complex 2D or 3D structured material, or Substrate / structural optical body), which is a hybrid of composite and direct structure, which performs infrared and / or near infrared filtering. And

  UV filtering. More specifically, they may be lattices / structures (photonic crystal structures) and / or bulk films of chemical composition that perform reflections and / or absorptions of undesired frequencies. These material construction options are well known in the relevant art and many options are commercially available.

  In some embodiments (especially in night vision applications), infrared filtering is eliminated and some elements of the functional stage sequence are reordered, eliminated or modified according to the patterns and structures of the present disclosure Be done. Details of this category and version of the embodiment have recently been treated as follows.

2. Polarization filtering (to reduce the intensity of incident pass-through illumination), or polarization filtering / sorting into channels, polarization rotation, and channel recombination to hold the maximum input or pass-through illumination stage: in the optical line-up sequence An optically similar, similar filter (figure below) of the first filter (The next element on the right) is either a polarization filter or a polarization sorting stage. Again, this is a practical feature for bulk "Polaroid" or polarizer films or deposited materials, and / or polarization grating structures or any other polarization filtering structure and / or any given embodiment. And materials that provide the best combination of benefits (ie, with respect to parameters that may require efficiency, manufacturing cost, weight, durability, and other optimization tradeoffs).

  3. As a result of the polarization filtering option: After this sequence of optical elements arranged over the entire range of optical elements / optical structural elements, the incident wavefronts are frequency grouped and grouped in polarization modes, sorted / separated by mode It is done. For visible light frequencies, the net brightness per mode channel is reduced by the grade of the polarization filtering means, which for convenience actually reflects 100% filtering efficiency, reflecting the latest efficiency of the periodic grating structure material It is close, meaning that about 50% of the light is excluded per channel.

  4. Polarization filtering, sorting, one-channel rotation, and re-combination results: for example, when combining two separated / sorted channels, the combined intensity is of the original incident light before filtering / separating / sorting Although not exactly the same as the strength, the strength is close to this.

  5. Advantages and Significance: As a result of these filterings (which may be implemented with the same layer / material structure or later with separate layer / material structure), the HVS is 1) protected from bad UV, 2) brightness Is reduced and 3) infrared and near infrared are removed (except for night vision applications where the visible spectrum is lowest and visible filtering is not required). Advantages / Features 2 & 3 are very important to the next stage of the system and the whole system, and will be discussed in more detail below.

  6. Pixelation or sub-pixelation of real-world pass-through illumination, and channels implementing it: sub-pixel subdivision of the incident wavefront, optical passive or active structure or operation stage implemented along the above, and preferably Content (as this tends to reduce manufacturing costs). This repartitioning can be performed in a wide variety of ways known in the art (as well as in other unplanned ways), including deposition of differential index bulk material, electrostatics / van der Use of photochemical resist mask etching processes or material fabrication of nanoparticles in colloidal solutions by means based on Worth force and other self-assembly methods, focused ion beam etching or embossing, and etching, cutting and embossing methods In particular, the fabrication of capillary microhole arrays implementing guiding with a modified total refractive index, or implementing other structures fabricated on photonic crystal Bragg grating type structures or other periodic grating or bulk materials according to Production of other periodic structures. Alternatively, or in combination with the mentioned or other known or later conceivable methods, the sub-pixel subdivision / inducing material structure for forming the array over the area of the macro-optical / structural element is compositionally It may be fabricated by assembly of parts (such as optical fibers and other optical element precursors), said fabrication being an author of the present disclosure for a preform assembly of a fiber device structure, or a fusion glass or composite assembly method. According to the method disclosed anywhere by and the method proposed by Fink and Bayindir.

  The specific details and requirements of the different embodiments and versions of the system, as applied to the present structural / operational stages of the system, cover at appropriate stages after the following structural / operational specifications of the system Be done.

  7. Integration of pass-through channels with internally generated sub-pixels in an integrated array, but to provide a means for subdividing the incident wavefront from the forward field of view into parts suitable for controlled light path control In addition, and after that the total view provided to the viewer using the proposed system on the way to the final pixel presentation to the viewer for further passive and / or active filtering and / or correction It is very important at this point to identify that there are two types of pixel / sub-pixel components of the array and two different "branched" processing sequences and manipulation structures. And this is one of the first stages and requirements for this composite structure and sequence of actuation processes in which light path control is performed at the appropriate stage, pixel by pixel and sub pixel by pixel.

  8.2 Two Pixel Signal Component Types-Pass Through and Generated or Artificial: In pixel signal processing, which is the pixel logic state encoding stage following the disclosure mentioned, this time two pixel types, or more precisely , Receive separately two pixel signal component types.

  9. Two preferred overall schemes and structural / architectures for handling and processing pass-through (real-world) lighting: other permutations and versions are enabled by the general features of the present disclosure, but two preferred implementations The main difference in morphology is basically the construction of an optical body that transmits light to the output surface of the composite optical body facing the inward / viewer via treatment of incident natural light and subsequent processing stages. Is different from the channel in FIG. In one case, all real-world pass-through lighting is downconverted to infrared and / or near-infrared "fake color" for efficient processing, and in another case real-world pass-through visible frequency illumination is / Processing / control directly without wavelength shifting.

  a. In one preferred version, the visible light channel, which is UV and infrared filtered and sorted into polarization modes (and optionally filtered to reduce the overall intensity of the pass-through illumination) is frequency shifted to infrared or near infrared However, in each case it will be an invisible frequency and realize a "fake color" range of width and intensity located in the same proportional band. The HVS will not detect anything and will not see anything after the frequency / wavelength modulation and downshifting photonic pixel signal processing methods. And, the subsequent photonic pixel signal processing of these channels is basically the same as proposed for the generated pixel signal channel, as disclosed in the following section.

  b. In another preferred embodiment, the pass-through channel is not frequency / wavelength modulated or downconverted to invisible infrared and / or near infrared. In this configuration, the preferred default configuration and pixel logic state of the pass-through channel is "on", eg, conventional for pixel state coding / modulation for subchannels sorted into any given polarization model. If a linear Faraday rotation switching scheme is used (including input and output polarization filtering means), the working linear Faraday effect pixel logic state encoder is addressed since the analyzer (or output polarization means) is essentially identical to the input polarization means If specified and activated, this is an operation to reduce the strength pass-through channel. Some details of the features and requirements of this embodiment are disclosed in the following sections, after the details described for the working function and structure of the generated signal channel).

  When polarization filtering is combined with the preferred embodiments and variations (mode sorting of separate mode channels later combined with integrated channels by polarization rotation means to preserve as much as possible original pixelated pass-through illumination And not the realization (such as by passive components (eg half-wave plates) and / or active magneto-optical or other mode / polarization angle modulation means), the overall brightness of the pass-through illumination is Typically, it is reduced by about 50%, which in some cases makes the preferred class and method more preferable given the relative visible range performance of the magneto-optical material as described herein.

  Thus, the maximum value of the brightness of the background pass-through illumination is proportionally reduced, and accordingly the subsystem that provides the "generated" (artificial non-pass-through) sub-pixel channel and associated methods and apparatus It may be easier to match, integrate and match generated image elements within a generally comfortable and realistic overall illumination range for "Augmented Reality" images and views.

  Alternatively, because the pass-through channel can be configured in a default "off" configuration, does the input polarization means (polarizer) and the output means (analyzer) face each other when using a typical linear Faraday rotator scheme? Or "cross". As the frequency dependent MO material (or other photonic modulation means using frequency dependent / performance-determined material continues to improve), by the subsequent photonic pixel signal processing steps and method It may be advantageous to adopt this default configuration in which the pass-through illumination intensity reference state is enhanced and managed from a default "off" or near zero or effective zero intensity.

  c. Considering the dependence of common materials and systems on the infrared and near-infrared performance optimization of photonic modulation means and methods, while down conversion to infrared is proposed as preferred, an option also includes UV And, in the future, it may be used to shift the input visible illumination into a convenient invisible spectral region for intermediate processing before final output.

  10. Generated / "Artificial" Sub-Pixels in an Integrated Array: First, the pixel signal processing, which is preferably a hybrid magnetic photonic pixel signal processing and photonic coding system, that is an imaging pixel signal component or, in other words, preferably a hybrid magnetic photonic pixel signal processing and photonic coding system Examine the structure and operation sequence.

  a. In the most general configuration of the proposed image acquisition / processing / display subsystem of the overall system for fully mobile AR in daylight conditions, the following structures, processes and elements in the sequence are optical infrared and / or near infrared light Infrared flat illumination distributed structure and pixel signal processing stage.

  b. Optical surfaces and structures (structural / film deposited or mechanically laminated to the substrate, or directly to the substrate, patterning or deposition of material, or a combination of methods known in the art) for this structure and operation process Uniformly distributes infrared and / or near infrared illumination evenly over the entire optical area of a 100+ FOV binocular lens or continuous visor type form factor. The infrared and / or near infrared illumination is distributed uniformly, for example, by the following means. 1) A combination of leaky fibers arranged in all X or Y directions or in a grid on the XY plane of the structure. Leaky fibers developed and marketed by companies such as Physical Optics leak light transmitted substantially through the fiber core laterally in a substantially uniform manner over a specified design distance, Luminit, Inc. And / or other diffusion materials and structures known in the art such as aperiodic 3D bump structured films (embossed aperiodic microsurfaces) commercially available from 2) Lateral illumination from infrared and / or near infrared LED edge array or infrared and / or near infrared edge laser array (such as VCSEL array), bulk illumination (planar sequential beam expander such as planar periodic grating structure / Spreader optics to be intercepted as holographic elements (HOEs) commercially available from Lumus, BAE, and other commercial parts manufacturers referred to in this specification and in the above-mentioned continuation application 1.) Structures, as well as other backplane diffusion structures, materials and means, and generally, other display backplane lighting methods, means and structures known in the art and may be developed in the future.

  c. The purpose of this stage / structure in the manipulation sequence and pixel signal processing is to emit infrared and / or near infrared backplane illumination limited to the relative interior of the composite optical / material structure proposed so far, infrared and / or near infrared An infrared filter is to reflect input infrared and / or near infrared illumination to the illumination layer / structure.

  d. It is important to note the fact that the infrared and / or near infrared rays are invisible to the HVS, even if the facts are obvious.

  e. The infrared and / or near infrared illumination sources may be LEDs, lasers (such as VCSEL arrays), or a hybrid of both, or other means known in the art or to be developed in the future.

  f. The infrared and / or near infrared illumination introduced is also illumination of a single polarization mode, preferably planar polarization.

  g. It splits infrared and / or near infrared LEDs and / or lasers and / or other illumination sources with polarization splitters or filter / reflector sequences (such as fiber optic splitters) and means for passive and / or active polarization rotation Passing one of the plane-polarized components through either (such as bulk magneto-optical or magneto-photonic rotators) or a sequence of passive means (such as a combination of half-wave plates) or hybrids thereof Can be achieved by means of polarization harmonization. A polarizing filter, such as an efficient grating or 2D or 3D periodic photonic crystal type structure set at an angle to the incident light, may reflect the rejected light back into the polarization rotating optical sequence and channel, and then Recombine with the unmodified part of the original light. In waveguides (of planar or optical fiber) where the polarization modes (plane polarized) are separated, one branch passes through the polarization harmonization means and then recombines with the other branches.

  h. Source illumination may also be constrained to create only plane polarized light at a given angle or range within its own structure.

  i. The light is generated and / or harmonized locally (within the HMD or remotely from the HMD (such as a wearable vest with power storage)) and transmitted to the HMD via an optical fiber. In the HMD, the illumination and / or harmonization stage and the structure / means are directly adjacent to the described composite optical structure or elsewhere elsewhere in the HMD, and more remote by means of optical fibers and / or more It can be optically propagated by a planar waveguide if close.

  j. The above structures and the previous operation structures and processes, and those described below, use pixel signal feature generation and transfer processes with the highest quality methods and are typically optimized for the type of process Among the features resolved into optimized stages operating at different wavelengths (especially in connection with pixel state logic encoding stages and processes), an example of pixel signal processing as disclosed in the mentioned application It is. Many MO and EO and other optical interaction phenomena work best for most material systems in the infrared or near infrared frequency band regime. The overall system, method, structure, operation structure and process, as well as their respective details (including essential and optional elements) are disclosed in the referenced application.

  k. Pixel Signal Processing, Pixel Logic Stage Encoding Stage-Modulator Array:

  1. After the illumination and harmonization stage, the infrared and / or near infrared / infrared illumination passes through the pixel signal stage logic encoding process, operation, structure and means, preferably in the category of magneto-optical modulation methods for the present disclosure. Pass through the entering modulation means. Of these, one preferred method is based on the Faraday effect. Details of this means and method are disclosed in the mentioned US patent application "Hybrid MPC pixel signal processing".

  m. In a binary pixel signal logic state system, the "on" state is encoded by rotating the angle of the incident plane polarized light so that the light is in a later stage of the pixel signal processing system, ie, a subsequent and opposite one In passing through polarization filtering means (known as an "analyzer"), light will pass through the analyzer.

  n. In this type of MO (or subtype MPC) pixel signal logic stage coding system, light is exposed to a magnetic field, medium or structure and material, ie uniform / bulk or structured photonic crystal Or pass through the metamaterial (which is typically solid) (although light can also pass through the gas or dilute vapor, or the encapsulated cavity containing the liquid), which allows rotation of the polarization angle In addition, it has an effective figure of merit that measures the efficiency of the medium or material / structure.

  o. Details of suitable types and options for this preferred type of pixel signal processing logic stage encoding stages and means are described in the referenced copending application and further variations can be found in the prior art or will be developed in the future It can be done.

  p. Other aspects of the preferred and mentioned class of hybrid MPC pixel signal processing systems that require the highlighted detailing include:

q. The hybrid MPC pixel signal processing system implements memory or "latching" and is powerless until the pixel logic state requires a system change. This is achieved by the following adjustment and implementation of the "residual" magnetic method known in the art, in which the magnetic material is bulk processed (e.g., commercially available Integrated Photonics commercial latching LPE thickness MOBi- YIG film [see our other disclosures]) and / or means of permanent domain area latching periodic 1D gratings such as Levy [see our other disclosures] or optimized Synthetic magnetic material (combined field, combined with MO material coupled with relatively “harder” magnetic material co-located / mixed (low applied field maintains magnetisation (latching of MO / MPC material as intermediate) (Latching to linear hysteresis curve material). This intermediate material may surround the MO / MPC material or may be mixed or structured in a periodic structure that is transparent to the transmission frequency [herein infrared or near / infrared]. This third synthetic method is a 2004 US provisional application by the author of the present disclosure. First proposed in / US patent application Included in Later, Belotelov et al. (Funded by a company founded on the basis of the 2004 disclosure) came to refer to this composite method as a "exchange coupled" structure, and for certain 1D multilayer magnetic photonic crystals. To be realized in the company's design, in the 1D multilayer magnetic photonic crystal, MO materials with different relative hardness were used with the less efficient variant of the 2004 composite approach.

  r. A combination of these methods is also a possible design option.

  s. The benefits of this "memory pixel" in the hybrid MPC regime are the same as bistable pixel switches such as electrophoretic or "E-Ink" monochrome displays. Non-volatile (relatively, at least according to the design of the hysteresis profile and the choice of material) as memory, as long as the infrared or near infrared illumination source is "transferred" and "processed" in the pixel signal processing channel and system Will remain formed.

  t. The second essential aspect and element of the preferred pixel signal processing, pixel logic encoding stage and method is the efficient generation of the magnetic field to switch the magnetic states of the sub-pixels (the basic primitives of color systems such as RGB Thus, for convenience, when considering the conventional components of the final color pixel, the naming convention is kept more general and distinctions made when necessary). In order to ensure that there is no magnetic crosstalk, it is preferred that field generating structures (e.g. "coils") be placed in the path of the pixel transmit axis rather than the sides. This reduces the required field strength by not placing any field generating means at the edge, either a (magnetically) impermeable material in the surrounding material / matrix or a domain such as Levy As in the case of the continuation method, the management of the field lines is carried out either by the implementation of a periodic structure which limits the substrate to the modulation zone. Transparent materials can include available materials such as ITO and other newer and emerging conductive materials that are transparent to the frequency of interest. And / or other material (e.g., metal) that is not necessarily large enough to be transparent, but that is a periodic element of appropriate periodic element size, geometry, and periodicity, also the modulation zone / subpixel transmission path May be deposited or formed.

u. This method was first filed by the author of the present disclosure in the 2004 US Provisional Application. Is assigned, and later a US patent application Proposed in the 2004 International Design Document for the company, which is disclosed in Later, 201+ At the same time, NHK researchers use his ITO in the pixel path and his method for Kerr rotators (this method has been generally proposed for MO and MPC devices) [Reference SE should investigate] was used.

v. The third important element of a preferred hybrid MPC pixel signal processing solution for the pixel signal processing subsystem is the method of addressing the array of sub-pixels. A preferred method, as mentioned earlier, is a pending US patent application. Wireless addressing and power of Device Arrays (wireless addressing and power of element arrays). For the present application, wireless addressing may be sufficient to integrate the powering of the wireless array (sub-pixel) elements, omitting the low frequency magnetic resonance wireless power method and considering the low power requirements. Depending on the material selection and design details, microring resonators may be more efficient than powering via microantennas. However, the wireless power supply as a whole HMD or wearable device mounted on the head when combined with the local high power density metacapacitor system or any other capacitance technology that can be powered up by a wireless low frequency pack It is the preferred method of powering the entire unit while reducing the weight and bulk of the process. Basic low frequency magnetic resonance solutions are available from Witricity, Inc. US Patent Applications for more complex systems See “Wireless Power Relay”.

w. Other preferred methods of array / matrix addressing and powering include voltage-based spin wave addressing, which is not specified in the mentioned application, and is thus a novel variant in the present proposal. Originally mentioned "Hybrid MPC pixel signal processing" application And other form factors and use cases thereof. High speed current based backplane / active matrix solutions (such as OLEDs) developed for other display technologies are also available options.

  x. Other less preferred pixel signal processing pixel logic coding technologies and methods also benefit from radio addressing and power delivery methods and voltage based spin wave methods depending on other specific design choices. It becomes.

  y. Such other pixel signal processing, pixel logic coding means (including modulators based on Mach-Zehnder interferometers) (the efficiency of which is typically based on frequency material systems, and in the infrared and / or near infrared) The most efficient) may also be used, but these are less preferred and, likewise, optimized for the most efficient frequency for the means of the class according to the teaching of the mentioned application Any number of other pixel signal logic encoding means may also be used in the architecture and / or material system.

  z. For the preferred embodiment of the proposed system, this [2008] US patent application entitled "Telecom-structured Pixel signal Processing methods (Telecom-structured pixel signal processing method)" is used to form a dual sub-pixel array system. It is also essential to identify with this particular variation and optimized version disclosed herein for the application, and other non-HMD and non-wearable display system applications having similar operating requirements or desirable benefits. It is.

  aa. Following pixel signal processing, the pixel logic state encoding stage of the manipulation structure and process is an optional signal gain stage. If this option applies, it will be taken at a clear point in the following description.

  bb. Wavelength / frequency shift stage: frequency up-conversion using phosphor color system with preferred nanophosphors and / or quantum dots (eg QD Vision) enhanced for this particular version of the preferred hybrid MPC pixel signal processing system Stages follow (although periodically polarized devices / material systems are also identified as options in the mentioned disclosure). Basic technologies that are commercially available include technologies from suppliers such as GE, Cree, and a wide range of other vendors known in commercial practice.

  cc. What is being done is the division or separation of the upconversion process that typically occurs in the illumination stage, and is optimized for operation at infrared and / or near infrared frequencies and for other reasons Delaying this division or separation until after several other stages have been completed will be apparent to one skilled in the art at this point.

  dd. Thus, optimization of the nanophosphor / quantum dot enhanced phosphor / structural formation tailored to a color system such as the RGB subpixel color system completely implements the color system. Again, a reconsideration of the concepts and operation of such a display system is described in more detail in the mentioned application.

  ee. The advantage of using the hybrid MPC pixel signal processing method is the high speed of the original MPC modulation rate, which has been demonstrated to be less than 10 nanoseconds for quite a long time, and is currently sub-ns Is the relevant standard. The rate of the phosphor-excitation reaction is relatively fast (although not very fast), but overall and net, the total color modulation total rate is less than 15 nanoseconds, and a theoretically lower net duration measurement Will even be optimized.

  ff. Variants in the proposed structure add band-pass filters to each of the infrared and / or near infrared sub-pixel channels, which are "on" or upscaling to R, G, or B at the end of the processing sequence. It will be either "off". This variant, while adding to the complexity of the filter element, may be suitable in the following cases. 1) In the case of composites of materials, where the hybrid MPC stage itself is an array of adapted materials that react more efficiently to different sub-bands of the infrared and / or near infrared domains (though this is not the case in this case) Although not likely, even bulk LPE MOL films commercially available in their wavelength domain are due to their near 100% transmission efficiency and very low power polarization rotation). 2) The efficiency of the formation of nanophosphors / fluorescent substances enhanced with different nanophosphors and / or quantum dots is sufficiently high so that it is more precise for each final R, G and B subpixel component In the case of being worthy of infrared and / or near infrared frequency bands summarized in. This design tradeoff adds the complexity of passing additional layers / structures / depositions to band together, as opposed to more “tuned” frequencies / wavelengths to different parts of the invisible input illumination spectrum We will end up with cost / benefit analysis of the efficiency gains from the possibility of using shifting materials.

  gg. Following this color processing stage, the sub-pixel groups realized from the first infrared and / or near infrared illumination source continue through the integrated optical pixel channel. If other final pixel components are not to be added, the output pixel will depend on the design choice for the modulation and dimensions of the color stage component, as appropriate, and is preferably an optional pixel expansion by diffusion means It may be necessary (including those mentioned and as disclosed in the mentioned application) (pixel spot size reduction is very unlikely to be necessary here, and this pixel spot size reduction Are known in the relevant art and require optical focusing or other methods as disclosed in the specific application of the mentioned applications, in particular [2008]).

  hh. Collimating optics (including lenslet arrays), an optical fiber array embedded in a textile composite in which the fibers are arranged parallel to the light transmission axis, for the purpose of realizing a virtual focal plane at an appropriate distance from the viewer's eye "Flat" or planar inverted index metamaterial structures and other optical methods known in the art are used. Preferably, all the elements are fabricated or realized in a composite layer on the macro-optical element / structure, rather than requiring additional bulk optical eyepiece elements / structures. Further questions of fiber type methods and, in contrast, laminated composites or deposited multilayer structures, or two or more combinations / hybrids, are addressed in the sections below on Structural / Mechanical Systems.

  ii. As mentioned above, pixel signal processing pixel logic array functional / optical / structural elements implementing the disclosed pixel signal processing pixel logic structure and operation stages (including preferred hybrid MO / MPC methods and manipulation structures) However, it is not a bulk device that operates across the field of the incident wavefront, which is a pixelated array (as would be expected by one skilled in the art).

  jj. Each final pixel may include at least two pixel components (beyond the aforementioned color system RGB sub-pixels). One is an arrayed component that generates the video image from scratch, which may include simple text and digital graphics, but for all purposes of this system, CGI or relatively remote High resolution images can be generated from either live or archived digital images, or from their composites and hybrids.

  11. Pass-Through Real-World Illumination and Pixelated Arrays-Detailed Conditions for Working with Visible Frequency Pass-Through (ie, Not Downconverted to Infrared / Near Infrared): Structured and Operating Optics and Photonic Structures and Returning to the transmission and processing of real world ungenerated rays from a view through the stage.

  a. It is either another set of pixels or other sub-pixel components that are co-located on the addressing array with these infrared and / or near infrared driven sub-pixel clusters. Above, the final pixel channel component resulting from the viewer of the HMD and the live view in front of the wearer. These are completely addressable components in the "pass-through" of the final pixel.

  b. These channels originate from the front complex optics / structures subdivided into pixels as specified.

  c. These light channels propagate wavefront portions with low wavefront losses by using available efficient splitting methods. A surface lenslet array or mirror funnel array may be used in combination with the proposed repartitioning method, which allows the light capture efficiency very close from edge to edge, so that the captured wavefront portion is then efficient Coupled together, they become the relative "core" of fragmented / pixelated guiding optics / array structures. Thus, whether the conventional step index bonding method is used, or a pixel that contributes to the bonding means, regardless of whether it is a MTIR microporous array, or a true photonic crystal structure, or a hybrid of two or more methods. The area of integrated array formation will receive the minimum percentage of wavefronts with minimal loss.

  d. Efficient wavefront capture, routing, and induced / pixelated segmentation focus on visible and infrared and / or near infrared frequencies for specific versions of the system and mode of operation and A broadband optical element is required which reflects and / or reflects, and it is also proposed to implement an infrared and / or near infrared filter as the first and first optical filtering structure and means in the optical lineup and sequence as described below Though needed.

  a. In most configurations, the infrared and near infrared illumination stages are "pass-through" captures that are interspersed throughout the stage and transparent to infrared and / or near infrared, but provide visible frequency light guidance / path confinement Because there is a guiding structure for the illuminated illumination, the infrared and / or near infrared can be uniformly distributed without interfering with the channelized "pass-through" pixel component.

  b. When the induced incident wavefront partial signal channel reaches the pixel signal processing, ie the pixel state coding stage, a bulk MO or multilayer MPC film or a periodically structured grating (film) which is otherwise 'bulk' Or parallel pixel signals if the efficiency of the material or structured material is optimized for infrared and / or near infrared, if there is a single formation of a 2D or 3D periodic structure) The processing, i.e. the pixel logic state structure, will be implemented in exactly the same way, but with much lower efficiency.

  c. However, as a broadband MO material fabricated by various means, both as bulk-formed and structured photonic crystal material, its efficiency has been optimized for infrared and near infrared MO / MPC materials / The efficiency of structured materials is not equal at the moment but continues to improve. Initial work, led by the author of the present disclosure in 2005, models and produces new MO and MPC materials, which for the first time demonstrate significantly improved transmission / Faraday rotational pairing for the green band regime. Not only did the first non-negligible, in fact, significant and acceptable, competitive, performance in the blue band for display applications was demonstrated.

  However, the fabrication of such materials tends to be more expensive, if different materials are to be deposited as "small films", for "genetic" pixel components and for pass-through pixel components. This adds to the complexity and cost of the fabrication process. However, such an arrangement would improve the efficiency of pixel logic state encoding of the final integrated pixel "pass-through" component, all conditions being equal.

  d. In the absence of deposition or formation of "fitted" MO category materials (this logic is less preferred, but its maximum efficiency is frequency dependent like MO / MPC, instead a single formation If all conditions are the same, which is also used, the strength of the pass-through final pixel component will be lower, to the extent that the modulation means is less efficient.

  e. Typically, for pass-through systems, it will be assumed that no phosphor type or other wavelength / frequency shifting means will be used. However, to the extent that the efficiency of the original MO / MPC material may be lower, in this case, different formations of band optimization material to partially address the lack of material performance at the pixel logic stage encoding stage May be used.

f. Furthermore, as suggested for low light or night vision operation, some of the applications mentioned (US patent application "Pixel signal processing" And US patent application hybrid MPC pixel signal processing An optional “gain” stage, as proposed as an option), in which a large amount of pumped gain material is introduced to provide an optical, electrical, sound, mechanical or magnetic The energy gain is implemented in the gain medium at any of the final pixels (as described in the referenced application for this detail and by other methods known in the art or may be devised in the future), As the transmitted "pass-through" component passes through the gain medium, it enhances the strength of this component. This is a variable and addressable stage, but if this design option is chosen, it is not preferred that it be a setting that would rather increase the blanket gain.

  g. Furthermore, when the induced incident wavefront sub-channel reaches the pixel signal processing, ie pixel state coding stage (as described above), an option for low illumination and night vision applications, but for the entire pixel signal processing and optical channel management system There is a valuable option configuration.

  h. In this variant, it is possible to remove the infrared filter, but since the goal is to pass the infrared and / or near infrared light from the incident real world wavefront to the active modulation array sequence, the incident "actual" Infrared is a pixel signal processing modulator with a similar color (monochrome or false color infrared gradient) image generated for the viewer to the extent that the infrared output is visible in the field of view, without the need for sensor array intervention. Pass through and directly.

  i. And, as mentioned above, a gain stage may be implemented to increase the intensity of pass-through infrared (if useful, + near infrared) to the wavelength / frequency shift stage.

  j. In addition, the basic infrared and / or near infrared background illumination (intensity modulated to set the appropriate base level) usually does not reach the threshold for activating the wavelength / frequency shift stage and medium through full color operation mode It can be adjusted.

  k. Although removal / deactivation of the infrared filtering means can be done mechanically, it may be provided that passive optics are deployed in a hinged or cantilevered device that can be "flipped up" or in an active configuration As an element, as in the electrophoretic type activated bulk encapsulated layer (in which several relatively flat filtering microelements are electrostatically (mechanically) rotated) When activated, the lowest incident angle is passed, and the multiple rotated elements no longer filter infrared). Other passive or active activation / removal methods may be used.

  l. Infrared filters and polarizing filters for low-light or night-vision operations were incident not only on the thresholding, but also on the pixelated array, depending on whether the production system is being used "actively" Both may be removed to superimpose data on a portion of the actual infrared wavefront portion. When actively used, in order to maximize the efficiency of the generation source, the preferred digital pixel signal processing system is an optical switch / modulator in which the first polarization filter encodes pixel logic states in the signal. Need to carry out.

  m. The disadvantage of pass-through systems is to reduce the intensity of incident infrared and / or near infrared radiation.

  n. Another embodiment of the present system designed to address this issue places the gain stage before the pixel signal processing, ie the pixel logic state encoding stage, to increase the incoming signal.

  o. The efficiency of the gain medium with non-coherent, non-collimated "natural" light should be considered in the design parameters of the present system and any system that uses a gain medium pumped with the input of "natural" incident light.

  p. A second separate component three system is implemented which includes a component sub-channel for the generating means, an incident visible light component and an incident infrared component which is not polarization filtered. In order to realize this variant, a pixelated polarization filter element exiting from this third subchannel / component without a polarization filter element has to be implemented.

  q. For the more basic integrated component two option system type with this type of low illumination night vision operating mode requirements, additional optics on the first incident wavefront input and channelization / pixilation stage Is required.

  r. The incident infrared radiation (and near infrared radiation, if necessary) is the usual "generating" source component of the final visible pixel, and the entire visible light portion of the incoming incident wavefront that source of the final visible pixel. It may be split between the sub-channel towards the pass-through channel leading to the component, but there is no specific efficiency gain to transmit infrared and / or near infrared to the visible light sub-channel and source for the final pixel .

  s. Rather, it is a frequency splitter that follows, or is integrated with, a lenslet or another optical capture means to maximize capture of the incident real wavefront. One method implements opposing filters of one band pass filter for visible light and another adjacent filter for infrared and / or near infrared light, which allows only infrared and / or near infrared light It is. The different geometrical arrangements of such opposing filters (both set in both planes or offset from each other by 45 degrees from the central focus of the incident wavefront optical capture structure) provide different advantages. The advantage is that the combined visible / infrared near infrared beam focused (by a lenslet or other optical element or means including a reverse index metamaterial "flat" lens) is a first separate The two band ranges allow the other beams to be reflected to the opposite filter surface, and for the portion of the focused beam which initially strikes the filter structure further from the central focus, the opposite is true. It is equally possible. The grating structure is the preferred method to implement the dual filter splitter arrangement, but in subsequent stages to implement the two filtering faces (a variety of methods known in the art and to be developed in the future) Other methods based on bulk material formation that can be deposited) are also known in the art. (Please note that UVs are filtered before this stage, but preferably after infrared rays. In some cases, the infrared and polarizer phases are first and second, and in some cases the UVs are The filter is third, otherwise UV, then infrared followed by UV, and polarizers.Different arrangements have different value for different use cases, have different impact on fabrication costs and specific process sequences).

  12. Combination of pass-through with generated / artificial pixel / subpixel array

  The two component light channels are co-located as described above and are preferably output together to the pixel harmonization means (diffuse and / or other mixing methods and known in the art or in the future This generation source is combined with the pass-through source to form the final composite pixel, similar to the RGB sub-pixels of the conventional color vision artificial color additive display system, as available by the other methods contemplated . And this is the most effective and most effective with HVS, given the ergonomic design goal of being close to the face, which is also part of the purpose of this disclosure, as described in detail above and in the mentioned application. In order to form an image at a virtual focal plane that is easy, it is further pixel beam shaped and in particular collimated and otherwise optically directed.

  a. Basic integrated two-component system operation with a "genetic" component (which itself consists of RGB sub-pixels) and a variable "pass-through" component-the first is the main mode of operation And the second is configured for optional low illumination night vision mode

  The wearer of the HMD in the proposed form looks out with integrated binoculars (two separate lens form factor device constructions) or a connected visor, in the daytime outdoors where the sun shines, which In contrast, the pixel array (which is itself formed by the integration of two input components: a generative high-performance pixel and a pass-through variable-intensity wavefront portion of the "window to the world" facing the viewer. Present the image formed by the integration of

  b. The composite color component for the final integrated pixel is formed by the "generating" pixel component, which starts as invisible infrared and / or near infrared "internal" "injected" back lighting, which is It is turned on or off in less than 10 nanoseconds (and recently less than 1 nanosecond) for a pixel. And the infrared and / or near infrared sub-pixels activate synthetic fluorescent materials / structures using the best modern materials and systems available to generate the widest possible full color range.

  c. When the sub-pixel state is set with very short waves, the "memory" switch remains on until the state changes, without applying constant power to the switch.

  d. Thus, the generative component is high frame rate, high dynamic range, low power, wide gamut pixel switching technology.

  e. The second component of the composite pixel is the pass-through component, which is a fraction of the efficiently high percentage of the total wavefront that strikes the front optical surface of the present HMD, which is incident from the direction facing the wearer It starts as a These wavefront portions are filtered for UV and infrared in normal mode, and are also polarization sorted or filtered (this is either a reduced real world illumination or a maximized selected design) It is chosen according to the strategy). A reduced base, ie polarization filtering, results in a substantial reduction (approximately 1/3 to 1/2) in the overall brightness of the visible field of view depending on the composition of the polarization mode event and the efficiency of the polarizer Be done.

  f. With particularly bright daytime light (but in generally all lighting conditions except very low to no light at all), the reduction of the pass-through intensity will "match" and match or match the illumination level of the incident wavefront portion Is easier to Thus, by means of passive optical means achieved by components of the system that perform two roles or produce two benefits, this is a need for a suitable modulation system (based on polarization modulation) to implement pixel logic state coding These components also reduce the power requirements and simplify the calibration, adjustment and synthesis of the values of the generation system and the pass-through system.

  g. The design features of this system take advantage of the fact that for most people, bright outdoor lighting conditions are managed by using polarized sunglasses. Indoors, excessively bright emissive or transmissive displays are known to lead to eyestrain, so reducing the illumination level overall even indoors results in a "competitive light environment in the field of view. In the production system, the problem of raising the light level relatively slightly is raised rather than creating the '' again. A reduced natural pass-through illumination (optional, but can optionally be enhanced by a gain stage that is less efficient than LEDs or laser light) and a generation system that adds graphic or synthetic elements to parts of the scene And the combination with E., results in a baseline that is more harmonized and less intense than otherwise. (The generation system, ie the relevant part of the integrated array, does not necessarily generate the entire FOV in AR mode but can be generated in full VR mode).

  h. Given the computed coordination and compositing of compositing elements and real elements in the perspective view of the user (which is the next aspect of the sensing and computing system), the hybrid of the generated source and the pass-through source is at the display level It can easily and quickly create a hybrid mobile AR / Mixed Reality view without any visible lag or appreciable latency.

  i. The pass-through pixel component sub-designated with the default "off" scheme (ie the polarizer and analyzer in the preferred polarization modulation form are "crossed" rather than the same), no pass-through wavefront portion is propagated In the channel, mobile HMDs that assume calibration of real scenery and motion tracking can function in mobile VR mode. As described below, in combination with the proposed sensor and associated processing system, the HMD can function as an "indirect viewing display" of Barrilleaux with pass-through turned off.

  j. Variable pass-through system (pixel illumination / image adding generative / enhanced channels, especially when the generation system is off and there is additional cost and complexity of optimized visible frequency MO / MPC material structure It is also possible to implement primitive information).

  In the reverse configuration of the "indirect vision display", further variants of the system are adopted and the "pass-through" channel filters are themselves respectively, as described later herein for the proposed sensor and associated processing system In the case of subdivision into RGB sub-pixel channels (following the pattern of infrared / near infrared and visible spectral filter splitters) with pixel signal logic state coding modulator of 0, the variable transmission means of the pass through system to be a direct view system It can be enhanced. The disadvantage is that it has a dynamic range and, by comparison, relatively low illumination, which has no means of complementing generation. Furthermore, such variants (modes or systems that simply eliminate generative structures) are doubled that can be handled by parallel processing systems, which simplifies the bottleneck in performing scene-integrated synthesis and perspective calculations. It will not have the benefits of an array. Furthermore, such a system based on an optimized MO / MPC material / structure in the visible spectrum with different adjustments will be more expensive and less efficient than a production system based on infrared / near infrared.

  k. An optimized system is one that combines an efficient generative component with a pass-through component of variable intensity but at a lower overall illumination level.

  l. The preferred wireless addressing and powering further reduces the power, heat, weight and bulk from the functional element part of the intelligent construction system.

  m. For systems that can remove or turn off the infrared filter, in very low illumination or night-vision modes, infrared (and near infrared if desired) passes through the pixel state system without loss, and the optional gain stage is an infrared signal Increasing intensity and / or infrared / near infrared internal injection lighting components increase threshold / basal intensity, in addition to which additional / superimposed intensity of incoming pixelated infrared, infrared / near infrared Passing the wavelength / frequency shifting means (preferred fluorescent type system), the system is set to monochrome or false color to realize a direct vision low illumination or night vision system. With the polarizing filter in place, the generation system can operate and add graphics and complete images, as suggested by the signal from the auxiliary sensor system (see below) or in other configurations, Reduced intensity of incident infrared radiation, either simply by adding a base level to ensure that the energy input to the wavelength / frequency shift is sufficient to produce sufficient power Make up for.

  II. Sensor system for mobile AR and VR

  According to the various cases of the disclosure mentioned, the composite generated image can be totally internal (and in some cases external lighting conditions that may be passed as desired or considered for efficiency) There is no image display structure that displays an image without a sensor system that optimizes and harmonizes, and follows the general case of the proposal, which does not take into account the user's position, viewing angle and motion tracking in general.

  1. In the preferred version of the system, at least some of the device components serve two structural elements, but in these cases they integrate other functional purposes and sensing to the extent that they can be sensed. It is impossible at all to be a combination that specifically distinguishes the device as an integrated overall system. (

  2. In the best general form the art is not a large individual macro camera system, but rather an array of multiple distributed sensors, including accelerometers, digital gyro sensors, optical tracking and other systems, the art In implementing a motion tracking sensor such as that known in the art, the system of the present disclosure is the preferred embodiment, which provides the benefits of distributed native and local processing, and in a "real time" manner in real time. Add image based / photogrammetric method to extract geometry data in real time to capture “real” lighting conditions and allow local updating to stored position / geolocation / terrain data Achieving specific benefits, calibration of composite image elements and their effective perspective view rendering, and / Is used to accelerate the integration and composition of the mixed view the scene.

  3. As disclosed in the mentioned application, and enlarged briefly, the commercially available Lytro system, among the "image-based" and photogrammetric methods that is particularly valuable for collecting used and proven real-time information There is a bright field method illustrated by the following which can image sample the space in real time from the multisampled (and optimally distributed sensor array) space and then sufficient initial data After input / import, a view-morphed 3D space can be generated. The virtual camera is then positioned at various locations in 3D space, as extracted from the photogrammetric data, at a predetermined resolution in real time.

  4. Extra local to allow for calibrated fluoroscopic image synthesis, including occlusion and opacity (using integrated dual generative and pass-through components of the preferred proposed display subsystem) Other image based methods can be used for geometric / terrain data in coordination and combination with the Lytro bright field method. Such methods are FOV in real time to obtain lighting parameters to match CGI shading / lighting or even simple graphic / text elements and live updates of navigated real world 3D terrain space It provides overall sampling as opposed to simply performing separate calculations on disconnected irrelevant pixel points from the file, GPS and conventional motion sensors only. General correction can be applied to lighting and relative position / geometry by parametric sampling, which greatly reduces the computational burden.

  5. In combination with GPS and other mobile network triangulation “absolute” positioning by signal methods, in combination with HMD and motion sensor tracking of any haptic interface, as well as to live updated images Image-based mapping of the user's body from the photogrammetry system, and relying on relative position and terrain parameters obtained from high-speed real-time image-based methods using multiple miniature sensors and cameras.

  6. In this regard, the Bayindir / Fink "optical fabric" camera developed at MIT is an example of verification of a particular physical method of implementing a distributed array. As proposed by the inventors of the present disclosure, a fiber device and intelligent textile composite method, or a simpler MIT fiber device fab method and an optical fabric embodiment, or other fiber device intelligent / active / photonic textile method Whether in compliance or not, they are distributed within the structure of the HMD machine frame and serve two roles by also adding to the structural system solution, rather than acting as a load that does not contribute to the system A textile synthesis camera array is a preferred version of implementing an advantageous multi-apparatus array system that provides for the acquisition of data distributed in parallel.

  7. A multipoint miniature sensor array, which may include multiple miniature camera optical sensor array devices, is another preferred embodiment of the multi-perspective system.

  8. The more basic integrated commercial Lytro system combined with some of the many other cameras / sensors in a small array is less preferred, but still a superior combination, enabling multiple image based methods Make it

  9. Again, the auxiliary infrared sensor, which is preferably located in a number of lower resolution element arrays, provides the display system with an override low intensity / night vision feed, as described above, or in harmony with the actual infrared pass-through. Corrections and supplemental data can be provided to the generation system to function in a coordinated and coordinated manner.

  10. The same layout Lytro type bright field system for the visible spectrum in patterns at the general level may be used for sensors in other frequency bands, depending on the application, low intensity / darkness As well as viewing, field analysis for other applications and use cases such as UV or microwave can be included. Given the limitation of resolution at longer wavelengths, spatial reconstruction may nevertheless be generated from reference data that is invisible or invisible and captured by GPS / LIDAR, and sensor scans of complex environments In doing so, other dimensional data collection correlations are obtained. As miniaturization progresses, compact mass spectrometry, which is now being realized with ever smaller form factors and miniaturizations, can also be contemplated for integration into the HMD.

  11. Finally, a compact global reflectance map is extracted in an image-based method that is advantageous for high-speed data sampling of the lighting parameters and knowing about the material, geometry and atmospheric conditions of the local environment from the lighting parameters The reflective surface on which the surface is imaged, the key vertices of the HMD (left and right corners or only the center, and the vertex paired with multiple imagers to capture the entire reflected surface Or, in order to extract a compact and compressed reflective surface, the concave reflective partial hemisphere "holes", either alone or preferably in combination with a spherical surface, are in each case maintained at the same speed via the magnetic field (Or can be used either on a powerful spindle or mounted mostly concealed) Black “light probe” is not only to accelerate live images and high-speed graphic integration of generated CGI / digital images (shading with shading, lighting, perspective rendering etc) but also in complex and rapidly changing environments Used in conjunction with other relevant methods from photogrammetry to photogram the lighting of the space, materials and geometry, in order to carry out a fast analysis of factors considered to be risks for sensitive manipulations. Provide a highly accelerated method.

  III. Machine and substrate system

  As will be apparent from the foregoing, the image display subsystem and the distributed and image-based sensing and auxiliary imaging system already proposed with emphasis on the preferred embodiments are structural, mechanical and human of the present disclosure. It has already provided substantial benefits and values for engineering goals.

  1. One preferred embodiment of structural functional integration that benefits in weight, bulk, size, balance, ergonomics, and cost is the tensioned thin film combined with the flexible optical structural substrate An embodiment of a textile synthetic structure, particularly preferred is an HMD frame formed of Corning Willow Glass, which folds in all the processing and functional electronics that must be integrated in the HMD (And preferably sealed), these electronic devices can include less preferred versions of the power supply without using a wireless power supply to assemble on the folded glass frame. A protective coating is applied / wrapped or otherwise added to functional optical components to protect the glass and the wearer, and for comfort and ergonomics (such as D30 based on a shockwave system) Although this is soft and resilient in the absence of an impact, shocks cause the shockwave to solidify the material, resulting in poor durability (but with an appreciable durability) Willow glass structural / Provide a protective barrier for functional systems). A folded willow glass (the inner surface of which is the location of the system's electronics on glass), to add strength and to better protect the electronics from shock, and also thereby a thinner substrate In order to make it possible, it is shaped into a cylindrical form or semi-cylinder.

  Fiber optic data and lighting are in the pocket via a cable wrapped and protected with flexible textiles (preferably with D30 as the outer synthetic layer or other impact resistant synthetic components) Lighting, power supply (preferably wireless), and data processing unit integrated in the intelligent textile synthetic wearable on or at the user's body and thus flattened, weight dispersed and balanced Delivered from

  2. When optical fiber (data, light and optionally power) cables are integrated with the synthetic willow glass frame, this optical fiber as a composite, at the data input point for E-O data transfer, and at the illumination insertion point on the display surface (Preferably by more expensive and unnecessary heat sealing).

  3. The display frame structural element in this version is also a material system of willow glass or willow glass type, and an optional additional component, but instead a solid that forms an optical form factor (binoculars pair or continuous visor) It may be a glass or polymer lens, which is a thin film composite layer following a lens-type preform that helps form the desired surface geometry, and compression ribs may also be used to achieve proper bending.

  4. The sequence of functional optics is a preferred option as seen in both the proposed structural and substrate systems after the first filter and at its most complex stage, the light guiding / confining channels are airgel tensioned As part of the membrane matrix, optical channel elements such as optical fibers will be realized. Or from a fixed (or semi-flexible) light channel for infrared pass-through to an infrared generative channel, and a visible pass-through channel, permeating the airgel into the cavity and the space between and including the airgel under positive pressure A hollow infrared / near infrared rigid shell may be used, which will provide a very powerful, low density lightweight reinforced structural system. Aerogel filament composites have been developed commercially and this category of synthetic aerogel systems continues to advance, offering a wide range of material options for silica and other aerogels, and now low cost manufacturing methods (Cabot, Aspen Aerogel Etc).

  5. An additional option and / or that can be used in hybrid form in Willow glass is a graphene-CNT (carbon nanotube) functional structural system (alone or in combination with but preferably synthesized with airgel).

  6. As graphene is further developed or functional electronics and photonics features, graphene layers or multilayers are formed either on thinned Willow glass substrate or in a sandwich system with aerogel for electronic interconnection Graphene and CNT are mixed, an optical fiber and a planar waveguide are combined on glass for optical interconnection, and otherwise combined with an SOG system element, and another heterogeneous material system (heterologous CMOS + on SOG) In the case of a system, post "pure" CMOS) would be the preferred structural implementation.

  7. In the near-term near future, graphene, CNT, and preferably a combination of graphene and CNT as a compression element (alone or in combination with rolled Willow glass and optional airgel cell sandwich) are superior It provides a suitable lightweight integrated structural system with different substrate quality. Thus, for both on-board processor, sensor deployment, and dense pixel signal processing array layers, semi-flexible willow glass, or similar glass products that may be developed by Asahi, Schott and others, and preferred Although less, in the short run, polymers or polymer glass hybrids can also function as deposition substrates.

  IV. Other mobile or semi-mountable form factors (such as tablets) may also implement many of the mobile AR and VR solutions given full application with the preferred HMD form factor.

  Although specific embodiments have been disclosed herein, they limit the application and scope of the proposed novel image display and projection based on the manipulation and stage resolution required for pixel modulation and the separate optimization. It should not be interpreted as.

  The above systems and methods have been described in general terms as an aid to understanding the details of the preferred embodiments of the present invention. In the description herein, numerous specific details, such as examples of components and / or methods, are provided in order to provide a thorough understanding of embodiments of the present invention. Several features and advantages of the present invention are realized in such a mode and are not required in any case. However, one skilled in the relevant art will recognize that certain embodiments of the present invention may be practiced without one or more of the specific details, or with other devices, systems, assemblies, methods, components, materials, parts and / or parts. You will recognize that it is feasible, as well as others. In other instances, well-known structures, materials, or operations have not been specifically shown or described to avoid obscuring aspects of embodiments of the present invention.

  References to “one embodiment,” “an embodiment,” or “specific embodiment” throughout this specification may be made to the specific feature, structure or features described in connection with that embodiment. It is meant to be included in at least one embodiment of the invention, and not necessarily included in all embodiments. Thus, the phrases “in one embodiment,” “in an embodiment,” or “in a particular embodiment” appear in various places throughout the specification, although the same embodiment is necessarily the same. Does not refer to Furthermore, the particular features, structures or characteristics of particular embodiments of the present invention may be combined in any suitable manner with one or more other embodiments. Other variations and modifications of the embodiments of the invention described and illustrated herein are possible in view of the teachings herein, and should be considered as part of the spirit and scope of the invention. Should be understood.

  Also, one or more of the elements shown in the drawings / figures may be useful in accordance with the particular application, in a more separated or integrated manner, or in certain cases removed or otherwise inoperable It will also be appreciated that it can be implemented as well.

  In addition, any signal arrows in the drawings / figures are to be regarded as illustrative only, and not limiting, unless otherwise noted. Further, as used herein, the term "or" is generally intended to mean "and / or" unless otherwise indicated. Combinations of components or steps are also to be regarded as being annotated, where the term is predicted as unclear in its ability to separate or combine.

  As used in the description herein, and throughout the claims that follow, "a", "an", and "the" refer to a plurality of unless indicated otherwise in the context. Including mention. Also, as used in the description herein, and throughout the claims that follow, unless the context indicates otherwise, the meaning of "in" is "in ( Including ") and" on ".

  The foregoing description of the disclosed embodiments of the present invention, including those described in the abstract, is intended to be exhaustive or to limit the precise forms disclosed herein. There is not. Specific embodiments of the present invention and examples for the present invention are described herein for illustrative purposes only, and as one of ordinary skill in the relevant art would recognize and understand, the spirit and scope of the present invention Various equivalent modifications are possible within the scope. As noted above, these modifications can be made to the invention in light of the foregoing description of the illustrated embodiments of the invention and are to be included within the spirit and scope of the invention.

  Thus, while the present invention has been described herein with reference to particular embodiments thereof, the above disclosure is intended to be flexible, various changes and substitutions, and in some cases, various modifications. It will be understood that without departing from the scope and spirit of the present invention as specified, several features of the embodiments of the present invention will be utilized without corresponding use of other features. I will. Accordingly, many modifications may be made to adapt a particular situation or material to the essential scope and spirit of the present invention. It is intended that the invention not be limited to the specific terms used in the following claims and / or to the specific embodiments disclosed as the best mode contemplated for carrying out the invention. The present invention will include all embodiments and equivalents that fall within the scope of the appended claims. Accordingly, the scope of the present invention should be determined solely by the appended claims.

Claims (10)

A first interface for creating a first set of channelized image composition signals from a real world environment;
A second interface for creating a second set of channelized image composition signals from the synthetic world environment;
A signal processing matrix coupled to the interface, configured to channelize, process, interleave, and deliver the channelized image composition signal as a processed set of channelized image composition signals. Signal processing matrix of the separated optical channel,
A set of signal manipulation structures coupled to the signal processing matrix, configured to create a set of display image primitives for a human vision system from the processed set of channelized image composition signals; Photonic augmented reality system, comprising a set of signal manipulation structures. A photonic system for visualizing an operation world including a synthetic world in a virtual reality mode,
An augmentor for creating a set of channelized composite world image composition signals from the composite world, wherein each set of channelized composite world image composition signals has an augmentor set of desirable attributes, An augmentor comprising a set of channelized composite world image composition signals in an output set of channelized augmentor image composition signals;
The desired attributes for each of the channelized augmentor image composition signals that create an output set of channelized visualizer image composition signals each having a visualizer set with desirable attributes, the visualizer coupled to the augmentor. Processing the output set of the channelized augmentor image composition signal to modify frequency / wavelength modulation or frequency / wavelength conversion attributes from an augmentor set of
An output constructor coupled to the visualizer, the output constructor creating a set of display image primitives from the output set of the channelized visualizer image composition signal.   Each of the desirable attribute augmentor sets includes frequency / wavelength attributes of each of the channelized composite world image composition signals, and all of the frequency / wavelength attributes of the desirable attribute augmentor sets are human in the electromagnetic spectrum. The frequency / wavelength modulation or the frequency / wavelength conversion attributes are all in the invisible portion relative to the vision system and have the frequency / wavelength attribute in the visible portion relative to the human vision system in the electromagnetic spectrum A photonic system according to claim 2, wherein a visualizer set of the desired attributes is created. The operation world further includes the real world of the augmented reality mode,
A real world interface for creating a set of channelized real world image composition signals from the real world, wherein each of the set of channelized real world image composition signals has a real world set of desired attributes, the reality Further equipped with a world interface,
The augmentor receives the set of channelized real world image composition signals and selectively selects the set of channelized real world image composition signals in an output set of the channelized augmentor image composition signals The photonic system according to claim 2, comprising.   The real world sets of the desired attributes each include the frequency / wavelength attributes of each of the channelized real world image composition signals, and the augmentor sets of the desired attributes are each of the channelized composite world image composition signals. All of the frequency / wavelength attributes of the augmenter set of the desired attributes, including frequency / wavelength attributes of the electromagnetic spectrum, in an invisible part relative to the human visual system in the electromagnetic spectrum and the frequency / wavelength modulation or the frequency / wavelength 5. A photonic according to claim 4, wherein all wavelength conversion attributes create a visualizer set of the desired attributes having the frequency / wavelength attributes that are in the visible part of the electromagnetic spectrum relative to the human vision system. system.   The real world interface converts the composite synthesis set of the electromagnetic wave surface of the real world into the set of channelized real world image composition signals, and the composite synthesis set of the electromagnetic wave surface is in the visible portion of the electromagnetic spectrum. And said visible portion of said electromagnetic spectrum comprising a wavefront having a frequency / wavelength in said invisible portion of said electromagnetic spectrum, and wherein said real world interface contributes a set of said channelized real world image composition signals 6. A photonic system according to claim 5, comprising an input structure for suppressing the input of the wavefront having.   The real world interface converts the composite synthesis set of the electromagnetic wave surface of the real world into the set of channelized real world image composition signals, and the composite synthesis set of the electromagnetic wave surface is in the visible portion of the electromagnetic spectrum. And including a wavefront having a frequency / wavelength in the invisible portion of the electromagnetic spectrum, the real world interface contributing the invisible portion of the electromagnetic spectrum to contribute to the set of channelized real world image composition signals. Claim including an input structure for suppressing the input of the wavefront, and wherein the real world interface converts and maps the wavefront in the visible portion of the electromagnetic spectrum into a signal in the invisible portion of the electromagnetic spectrum. Photonic system described in 5. Creating a first set of channelized image composition signals from the real world environment;
Creating a second set of channelized image composition signals from the synthetic world environment;
Processing the channelized image composition signal as a processed set of channelized image composition signals using a separated light channel signal processing matrix;
Creating a set of display image primitives for a human vision system from the processed set of channelized image composition signals.   Said device substantially as disclosed herein.   Said method substantially as disclosed herein.

Images