JP4889036B2 - Image processing apparatus and image processing method - Google Patents

Description

  The present invention relates to a technique for providing mixed reality.

  In recent years, a mixed reality, so-called MR (Mixed Reality) technology is known as a technology for seamlessly combining the real world and the virtual world in real time. The following techniques are known as one of MR techniques. That is, a video camera or the like captures a subject that substantially matches the subject observed from the pupil position of the wearer of a video see-through HMD (Head Mounted Display). Then, a mixed reality image in which CG (Computer Graphics) is superimposed and displayed on the captured image is provided to the wearer of the HMD.

  FIG. 7 is a block diagram showing a functional configuration of a general mixed reality presentation system using a video see-through HMD.

  Reference numeral 701 denotes a video see-through type HMD. The HMD 701 includes an imaging unit 703, a display unit 704, a three-dimensional position / orientation sensor 705, and an I / F (interface) 706.

  Reference numeral 702 denotes an image processing apparatus that generates a virtual space image (CG) based on the three-dimensional position and orientation information received from the HMD 701 and performs synthesis processing with the captured image received from the HMD 701. The image processing apparatus 702 is generally configured by an apparatus having a high-performance arithmetic processing function and a graphic display function such as a PC (personal computer) or a workstation. The image processing apparatus 702 includes an I / F 707, a position / orientation information generation unit 708, a CG drawing synthesis unit 710, and a content DB (database) 709.

  First, the operation of each part constituting the HMD 701 will be described.

  The imaging unit 703 captures an image of the outside world that substantially matches the line-of-sight position of the user wearing the HMD 701 on the head. The imaging unit 703 includes a right-eye imaging device, a left-eye imaging device, an optical system, and a DSP (Digital Signal Processing) for subsequent image processing in order to capture a stereo image.

  The display unit 704 is for displaying a right-eye MR image and a left-eye MR image sent from the image processing apparatus 702. Therefore, the display unit 704 includes a right-eye display device, a left-eye display device, and an optical system. As each display device, a small liquid crystal display or a retinal scan type device using MEMS (Micro Electro-Mechanical Systems) is used.

  The three-dimensional position and orientation sensor 705 is for measuring its own position and orientation. As the three-dimensional position / orientation sensor 705, a magnetic sensor or a gyro sensor (acceleration, angular velocity) is used.

  The I / F 706 is used to perform data communication with the image processing apparatus 702. The captured image captured by the imaging unit 703 and the position and orientation information measured by the three-dimensional position and orientation sensor 705 are the I / F 706. Is transmitted to the image processing apparatus 702 via / F706. Further, the MR image transmitted from the image processing apparatus 702 is received via this I / F 706. For the I / F 706, an optical fiber such as a USB or IEEE 1394 metal line or Gigabit Ethernet (registered trademark), which is required to have real-time characteristics and can transmit a large amount of capacity, is used.

  Next, the operation of each part constituting the image processing apparatus 702 will be described.

  The I / F 707 is used for data communication with the HMD 701, and the MR image generated by the image processing apparatus 702 is transmitted to the HMD 701 via the I / F 707. The captured image and position / orientation information transmitted from the HMD 701 are received via the I / F 707.

  The position and orientation information generation unit 708 generates position and orientation information indicating the position and orientation of the eyes of the user wearing the HMD 701 based on the position and orientation information transmitted from the HMD 701. As another method for generating position and orientation information indicating the position and orientation of the user's eyes, for example, there is a method using an image captured by the imaging unit 703, for example.

  In the content DB 709, data related to each virtual object constituting the virtual space is stored.

  The CG drawing composition unit 710 constructs a virtual space using data stored in the content DB 709, and the constructed virtual space is viewed from the viewpoint having the position and orientation indicated by the position and orientation information generated by the position and orientation information generation unit 708. The viewed image is generated as a virtual space image. Then, an MR image is generated by synthesizing the generated virtual space image on the captured image received from the HMD 701 via the I / F 707. The generated MR image is transmitted to the HMD 701 via the I / F 707.

  The generation of the virtual space image is performed based on the measurement result by the three-dimensional position and orientation sensor 705 as described above, and the virtual space image is superimposed with the captured image as a background. Here, it is preferable that the timing at which the position / orientation information used for generating the virtual space image is measured substantially coincides with the imaging timing of the captured image on which the virtual space image is to be superimposed. Therefore, when taking into consideration effects such as transmission delays between systems and processing time that impair the real-time property, this virtual space image is applied to the captured image captured at the timing closest to the timing at which the virtual space image was generated. It is preferable to superimpose.

  By the processing with the configuration described above, a mixed real world in which the real world and the virtual world are seamlessly fused can be provided to a user wearing the HMD 701 on the head. A general MR technique and system configuration are disclosed in Patent Document 1.

  Here, in order to increase the sense of reality and immersion in the mixed reality world, it is possible to increase the field of view and widen the field of view. Since the optical system lens distortion relatively increases as the display angle of view is optically widened, it is necessary to correct this lens distortion in order to present a display image without a sense of incongruity. As one method of correcting lens distortion, there is a method of electrically correcting an input image to the optical system in advance.

  FIG. 8A is a diagram illustrating an example of a two-dimensional distortion of the optical system. As shown in FIG. 8A, vertical and asymmetric distortion may occur depending on the configuration of the optical system. Therefore, when a rectangular image displayed in the rectangular display area is observed through an optical system having distortion as shown in FIG. 8A, it is observed as a distorted image as shown in FIG. 8A. As a method of electrically canceling such distortion, there is a method of preparing an image distorted in advance so as to cancel the amount of optical distortion.

  FIG. 8B is a diagram illustrating an image obtained when a distortion reverse to the distortion illustrated in FIG. 8A is applied to the rectangular image. Therefore, when the image shown in FIG. 8B is observed through the optical system having the distortion shown in FIG. 8A, the original rectangular image can be observed as shown in FIG. 8C. FIG. 8C is a diagram illustrating an original rectangular image.

  FIG. 9 is a block diagram illustrating a functional configuration of the mixed reality presentation system in which a configuration in which an image is deformed before display in consideration of optical distortion is added to the configuration illustrated in FIG. The configuration shown in FIG. 9 is different from the configuration shown in FIG. 7 in that an HMD 901 in which a distortion correction unit 911 is added to the HMD 701 is used instead of the HMD 701.

  Depending on the configuration of the optical system, a distortion in which the aspect between the vertical direction and the horizontal direction is shifted occurs. Therefore, the distortion correction unit 911 performs optical distortion of the optical system included in the display unit 704 from the MR image received via the I / F 706. Cut out the part determined based on the characteristics. Then, image mapping for mask processing and optical distortion correction is performed.

FIG. 8D is a diagram illustrating a relationship between an image when the image is electronically distorted in consideration of optical distortion and a display area. Since the display image is mapped to the display area, it can be seen that the upper part of the display image (the area indicated by the oblique lines) is cut off. In FIG. 8D, the upper part of the image is cut out, but depending on the optical distortion, either the top, bottom, left, or right, or the combined area is cut out.
Japanese Patent Laid-Open No. 11-88913 (FIG. 7, paragraph [0035])

  However, the conventional techniques described above have the following problems.

  Although the hatched area in FIG. 8D is a clipped area that is not a display target, the image processing apparatus 702 also transmits data of the area to the HMD 701. That is, even image information that is not originally necessary is transmitted every frame.

  Further, the image processing apparatus 702 needs to secure a communication band different from the image information of the MR image in order to transmit the display image control information used for controlling the display on the display unit 704 to the HMD 701. Further, when the display image is controlled in real time using the display image control information, a configuration in which the display image control information and the display image are temporally linked is necessary.

  The present invention has been made in view of the above problems, and control information for controlling display in the HMD without providing a communication band different from the communication band for image information transmitted to the HMD. An object of the present invention is to provide a technique for transmitting a message to an HMD.

  In order to achieve the object of the present invention, for example, an image processing apparatus of the present invention comprises the following arrangement.

That is, an image processing device connected to a head-mounted display device that presents an image displayed by a display device to a user via an optical system,
Generating means for generating a first image;
Identifying means for identifying a region to be non-displayed in the first image based on an optical distortion characteristic opposite to the optical distortion characteristic of the optical system;
Embedding means for generating a second image by embedding control information used to perform display control in the non-display target area;
Means for transmitting the second image to the head-mounted display device.

  In order to achieve the object of the present invention, for example, an image processing method of the present invention comprises the following arrangement.

That is, an image processing method performed by an image processing device connected to a head-mounted display device that presents an image displayed by a display device to a user via an optical system,
A generating step of generating a first image;
A specifying step of specifying a non-display target region in the first image based on an optical distortion characteristic opposite to an optical distortion characteristic of the optical system;
An embedding step of generating a second image by embedding control information used for performing display control in the non-display target region;
Transmitting the second image to the head-mounted display device.

  According to the configuration of the present invention, control information for controlling display in the HMD can be transmitted to the HMD without providing a communication band different from the communication band for image information transmitted to the HMD. .

  Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings.

[First Embodiment]
FIG. 1 is a block diagram showing a functional configuration of a system according to this embodiment for presenting a user with a mixed reality space image obtained by synthesizing a virtual space image and a real space image. As shown in FIG. 1, the system according to this embodiment includes an HMD 101 and an image processing apparatus 102.

  First, the HMD 101 will be described. The HMD 101 is an apparatus used as an example of a head-mounted display device, and a head-mounted display device other than the HMD 101 may be used as long as the HMD 101 has a function described later as a function of the HMD 101.

  The HMD 101 includes an imaging unit 103, a display unit 104, a three-dimensional position and orientation sensor 105, an I / F 106, a display image control information decoding unit 113, a distortion correction unit 111, and a lens distortion information storage table 115. Hereinafter, each of these parts constituting the HMD 101 will be described.

  The imaging unit 103 captures a moving image of the real space that can be seen according to its own position and orientation, and the captured images of each frame (captured image and real space image) are sequentially transmitted to the I / F 106 at the subsequent stage. . In the present embodiment, the imaging unit 103 includes a right-eye imaging device, a left-eye imaging device, an optical system, and a DSP (Digital Signal Processing) for subsequent image processing in order to capture a stereo image. Yes. Below, unless it is the description specialized for one eye, the imaging part 103 with respect to each eye is demonstrated collectively.

  The display unit 104 is for displaying a result of correcting the MR image for the right eye and the MR image for the left eye sent from the image processing apparatus 102 by the distortion correction unit 111. Therefore, the display unit 104 includes a right-eye display device, a left-eye display device, and an optical system that optically deforms an image displayed by the display device and presents the image to the user's eyes. As each display device, a small liquid crystal display or a retinal scan type device using MEMS is used. The display unit 104 having such a configuration is attached to the HMD 101 so as to be positioned in front of the eyes of the user wearing the HMD 101 on the head. The display by the display unit 104 is controlled based on display image control information (control information) acquired from the MR image received from the image processing apparatus 102 by the display image control information decoding unit 113 via the I / F 106.

  The three-dimensional position and orientation sensor 105 measures its own position and orientation. For example, when a magnetic sensor is used as the three-dimensional position and orientation sensor 105, a transmitter that generates a magnetic field is installed at a predetermined position in the real space. The three-dimensional position / orientation sensor 105 detects a change in magnetism according to its own position / orientation in the magnetic field generated by the transmitter, and the detection result is sent to a sensor control device that controls the operation of the transmitter. The sensor control device obtains the position and orientation of the three-dimensional position and orientation sensor 105 in the sensor coordinate system based on the detection result. The obtained position and orientation are sent to the I / F 106. Here, the sensor coordinate system is a coordinate system in which the position of the transmitter is the origin, and the three axes orthogonal to each other at the origin are the x axis, the y axis, and the z axis. The sensor applicable as the three-dimensional position and orientation sensor 105 may be other than this, and for example, an optical sensor, an ultrasonic sensor, a gyro sensor (acceleration, angular velocity), or the like may be used.

  The display image control information decoding unit 113 acquires control information from the MR image received from the image processing apparatus 102 via the I / F 106, and performs various image processing on the received MR image based on the acquired control information. Do. The MR image after the image processing and the control information are sent to the subsequent distortion correction unit 111.

  The distortion correction unit 111 deforms the MR image received from the display image control information decoding unit 113 based on the optical distortion characteristic opposite to the distortion characteristic (optical distortion characteristic) of the optical system included in the display unit 104. Is generated. Then, the generated MR image is sent to the display unit 104 in the subsequent stage.

  The lens distortion information storage table 115 holds lens distortion information indicating an optical distortion characteristic opposite to the distortion characteristic of the optical system included in the display unit 104, and the lens distortion information is appropriately transmitted to the outside as required. . Details of the lens distortion information will be described later.

  The I / F 106 is for performing data communication with the image processing apparatus 102, and is optical such as USB or IEEE 1394 metal wire or Gigabit Ethernet (registered trademark) that requires real-time performance and can transmit large volumes. It is composed of fiber. The real space image captured by the imaging unit 103, the position / orientation information as a result measured by the three-dimensional position / orientation sensor 105, and the lens distortion information held by the lens distortion information storage table 115 are processed through the I / F 106. Transmitted to the device 102.

  Next, the image processing apparatus 102 will be described. The image processing apparatus 102 generates a virtual space image based on the position and orientation information transmitted from the HMD 101, and generates an MR image in which the generated virtual space image is superimposed on the real space image transmitted from the HMD 101. Then, control information to be described later is embedded on the generated MR image. Then, the MR image in which the control information is embedded is transmitted to the HMD 101. The image processing apparatus 102 is generally configured by an apparatus having a high-performance arithmetic processing function or graphic display function such as a personal computer or a workstation.

  The image processing apparatus 102 includes an I / F 107, a position / orientation information generation unit 108, a CG drawing synthesis unit 110, a content DB 109, a display image control information encoding unit 112, and a lens distortion information analysis unit 114.

  The I / F 107 is used for data communication with the HMD 101. Like the I / F 106, the I / F 107 is required to have real-time characteristics and can transmit a large capacity, such as a USB or IEEE 1394 metal line, Gigabit Ethernet (registered trademark), or the like. It is comprised by the optical fiber of.

  The position and orientation information generation unit 108 obtains the position and orientation of the imaging unit 103 from the position and orientation information sent from the HMD 101. The position and orientation information sent from the HMD 101 indicates the position and orientation of the three-dimensional position and orientation sensor 105. Therefore, for example, the position and orientation relationship between the imaging unit 103 and the three-dimensional position and orientation sensor 105 is obtained in advance, and the position and orientation relationship of the imaging unit 103 is obtained by adding the position and orientation relationship to the position and orientation of the three-dimensional position and orientation sensor 105. You can ask for information. Of course, there are other methods for obtaining the position / orientation information of the imaging unit 103, for example, there is a method for obtaining the position / orientation information of the imaging unit 103 using an image captured by the imaging unit 103.

  In the content DB 109, data about each virtual object constituting the virtual space is registered. For example, when the virtual object is composed of polygons, the color and normal vector data of each polygon, the coordinate data of each vertex constituting the polygon, and the like are registered in the content DB 109.

  The CG drawing composition unit 110 constructs a virtual space using data registered in the content DB 109. Then, an image that can be seen from the viewpoint having the position and orientation indicated by the position and orientation information generated by the position and orientation information generation unit 108 is generated as the virtual space image. Then, the generated virtual space image is synthesized on the real space image received from the HMD 101 via the I / F 107, thereby generating an MR image. The generated MR image is sent to the display image control information encoding unit 112 at the subsequent stage.

  The lens distortion information analysis unit 114 acquires the lens distortion information from the lens distortion information storage table 115 in the HMD 101, and uses the acquired lens distortion information to output control information in the MR image generated by the CG drawing synthesis unit 110. Specify the area to embed.

  Here, the control information is used to perform display control on the display unit 104 and control related to an image to be displayed. The control information includes, for example, information for controlling the image quality such as brightness, color, and gamma of the display image, information for controlling the screen itself, such as the display position and size of the display image, and a warning message superimposed on the MR image. Information that controls the superimposed characters is included.

  The acquisition of the lens distortion information by the lens distortion information analysis unit 114 is preferably performed in an initialization process immediately after the image processing apparatus 102 is turned on. This is because the HMD having different lens distortion information can cope with the connection to the image processing apparatus 102.

  The display image control information encoding unit 112 embeds control information in the region specified by the lens distortion information analysis unit 114 in the MR image received from the CG drawing synthesis unit 110. Details of the process of embedding control information in the MR image will be described later. Then, the display image control information encoding unit 112 transmits the MR image in which the control information is embedded to the HMD 101 via the I / F 107.

  FIG. 3A is a flowchart of a series of processes for the image processing apparatus 102 to generate an MR image in which control information is embedded and transmit it to the HMD 101.

  In step S <b> 300, the lens distortion information analysis unit 114 accesses the HMD 101 and acquires lens distortion information registered in the lens distortion information storage table 115.

  Here, the lens distortion information is information indicating an optical distortion characteristic opposite to the optical distortion characteristic of the display unit 104. Therefore, if lens distortion information is used, for example, the rectangular image shown in FIG. 8C can be transformed into the image (image B) shown in FIG. 8B. Thereby, an image observed through the optical system having the optical distortion characteristics shown in FIG. In other words, for each pixel constituting the image C, a corresponding pixel on the image B can be specified.

  In step S301, the lens distortion information analysis unit 114 refers to the lens distortion information acquired in step S300. Then, when the MR image is deformed based on the optical distortion characteristic opposite to the optical distortion characteristic of the optical system included in the display unit 104, it is included in a pixel position range that is predetermined as a non-display target area. Identify the area. Taking FIGS. 8C and 8D as an example, in step S301, an area corresponding to the hatched portion (a pixel position range predetermined as a non-display target area) in the image shown in FIG. 8C is obtained. It will be. That is, in step S301, area information indicating an area to be non-displayed in the MR image is obtained in advance.

  The processes in steps S300 and S301 are preferably performed in an initialization process immediately after the image processing apparatus 102 is turned on.

  On the other hand, as described above, from the HMD 101, position / posture information indicating the position / posture of the three-dimensional position / posture sensor 105 measured by the three-dimensional position / posture sensor 105 and the real space image picked up by the image pickup unit 103 are I / O. Sent via F106. Accordingly, in step S302, the transmitted real space image and position / orientation information are received (acquired) via the I / F 107. The real space image received via the I / F 107 is input to the CG drawing synthesis unit 110, and the position / orientation information received via the I / F 107 is input to the position / orientation information generation unit 108.

  In step S303, first, the position / orientation information generation unit 108 uses the position / orientation information received via the I / F 107 and the position / orientation relationship between the imaging unit 103 and the three-dimensional position / orientation sensor 105 as described above. Thus, the position and orientation information of the imaging unit 103 is obtained. After that, the CG drawing synthesis unit 110 constructs a virtual space using data registered in the content DB 109 and generates an image obtained by viewing the virtual space from the position and orientation of the imaging unit 103 as a virtual space image. . Then, the generated virtual space image is combined with the real space image received via the I / F 107 to generate an MR image (first image) as a combined image. The generated MR image is sent to the display image control information encoding unit 112 at the subsequent stage.

  Next, when the control process is performed on the MR image generated at step S303, the process proceeds to step S305 via step S304. When the control process is not performed, the process proceeds to step S390 via step S304. Proceed to Whether to perform control processing on an MR image in a certain frame may be set in advance, or may be appropriately determined by detecting user instruction input or the like.

  In step S390, the display image control information encoding unit 112 specifies the header information indicating that the control process is not performed on the MR image generated in step S303 in step S301 (in the first image). Embed the area to be displayed. The embedding process here may be a process of simply replacing the data constituting the non-display target area with the data constituting the header information, or may be the process of embedding as a digital watermark. In addition, when there are a plurality of non-display target areas, they may be determined in advance, for example, embedded in the upper left corner area.

  On the other hand, in step S305, the display image control information encoding unit 112 embeds control information in the non-display target area specified in step S301. The embedding process in step S305 is performed in the same manner as in step S390 except for the information to be embedded.

  Next, in step S391, the display image control information encoding unit 112 converts any one (second image) of the MR images in which the control information is embedded in step S305 or the header information in step S390 (second image) to I / O. It transmits to HMD101 via F107. In the MR image, the data of each pixel constituting the non-display target region is, for example, data indicating a pixel value of 0 (for example, when the MR image is an RGB image, R = G = B = (Data indicating 0).

  Next, when it is detected that an instruction to end the process is input, or when a condition for ending the process is satisfied, the process ends. On the other hand, if the instruction to end this process has not been input and the conditions for ending this process have not been satisfied, the process returns to step S302 via step S392, and the process from step S302 onward for the next frame. I do.

  FIG. 2 is a diagram for explaining the control information embedding process performed in step S305.

  2A shows an MR image 201 deformed based on an optical distortion characteristic opposite to the optical distortion characteristic of the optical system included in the display unit 104, and a display screen 202 of the display device included in the display unit 104. FIG. Showing the relationship. Since the distortion of the aspect of the up-down direction and the left-right direction is caused by the lens of the optical system, the upper part of the MR image 201 after the deformation is an area that is not actually displayed. Data of pixels constituting a region not to be displayed is replaced with data indicating a certain pixel value, or such a region itself is physically shielded so that the region cannot be observed.

  FIG. 2B shows a non-display target region in the undeformed MR image. The hatched area shown in FIG. 2B corresponds to the hatched area shown in FIG. 2A, that is, the area to be hidden.

  FIG. 2 (c) shows an MR image when control information is embedded in a region that is a non-display target in an undeformed MR image. Since the control information is embedded in the image, it is desirable to manage whether or not various processes can be executed on the image, and the size and attributes of the control information.

  FIGS. 10A and 10B show a configuration example of header information constituting the control information and superimposed character control information. Of course, the information that can be included in the control information is not limited to that shown in FIGS. 10A and 10B.

  FIG. 10A is a diagram illustrating a configuration example of header information.

  The “executability” indicates whether or not various display controls are performed based on the control information.

  “Size” means the total data size of the control information, how many frames of the control information are embedded when the control information cannot be embedded in one frame image, and in that case, in what frame the current header information is embedded It is shown that.

  The “attribute” indicates whether display control based on the control information is image control, screen control, or superimposed character control.

  FIG. 10B is a diagram illustrating a configuration example of superimposed character control information.

  “Display time” indicates the length of time for displaying characters.

  “Display position” indicates the display position (coordinate value) of the character to be displayed.

  “Display color” indicates the color of a character to be displayed.

  “Character size” indicates the size (number of points) of characters to be displayed.

  “Display content” indicates a character string to be actually displayed.

  Although not particularly mentioned in the above description, it is assumed that information (code) unique to the header information is recorded at the head portion of the header information. This is used when searching for where the header information is recorded in the MR image on the side of extracting the control information.

  Next, a series of processing performed by the HMD 101 for receiving the MR image transmitted from the image processing apparatus 102 and displaying the MR image on the display unit 104 will be described below with reference to FIG. 4A showing a flowchart of the processing. To do.

  In step S400, the lens distortion information registered in the lens distortion information storage table 115 requested from the lens distortion information analysis unit 114 in step S300 is transmitted to the image processing apparatus 102 via the I / F 106. To do. The processing in step S400 is preferably performed by initialization processing immediately after the HMD 101 is turned on.

  Next, in step S <b> 401, the real space image captured by the image capturing unit 103 and the position / orientation information measured by the three-dimensional position / orientation sensor 105 are transmitted to the image processing apparatus 102 via the I / F 106.

  In step S <b> 402, the display image control information decoding unit 113 acquires the MR image embedded with the control information transmitted from the image processing apparatus 102 according to the flowchart of FIG. 3A via the I / F 106.

  In step S403, the display image control information decoding unit 113 searches the MR image acquired in step S402 to search for information (code) unique to the header information. That is, in step S403, the control information embedded in the MR image is retrieved and acquired. Accordingly, in step S403, extraction processing according to the control information embedding method is performed.

  Then, as a result of extracting the control information, the display image control information decoding unit 113 refers to “executability” in the header information in the extracted control information and performs display control (value is “0”) or display control. Is not performed (value is “1”). If it is determined that display control is to be performed, the process proceeds to step S404. If display control is not to be performed, the process proceeds to step S490.

  In step S404, the display image control information decoding unit 113 refers to the header information and superimposed character control information in the control information extracted in step S403, and performs corresponding display control processing on the MR image acquired in step S402. . Then, the distortion correction unit 111 generates an MR image obtained by deforming the MR image as the display control processing result based on the optical distortion characteristic opposite to the optical distortion characteristic of the optical system included in the display unit 104, and the display unit 104. Is sent out. That is, the MR image is mapped on the display area (on the display screen) of the display unit 104.

  Next, when it is detected that an instruction to end the process is input, or when a condition for ending the process is satisfied, the process ends. On the other hand, if the instruction to end this process has not been input and the conditions for ending this process are not satisfied, the process returns to step S401 via step S490, and the processes in and after step S401 are performed for the next frame. I do.

  As described above, according to the present embodiment, the control information is embedded in the region that is not displayed in the MR image, so that there is no influence on the image quality for the display region, and the MR image and the MR image are transmitted within the transmission band of the MR image. Both control information can be transmitted. In addition, since the control information and the MR image are transmitted to the HMD 101 in a one-to-one relationship, the control process based on the control information and the MR image display process in which the control information is embedded can be easily synchronized. Can do.

[Second Embodiment]
In the first embodiment, it is premised that the data size constituting the non-display target area in the MR image is larger than the data size of the control information. However, when the data size constituting the non-display target area in the MR image is equal to or smaller than the data size of the control information, the control information is divided, and the divided division control information is divided into a plurality of consecutive frames. It may be dispersed and embedded in each MR image.

  In this case, the process in step S305 in the flowchart of FIG. 3A is as follows.

  FIG. 3B is a flowchart showing details of processing in step S305 according to the present embodiment. In the present embodiment, the display image control information encoding unit 112 performs the process in step S305.

  First, in step S306, the data size X constituting the area specified in step S301 is compared with the data size Y of the control information. As a result of such size comparison, if X> Y, the process proceeds to step S310, and if X ≦ Y, the process proceeds to step S308.

  When the process proceeds from step S306 to step S310, in step S310, control information is embedded as in the first embodiment.

  On the other hand, in step S308, the control information is divided into units that are smaller than the data size X, but what is actually divided is superimposed character control information, and each divided unit includes the original information. New header information is issued based on the header information. The newly issued header information is only the item “size” in the original header information. In step S310, the division control information in which the newly issued header information is added to the divided unit is embedded in the area specified in step S301. That is, the division control information is divided so that the data size is smaller than the data size X.

  On the other hand, in step S404 according to the present embodiment, the following processing is performed.

  FIG. 4B is a flowchart showing details of the processing in step S404 according to the present embodiment.

  First, in step S405, the display image control information decoding unit 113 refers to the header information in the control information extracted in step S403, and the “number of divided frames” is 2 or more, that is, whether the control information is the divided control information. Judge whether or not. As a result of the determination, if the “number of divided frames” is 1, that is, if the control information is not the divided control information, the process proceeds to step S409, and if it is 2 or more, the process proceeds to step S406. .

  In step S409, the display image control information decoding unit 113 refers to the superimposed character control information in the control information extracted in step S403. In step S410, the display image control information decoding unit 113 performs a corresponding display control process on the MR image acquired in step S402. Then, the distortion correction unit 111 generates an MR image obtained by deforming the MR image as the display control processing result based on the optical distortion characteristic opposite to the optical distortion characteristic of the optical system included in the display unit 104, and the display unit 104. Is sent to.

  On the other hand, in step S406, the display image control information decoding unit 113 holds the division control information extracted in step S403. When the “number of divided frames” in the header information in the obtained division control information becomes equal to “the current frame number in the case of divided frames” in the header information, the process proceeds to step S408 via step S407. move on. That is, when the number of pieces of division control information that can be restored by the division source is collected, the process proceeds to step S408 via step S407. On the other hand, if not yet gathered, this process is terminated to obtain the division control information in the next frame.

  In step S408, the display image control information decoding unit 113 combines all the collected division control information into one. That is, the superimposed character control information in each division control information is merged into one.

  In step S409, the display image control information decoding unit 113 refers to the superimposed character control information merged into one. In step S410, the display image control information decoding unit 113 performs a corresponding display control process on the MR image acquired in step S402. Then, the distortion correction unit 111 generates an MR image obtained by deforming the MR image as the display control processing result based on the optical distortion characteristic opposite to the optical distortion characteristic of the optical system included in the display unit 104, and the display unit 104. Is sent to. That is, the MR image is mapped on the display area of the display unit 104.

[Third Embodiment]
In the first embodiment, the HMD 101 side performs the process of deforming the MR image to be displayed according to the distortion characteristic opposite to the distortion characteristic of the optical system included in the display unit 104, but the image processing apparatus 102. It may be performed on the side.

  FIG. 5 shows the functional configuration of the system when the MR image to be displayed is deformed according to the optical distortion characteristic opposite to the optical distortion characteristic of the optical system included in the display unit 104 on the image processing apparatus side. FIG. Although the system according to the present embodiment is different from the configuration of the system according to the first embodiment, the purpose thereof is the same. That is, the MR image obtained by combining the real space image and the virtual space image is provided in front of the user wearing the HMD on the head.

  5 that are the same as those shown in FIG. 1 are given the same reference numerals, and descriptions thereof are omitted. The HMD 501 has a configuration in which the distortion correction unit 111 is omitted from the HMD 101, and the image processing apparatus 502 has a configuration in which a distortion correction unit 511 that performs the same operation as the distortion correction unit 111 is added to the image processing apparatus 102.

  That is, on the image processing apparatus 502 side, the distortion correction unit 511 deforms the MR image generated by the CG drawing synthesis unit 110 in the same manner as the distortion correction unit 111, and the display image control information encoding unit 112 deforms this. Control information is embedded in the MR image. Then, the deformed MR image in which control information is embedded is transmitted to the HMD 501.

  On the HMD 501 side, control information is simply extracted from this MR image, and this MR image is displayed on the display unit 104 based on this control information without any modification.

  FIG. 6 is a diagram illustrating a relationship between the MR image embedded with the control information transmitted from the image processing apparatus 502 to the HMD 501 and the display screen of the display unit 104. Due to the optical lens of the display unit 104, there is an area that is not actually displayed, which is indicated by diagonal lines in the vertical and horizontal directions. As in the first embodiment, the area not to be displayed is not observed by a means such as filling with data of a certain pixel value or physically shielding light. The other handling of the MR image is the same as that in the first embodiment.

[Fourth Embodiment]
As the image processing apparatuses 102 and 502 shown in FIGS. 1 and 5, it is preferable to use a general PC (personal computer). In this case, as an example of the hardware configuration of such a computer, a hardware as shown in FIG. Hardware configuration.

  FIG. 11 is a block diagram illustrating a hardware configuration example of a computer applicable to the image processing apparatuses 102 and 502.

  The CPU 1101 uses the programs and data stored in the RAM 1102 and the ROM 1103 to control the entire computer, and executes the processes described above as performed by the image processing apparatuses 102 and 502. For example, the CPU 1101 functions as the position / orientation information generation unit 108, the CG drawing synthesis unit 110, the display image control information encoding unit 112, the lens distortion information analysis unit 114, and the distortion correction unit 111.

  The RAM 1102 has an area for temporarily storing programs and data loaded from the external storage device 1106, real space images received from the HMD 101 (501) via the I / F 107, position and orientation information, and lens distortion information. . The RAM 1102 also has a work area used when the CPU 1101 executes various processes. That is, the RAM 1102 can provide various areas as appropriate.

  The ROM 1103 stores setting data of the computer, a boot program, and the like.

  The operation unit 1104 is configured by a keyboard, a mouse, and the like, and can input various instructions to the CPU 1101 by being operated by an operator of the computer.

  The display unit 1105 is configured by a CRT, a liquid crystal screen, or the like, and can display a processing result by the CPU 1101 using an image, text, or the like.

  The external storage device 1106 is a large-capacity information storage device represented by a hard disk drive device. Here, programs and data for causing the CPU 1101 to execute the processing described above as being performed by the OS (operating system) and the image processing apparatus 102 (502) are stored. The program includes a program for causing the CPU 1101 to execute the functions of the position / orientation information generation unit 108, the CG drawing synthesis unit 110, the display image control information encoding unit 112, the lens distortion information analysis unit 114, and the distortion correction unit 111. It is. In addition, the data includes data of each virtual object constituting the virtual space. That is, the content DB 109 is included in the external storage device 1106.

  Programs and data stored in the external storage device 1106 are appropriately loaded into the RAM 1102 under the control of the CPU 1101. The CPU 1101 executes processing using the loaded program and data.

  The I / F 107 is the same as that shown in FIG. 1, and data communication with the HMD 101 (501) is performed via the I / F 107.

  A bus 1108 connects the above-described units.

[Other Embodiments]
Needless to say, the object of the present invention can be achieved as follows. That is, a recording medium (or storage medium) in which a program code of software that realizes the functions of the above-described embodiments is recorded is supplied to the system or apparatus. Needless to say, such a storage medium is a computer-readable storage medium. Then, the computer (or CPU or MPU) of the system or apparatus reads and executes the program code stored in the recording medium. In this case, the program code itself read from the recording medium realizes the functions of the above-described embodiment, and the recording medium on which the program code is recorded constitutes the present invention.

  Further, by executing the program code read by the computer, an operating system (OS) or the like running on the computer performs part or all of the actual processing based on the instruction of the program code. Needless to say, the process includes the case where the functions of the above-described embodiments are realized.

  Furthermore, it is assumed that the program code read from the recording medium is written in a memory provided in a function expansion card inserted into the computer or a function expansion unit connected to the computer. After that, based on the instruction of the program code, the CPU included in the function expansion card or function expansion unit performs part or all of the actual processing, and the function of the above-described embodiment is realized by the processing. Needless to say.

  When the present invention is applied to the recording medium, program code corresponding to the flowchart described above is stored in the recording medium.

It is a block diagram which shows the function structure of the system which concerns on the 1st Embodiment of this invention for showing a user the mixed reality space image which synthesize | combined the image of the virtual space, and the image of the real space. It is a figure explaining the embedding process of the control information performed at step S305. 5 is a flowchart of a series of processes for the image processing apparatus 102 to generate an MR image in which control information is embedded and transmit it to the HMD 101. It is a flowchart which shows the detail of the process in step S305 which concerns on the 2nd Embodiment of this invention. 6 is a flowchart of a series of processes performed in order for the HMD 101 to receive an MR image transmitted from the image processing apparatus 102 and display the MR image on the display unit 104. It is a flowchart which shows the detail of the process in step S404 which concerns on the 2nd Embodiment of this invention. The block diagram which shows the function structure of a system in the case of performing the deformation | transformation according to the optical distortion characteristic opposite to the optical distortion characteristic of the optical system which the display part 104 has with respect to MR image of a display object by the image processing apparatus side. It is. 6 is a diagram illustrating a relationship between an MR image in which control information is embedded, which is transmitted from the image processing apparatus 502 to the HMD 501, and a display screen of the display unit 104. FIG. It is a block diagram which shows the function structure of the general mixed reality presentation system using video see-through type HMD. It is a figure which shows an example of the two-dimensional distortion of an optical system. It is a figure which shows the image obtained when the distortion reverse to the distortion shown to FIG. 8A is given with respect to the rectangular image. It is a figure which shows the original rectangular image. It is a figure which shows the relationship between the image when an image is distorted electronically in consideration of optical distortion, and a display area. It is a block diagram which shows the function structure of a mixed reality presentation system which added the structure which deform | transforms an image before a display in consideration of optical distortion to the structure shown in FIG. It is a figure which shows the structural example of header information. It is a figure which shows the structural example of superimposition character control information. FIG. 3 is a block diagram illustrating a hardware configuration example of a computer applicable to the image processing apparatuses 102 and 502.

Claims (10)

An image processing device connected to a head-mounted display device that presents an image displayed by a display device to a user via an optical system,
Generating means for generating a first image;
Identifying means for identifying a region to be non-displayed in the first image based on an optical distortion characteristic opposite to the optical distortion characteristic of the optical system;
Embedding means for generating a second image by embedding control information used to perform display control in the non-display target area;
An image processing apparatus comprising: means for transmitting the second image to the head-mounted display device. The generating means includes
Means for acquiring a real space image captured by the imaging device;
Means for generating a virtual space image according to the position and orientation of the imaging device;
The image processing apparatus according to claim 1, further comprising: a unit that generates a composite image of the real space image and the virtual space image as the first image.   The image processing apparatus according to claim 2, wherein the imaging device is included in the head-mounted display device. The specifying means is:
When an image obtained by deforming the first image based on the optical distortion characteristic opposite to the optical distortion characteristic of the optical system is mapped on the display screen of the display device, a non-display target is displayed on the mapped image. The image processing apparatus according to claim 1, wherein a region in the first image corresponding to a region set in advance is specified. The embedding means is
Data size of the area to be the non-display target, if the is equal to or less than the data size of the control information comprises means for dividing the control information,
The image processing apparatus according to claim 1, wherein the divided division control information is embedded in the non-display target area of the first image in each successive frame. It said embedding means includes a feature in that with respect to the first image and the optical distortion characteristic of pre-Symbol optics image obtained by Rukoto deformed based on optical distortion characteristics of the reverse, embedding the control information The image processing apparatus according to claim 1. 7. The control information according to claim 1, wherein the control information includes information indicating at least one of display time, display position, display color, character size, and display content. The image processing apparatus described. An image processing method performed by an image processing device connected to a head-mounted display device that presents an image displayed by a display device to a user via an optical system,
A generating step in which the generating means of the image processing device generates a first image;
A specifying step of specifying a region to be non-displayed in the first image based on an optical distortion characteristic opposite to the optical distortion characteristic of the optical system , by the specifying unit of the image processing apparatus ;
An embedding step in which the embedding unit of the image processing device generates a second image by embedding control information used to perform display control on the non-display target region;
And a step of transmitting the second image to the head-mounted display device.   A program for causing a computer to execute the image processing method according to claim 8.   A computer-readable storage medium storing the program according to claim 9.

Images