JP2009087161A - Image processing apparatus and image processing method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a technique for arranging a virtual object imitating a hand in a virtual space according to the state of the hand.
An attribute setting unit obtains a position / posture relationship between a hand position / posture and a gripper position / posture. Of the hand models representing each of a plurality of types of hand states that are stored in advance as hand models in the storage unit 308, model data attributes that are used as targets to be placed in the virtual space are hand models corresponding to the obtained position and orientation relationship The setting unit 306 and the image generation unit 307 select.
[Selection] Figure 3

Description

  The present invention relates to a technique for providing mixed reality.

  Conventionally, there is a system (MR system) using a mixed reality space (MR) in which a virtual space and a real space are fused. MR is a technique that complements and enhances information in both spaces by superimposing a virtual space generated by a computer in a three-dimensionally aligned state and presenting it to an observer. The mixed reality technology is disclosed in Non-Patent Document 1, for example.

  There are various methods for presenting the mixed reality space to the user. One of them is a method of presenting the head mounted on the user's head with an HMD (Head Mount Display).

  Currently, MR systems using HMD are used for design evaluation, assemblyability verification, usability verification, and so on.

When the user's evaluation and verification work is performed, if the user's hand can be grasped, the work can be performed smoothly. For example, when performing usability verification, the verifier wants to check whether the hand of a virtual product button displayed in CG is accessible or accessible. Also, it is desired to check whether the driver gripped by the hand enters the gap between the virtual products displayed in CG, or whether maintenance by the driver is possible. Therefore, CG corresponding to real hands and tools is displayed in the virtual space and the mixed reality space.
H. Tamura, H .; Yamamoto and A.M. Katayama: “Mixed reality: Future dream sense at the border betreal real and virtual worlds,” Computer Graphics and Applications, vol. 21, no. 6, pp. 64-70, 2001. A. Takagi, S .; Yamazaki, Y. et al. Saito, and N.K. Taniguchi: "Development of a stereo video see-through HMD for AR systems", ISAR2000, pp. 68-77, 2000.

  However, in the conventional method of displaying images in the virtual space and the mixed reality space, it is not easy to change the shape of the hand CG corresponding to the movement of the hand when the movement state of the actual hand changes. There wasn't. For example, if the hand in the par state initially has any tool, it will change to a state such as goo. Accordingly, in response to the movement of the actual hand, the shape of the hand CG in the virtual space and the mixed reality space is also desired to be changed from par to goo, but the shape of the hand CG remains par. .

  In order to solve such a problem, there is a method in which an operator performs a hand CG switching input operation. In this method, an operator is required in addition to the user wearing the HMD, and the actual hand The hand CG cannot be switched automatically according to the movement of the camera.

  In addition, there is a method in which a glove-type input device capable of inputting hand movement is attached to the hand, and the hand CG is switched according to the input. However, such an input device is expensive and the weight of the input device is heavy, which places a burden on the user who wears the input device.

  Thus, there is currently no method for displaying a hand CG corresponding to an actual hand motion without using an input by an operator or a hand motion input device.

  In addition, in the conventional method of displaying images in the virtual space and the mixed reality space, an input operation using a keyboard, a mouse, or the like is necessary to switch and display the CG of the tool held by the user. For example, if you want to switch from a CG holding a driver with your hand to a CG holding a pliers with your hand, the user wearing the HMD will operate the tool with his / her hands. The CG of the tool cannot be switched.

  The present invention has been made in view of the above problems, and an object thereof is to provide a technique for arranging a virtual object imitating a hand in a virtual space according to the state of the hand.

  In order to achieve the object of the present invention, for example, an image processing apparatus of the present invention comprises the following arrangement.

That is, means for acquiring the position and orientation of the user's viewpoint,
Means for acquiring the position and orientation of the user's hand;
Means for acquiring a position and orientation of a gripping tool as a target to be gripped by the hand;
A first placement means for placing a virtual object imitating the hand in a virtual space at the position and orientation of the hand;
A second placement means for placing a virtual object imitating the gripping tool in the virtual space at the position and orientation of the gripping tool;
Means for generating an image of the virtual space in which a virtual object is arranged by the first arrangement means and the second arrangement means based on the position and orientation of the viewpoint;
Output means for outputting an image of the virtual space,
The first arrangement means includes
Calculating means for obtaining a position and orientation relationship between the position and orientation of the hand and the position and orientation of the gripping tool;
A virtual object corresponding to the position and orientation relationship is selected as a target to be placed in the virtual space from among virtual objects representing each of a plurality of types of hand states that are held in advance as virtual objects that imitate the hand. And a selection means.

  In order to achieve the object of the present invention, for example, an image processing method of the present invention comprises the following arrangement.

That is, obtaining the position and orientation of the user's viewpoint,
Obtaining a position and orientation of the user's hand;
Obtaining a position and orientation of a gripping tool as a target to be gripped by the hand;
A first placement step of placing a virtual object imitating the hand in a virtual space at the position and orientation of the hand;
A second arrangement step of arranging a virtual object imitating the gripping tool in the virtual space at the position and orientation of the gripping tool;
Generating an image of the virtual space in which a virtual object is arranged in the first arrangement step and the second arrangement step based on the position and orientation of the viewpoint;
An output step of outputting an image of the virtual space,
The first arrangement step includes
A calculation step for obtaining a position and orientation relationship between the position and orientation of the hand and the position and orientation of the gripping tool;
A virtual object corresponding to the position and orientation relationship is selected as a target to be placed in the virtual space from among virtual objects representing each of a plurality of types of hand states that are held in advance as virtual objects that imitate the hand. And a selection step.

  According to the configuration of the present invention, it is possible to arrange a virtual object imitating a hand in the virtual space according to the state of the hand.

  Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings.

[First Embodiment]
First, an outline of the present embodiment will be described.

  FIG. 1A is a diagram illustrating a user's hand observing a mixed reality space obtained by superimposing a virtual space on the real space, and a gripping tool to be gripped by the hand. FIG. 1B is a diagram illustrating a virtual object to be superimposed on the user's hand and a virtual object to be superimposed on the gripping tool.

  In FIG. 1A, 101 is a user's hand (real object), and 104 is a wand as a gripping tool. A jig 103 is attached to the hand 101, and retroreflective markers 102 a to 102 c are attached to the jig 103. That is, the jig 103 functions as a jig for fixing a plurality of retroreflective markers (in the case of FIG. 1A, retroreflective markers 102a to 102c) as one group. As a result, the position and orientation relationship between the plurality of retroreflective markers remains unchanged. Since the jig 103 is attached to the hand 101, the jig 103 moves in conjunction with the hand 101.

  On the other hand, similar retroreflective markers 102 d to 102 g are also attached to the wand 104. Reference numeral 105 denotes a hand holding the wand 104.

  In FIG. 1B, reference numeral 106 denotes a virtual object imitating the user's hand, and more specifically, an open hand, that is, a virtual object imitating the hand 101 shown in FIG. 1A. The virtual object 106 is arranged in the virtual space with the position and orientation of the user's hand.

  Reference numeral 107 denotes a virtual object arranged in the virtual space at the position and orientation of the wand 104, and is a virtual object that imitates a driver in FIG. 1B.

  Reference numeral 108 denotes a virtual object imitating a hand holding the virtual object 107. In the present embodiment, when the user's hand changes from the hand 101 to the hand 105, the virtual object imitating the hand is changed from the virtual object 106 to the virtual object 108 following the change. That is, the virtual object imitating the hand is also changed according to the change in the state of the user's hand. Of course, the actual hand state and the hand state represented by the virtual object imitating the hand are substantially the same.

  Here, conventionally, the virtual object arranged in the position and orientation of the user's hand has nothing to do with the change in the hand state. Therefore, when the user's hand is the hand 101, the virtual object 106 may be arranged in the position and orientation of the hand 101. However, even when the user's hand is the hand 105, the same virtual object 106 is used. Is arranged with the position and orientation of the hand 105. Therefore, as shown in FIG. 2, the virtual object 106 representing the open hand is gripped by the virtual object 107 imitating the driver, and an image of the virtual space with the composition is presented to the user. FIG. 2 is a diagram illustrating a virtual object to be superimposed on the user's hand and a virtual object to be superimposed on the gripping tool.

  In this embodiment, a virtual object to be arranged in the position and orientation of the user's hand is selectively switched according to a change in the hand state, thereby presenting the user with a hand virtual object that substantially matches the actual hand state. To do.

  FIG. 3 is a block diagram showing a functional configuration of the system according to the present embodiment.

  As shown in FIG. 3, the system according to this embodiment includes an HMD 350 as an example of a head-mounted display device, a position / orientation sensor 304, and an image processing device 300.

  First, the HMD 350 will be described. The HMD 350 includes an imaging unit 301 and a display unit 309.

  The imaging unit 301 captures a moving image of a real space that can be seen according to its position and orientation, and the captured images of each frame (real space image) are sequentially input to the image processing apparatus 300.

  The display unit 309 is configured by a liquid crystal screen or the like, and is attached to the HMD 350 so as to be positioned in front of the user wearing the HMD 350 on the head. The display unit 309 displays an image sent from the image processing apparatus 300. Note that the focal points of the optical systems of the imaging unit 301 and the display unit 309 are the same.

  Further, it is assumed that a plurality of retroreflective markers as shown in FIGS. 1A and 1B are arranged in the HMD 350. This is attached to measure the position and orientation of the HMD 350 by the position and orientation sensor 304. If the position and orientation of the HMD 350 is obtained by another method, it is not necessary to attach the retroreflective marker to the HMD 350.

  Next, the position / orientation sensor 304 will be described.

  The position / orientation sensor 304 is, for example, an optical reflection type real-time three-dimensional motion analysis system. The optical reflection type real-time three-dimensional motion analysis system calculates the position and orientation of an object composed of a group of retroreflective markers by photographing a group of retroreflective markers in a real space from a plurality of cameras. Output as 3D position and orientation data. In the present embodiment, the retroreflective marker is attached to the user's hand, the gripper that the user grips with the hand, and the HMD 350, so the position / orientation sensor 304 measures each position / orientation. Then, the measurement result is sent to the image processing apparatus 300. Since a technique for obtaining the position and orientation of a target to which a retroreflective marker is attached using a retroreflective marker and an optical reflection type real-time three-dimensional motion analysis system is well known, here, Description is omitted.

  Next, the image processing apparatus 300 will be described. The image processing apparatus 300 includes a captured image capturing unit 302, an image composition unit 303, an image generation unit 307, a state determination unit 305, a model data attribute setting unit 306, and a storage unit 308.

  The captured image capturing unit 302 sequentially transmits the image of each frame transmitted from the imaging unit 301 to the image composition unit 303. When the captured image capturing unit 302 receives an image from the image capturing unit 301, the captured image capturing unit 302 converts the format of the image into a format that can be handled by the image combining unit 303, and then transmits the image to the image combining unit 303.

  The image composition unit 303 generates a composite image by superimposing the virtual space image generated by the image generation unit 307 on the real space image received from the captured image capture unit 302. The generated composite image is sent to the display unit 309. There are various techniques for superimposing a virtual space image on a real space image, and any technique may be used in this embodiment.

  The state determination unit 305 acquires position / posture information indicating the position / posture of the user's hand and position / posture information indicating the position / posture of the gripper, which are measured by the position / posture sensor 304, and uses the acquired position / posture information. , Determine the state of the hand. Information indicating the determination result is transmitted to the model data attribute setting unit 306 at the subsequent stage.

  Based on the information received from the state determination unit 305, the model data attribute setting unit 306 identifies the hand attribute corresponding to the hand state, and notifies the identified image generation unit 307 of the identified attribute.

  In the storage unit 308, data on each virtual object constituting the virtual space is registered. In such data, a virtual object imitating a user's hand and a virtual object imitating a gripping tool are registered. The virtual object imitating the user's hand includes a virtual object of the hand corresponding to each of various hand states. For example, a virtual object that imitates an open hand, a virtual object that imitates a hand in a goo state, a virtual object that imitates a hand that points the direction with an index finger, etc. Virtual objects corresponding to each of the seed attributes) are registered in the storage unit 308.

  In the following description, a virtual object that imitates a user's hand is referred to as a “hand model”, and a virtual object that imitates a gripping tool is referred to as a “gripping tool model”.

  The image generation unit 307 reads data of the hand model corresponding to the attribute notified from the model data attribute setting unit 306 from the storage unit 308. Then, a hand model based on the hand model data is arranged in the virtual space with the “position and orientation of the user's hand” measured by the position and orientation sensor 304 (first arrangement).

  On the other hand, the image generation unit 307 reads the gripping tool model data registered in the storage unit 308, and uses the “grip tool position and orientation” measured by the position and orientation sensor 304 for the gripping tool model based on the read data. And arrange in the virtual space (second arrangement). In this way, a virtual space is constructed. Of course, a virtual object may be arranged in the virtual space.

  Then, the image generation unit 307 generates an image of a virtual space that can be seen from the “position and orientation of the HMD 350” (the position and orientation of the viewpoint) measured by the position and orientation sensor 304. Since a technique for generating an image of a virtual space that can be seen from a viewpoint having a predetermined position and orientation is a known technique, a description thereof will be omitted.

  Then, the image generation unit 307 sends the generated virtual space image to the image composition unit 303. As described above, the image synthesis unit 303 synthesizes the virtual space image on the real space image received from the imaging unit 301.

  Next, the processing for generating a composite image in which an image in a virtual space in which a hand model corresponding to the state of the user's hand is arranged in the position and orientation of the user's hand is superimposed on an image in the real space will be described. This will be described below with reference to FIG. 4 showing a flowchart of processing.

  First, in step S <b> 401, the image generation unit 307 reads data used in subsequent processing from the storage unit 308.

  In step S <b> 402, the image generation unit 307 displays the “position / posture information of the HMD 350”, “position / posture information of the user's hand”, and “position / posture information of the gripper” measured by the position / posture sensor 304 as sensor information. Get as.

  Next, in step S <b> 403, the state determination unit 305 acquires “position / posture information of the user's hand” and “position / posture information of the gripper” measured by the position / posture sensor 304, and uses these position / posture information. To determine the state of the user's hand. Details of the processing in step S403 will be described later.

  Next, in step S404, the model data attribute setting unit 306 determines the attributes of the hand model to be placed in the virtual space with the position and orientation of the user's hand according to the hand state determined in step S403. Details of the processing in step S404 will be described later.

  In step S <b> 405, the image generation unit 307 reads from the storage unit 308 the hand model data corresponding to the attribute determined by the model data attribute setting unit 306. Then, the hand model based on the hand model data is arranged in the virtual space with the “position and orientation of the user's hand” measured by the position and orientation sensor 304.

  Further, the image generation unit 307 reads the data of the gripping tool model registered in the storage unit 308, and uses the “position / posture of the gripping tool” measured by the position / posture sensor 304 for the gripping tool model based on the read data. Place in the virtual space.

  Then, the image generation unit 307 generates an image of the virtual space constructed in this way as viewed from “the position and orientation of the HMD 350” measured by the position and orientation sensor 304.

  In step S406, the image composition unit 303 generates a composite image by superimposing the virtual space image generated by the image generation unit 307 in step S405 on the real space image received from the captured image capture unit 302. To do. The generated composite image is sent to the display unit 309. Of course, when another display device is connected to the image processing device 300, the composite image may be output to the display device.

  Note that when a device that requires only an image in the virtual space is connected to the image processing apparatus 300, such as the HMD 350 is an optical see-through HMD, the image synthesis unit 303 for the device includes the image generation unit 307. Only the image of the virtual space received from is sent out.

  Next, when it is detected that an instruction for ending this process is input to the image processing apparatus 300, or when it is detected that a condition for ending this process is satisfied, step S407 is performed. Then, the present process is terminated. On the other hand, if none is detected, the process returns to step S402 via step S407, and the processes after step S402 are performed to generate a composite image of the next frame.

  Next, the process in step S403 will be described in detail with reference to the flowchart of FIG. FIG. 5 is a flowchart showing details of the process in step S403. Note that the process according to the flowchart of FIG. 5 is performed by the state determination unit 305.

  First, in step S501, the state determination unit 305 includes position information in the “position / posture information of the user's hand” and position information in the “position / posture information of the gripper” measured by the position / posture sensor 304; Is used to obtain distance information between positions indicated by the respective position information.

  Such calculation is performed based on the following equation, for example.

Distance information = Pos (H−W) = | PosH −PosW |
PosH is “position information of the user's hand”, and PosW is “position information of the gripper”. Here, “position information” is composed of an x-coordinate value, a y-coordinate value, and a z-coordinate value in the three-dimensional virtual space. In addition, about the formula for calculating | requiring distance information, it is not limited to the type | formula which concerns, You may obtain | require by another method.

  Next, in step S502, the state determination unit 305 determines whether or not the distance indicated by the distance information obtained in step S501 is equal to or less than a predetermined threshold value (distance or less). For example, it is determined whether the distance indicated by the distance information is 30 cm or less.

  As a result of the determination, if the distance indicated by the distance information obtained in step S501 is equal to or smaller than the threshold value, the process proceeds to step S503. On the other hand, if it is larger than the threshold value, the process proceeds to step S504.

  In step S504, the model data attribute setting unit 306 is notified of information indicating that the current hand state is the default state, for example, information indicating that the hand is an open hand. Then, the process according to the flowchart of FIG. 5 ends, and the process returns to step S404 of FIG.

  On the other hand, in step S <b> 503, the state determination unit 305 uses the “position / posture information of the user's hand” and the “position / posture information of the gripper” measured by the position / posture sensor 304 to make the user's hand relative to the gripper. To find the correct position and orientation. That is, the position / orientation relationship information indicating the position / orientation relationship between the user's hand and the gripper is obtained. Such calculation is performed based on the following equation, for example.

Relative position and orientation = Mat (H−W) = MatH ⁻¹・ MatW
MatH is matrix data including “position / posture information of user's hand”, and MatW is matrix data including “position / posture information of gripper”. Here, as is well known, the position and orientation information is composed of position information and orientation information, and the position information is composed of coordinate values of x, y, and z as described above. , X-axis, y-axis, and z-axis.

  In step S505, the state determination unit 305 determines whether the position / orientation relationship information obtained in step S503 indicates a position / orientation relationship within a predetermined range. For example, it is determined whether or not the roll angle of the posture components in the position / posture relationship information obtained in step S503 is within a specified range of “within a range of 40 degrees to 50 degrees”. Data indicating the specified range is registered in the storage unit 308 in advance.

  In addition to the roll angle, the same specified ranges may be defined for each of the x, y, and z coordinate values, the pitch angle, and the yaw angle. In this case, in step S505, all elements (x, y, z coordinate values and rotation angles around each axis) constituting the position / orientation relationship information obtained in step S503 are not within the specified range for each element. In this case, it is determined as “NO”. On the other hand, when all elements (x, y, z coordinate values and rotation angles around each axis) constituting the position / orientation relationship information obtained in step S503 are within the range defined for each element. Is determined as “YES”.

  If "YES" is determined, the process proceeds to step S506. If "NO" is determined, the process proceeds to step S504.

  Here, the following function may be used in the processing in step S505.

Determination (Mat (H-W))
Here, Determination (x) is a state determination function, and has a function of determining whether each element constituting the argument x is within a range defined for each element and returning the determination result.

  In the above description, it is assumed that in step S504, it is determined whether all elements constituting the position / orientation relationship information obtained in step S503 are within the range defined for each element. explained. However, it is not only the result of the single determination process that is used, but the determination result “contained” is the final result only when it is determined “contained” in n consecutive determination processes. It is good also as a determination result. In addition, even if it is determined that it is continuously included in the m (m <n−1) determination processes, if it is determined that it is not included in the (m + 1) th determination process, “at that time” The final decision result “No” is output.

  The function used for the determination process in that case may be determined as follows.

Determination (Mat (n) (H−W))
Here, Mat (n) (H−W) is the position and orientation information obtained continuously in step S503.

  For example, a range in which ± α is added to the position and orientation relationship between the user's hand and the gripping tool when the user's hand is gripping the gripping tool is registered in the storage unit 308 as the specified range. Suppose that Furthermore, it is assumed that the variable n indicating the number of times is defined as n = 3 (the value of the variable n is also registered in the storage unit 308). In this case, when the relative position and orientation of the user's hand with respect to the gripping tool is within the range determined by the specified range, it is determined “YES” in step S505 for the first time when it is determined three times in succession. To do.

  In step S506, for example, when the data of the specified range used in step S505 is for determining whether or not the state is a goo state, the information indicating the hand state is “a goo state”. To the model data attribute setting unit 306. In addition, when the data of the specified range used in step S505 is for determining whether or not the finger is in the state of pointing with the index finger, the information indicating the hand state is in the state of pointing with the index finger. To the model data attribute setting unit 306. As described above, in step S506, the model data attribute setting unit 306 is notified of information indicating the hand state to be determined using the data in the specified range.

  Next, the process in step S404 will be described with reference to FIG. 6 showing a flowchart of the process. FIG. 6 is a flowchart showing details of the process in step S404. 6 is performed by the model data attribute setting unit 306.

  First, in step S <b> 601, the model data attribute setting unit 306 specifies the attribute of the hand model corresponding to the “information indicating the hand state” received from the state determination unit 305. Information indicating various hand states is registered in the storage unit 308 in a state where attributes are associated with each other. That is, information indicating the hand state and the attributes of the hand model imitating the hand have a one-to-one relationship and are registered in the storage unit 308 for each hand state. For example, information indicating a goo state and an attribute of a hand model imitating a hand in the goo state are registered in the storage unit 308 as a set.

  Therefore, for example, when the model data attribute setting unit 306 receives “information indicating the goo state” from the state determination unit 305, the attribute of the hand model imitating the hand of the goo state is specified.

  In step S602, the image generation unit 307 is notified of the attribute specified in step S601.

  As described above, according to the present embodiment, the position and orientation of the virtual object of the hand according to the state of the actual hand can be obtained without any complicated operation such as an operator-mediated operation or an input using a glove-type input device. It is possible to generate an image of the virtual space arranged on the screen.

[Second Embodiment]
In the present embodiment, the data of the specified range used in step S505 is registered in the storage unit 308 for each of various hand states, and the hand state determination indicated by the position / orientation relationship information obtained in step S503 is performed. Do more finely. That is, there may be a plurality of states to be determined. In the present embodiment, the same system as in the first embodiment is used.

  FIG. 7A is a diagram illustrating a user's hand observing a mixed reality space in which a virtual space is superimposed on the real space, and a gripping tool to be gripped by the hand. FIG. 7B is a diagram illustrating a virtual object to be superimposed and displayed on the user's hand and a virtual object to be superimposed and displayed on the gripping tool. 7A and 7B, the same parts as those in FIGS. 1A and 1B are denoted by the same reference numerals, and the description thereof is omitted.

  As shown in FIG. 7A, the hand 701 holds the wand 104, but has a grip type different from the hand 105 shown in FIG. 1A. In this case, the position / posture relationship between the hand 105 and the wand 104 and the position / posture relationship between the hand 701 and the wand 104 are different. In this embodiment, when the position and orientation relationship between the hand 701 and the wand 104 is the position and orientation relationship as shown in FIG. 7A, as shown in FIG. 7B, a hand model 702 that imitates the hand 701 is It arrange | positions with the position and orientation of the hand 701. FIG.

  That is, in the present embodiment, the hand state is determined more finely than in the first embodiment, and a hand model corresponding to the determination result is arranged. Note that when the hand model is selectively switched, the gripper model may be switched as appropriate.

  FIG. 8A is a diagram illustrating a user's hand observing a mixed reality space in which a virtual space is superimposed on the real space, and a gripping tool that is a target to be gripped by the hand. FIG. 8B is a diagram illustrating a virtual object to be superimposed and displayed on the user's hand and a virtual object to be superimposed and displayed on the gripping tool. 8A and 8B, the same parts as those in FIGS. 1A and 1B are denoted by the same reference numerals, and the description thereof is omitted.

  As shown in FIG. 8A, when the position / posture relationship between the hand and the gripping tool becomes the position / posture relationship between the hand 801 and the wand 104, as shown in FIG. The tool model is switched from the previous virtual object 107 to the virtual object 850.

  Next, this embodiment for generating a composite image in which a virtual space image in which a hand model corresponding to the state of the user's hand is arranged in the position and orientation of the user's hand is superimposed on an image in the real space The process which concerns on is demonstrated. The processing according to the present embodiment is the same as that of the first embodiment except for the processing at step S505. That is, the processes shown in FIGS. 4 to 6 except for step S505 are performed in the same manner as in the first embodiment.

  Therefore, the process in step S505 according to the present embodiment will be described here.

  The “specified range data” varies depending on the state of the hand. For example, a specified range that the position / posture relationship information satisfies to determine that the current hand state is an open state, and a position / posture relationship information that satisfies the current hand state to be a goo state It is different from the specified range. In the present embodiment, description will be made on the assumption that data (state information) indicating what kind of state it is for determining is associated with the data in the specified range. For example, the state information indicating “open hand” is associated with the data indicating the specified range that the position and orientation relation information satisfies in order to determine that the current hand state is the open state. A set of data and state information in such a prescribed range is registered in the storage unit 308 for each state.

  Therefore, in this embodiment, in step S505, the processing in step S505 described in the first embodiment is performed for each data in the specified range corresponding to each state information. For example, first, determination processing is performed using the position / orientation relationship information obtained in step S503 and data indicating a specified range satisfied by the position / orientation relationship information in order to determine that the current hand state is an open state. . As a result of the determination process, if the position / orientation relationship information obtained in step S503 indicates a position / orientation relationship within the specified range, it is determined that the hand state is open. On the other hand, when the position / orientation relationship information obtained in step S503 does not indicate the position / orientation relationship within the specified range, a determination process using data of the next specified range is performed. For example, the determination processing is performed using the position / orientation relationship information obtained in step S503 and data indicating a specified range satisfied by the position / orientation relationship information in order to determine that the current hand state is a goo state.

  In this way, using the data of each specified range, it is determined what hand state the position and orientation relationship obtained in step S503 indicates.

[Third Embodiment]
In the first and second embodiments, only one type of gripping tool is used. However, a plurality of types of gripping tools may be prepared and arbitrary gripping tools may be gripped. A corresponding gripping tool model is prepared for each gripping tool. Therefore, the user can observe different gripping tool models each time the gripping tool is changed.

  In this case, a plurality of retroreflective markers are attached to each gripping tool as in the first embodiment, but the arrangement relationship of the retroreflective markers is different for each gripping tool. This is because, when the position / orientation sensor 304 detects a retroreflective marker group attached to the gripper, it identifies the gripper based on the position / posture relationship between the retroreflective markers. When the gripping tool being gripped is identified, the gripping tool model associated with the gripping tool is placed at the position and orientation of the gripping tool.

  In the present embodiment, the same system as in the first embodiment is used.

  FIG. 9A is a diagram illustrating a user's hand observing a mixed reality space in which a virtual space is superimposed on the real space, and a gripping tool to be gripped by the hand. FIG. 9B is a diagram illustrating a virtual object to be superimposed on the user's hand and a virtual object to be superimposed on the gripping tool. 9A and 9B, the same portions as those in FIGS. 1A and 1B are denoted by the same reference numerals, and the description thereof is omitted.

  As shown in FIG. 9A, a jig 103 is attached to the hand 902 as in the first embodiment, and retroreflective markers 102 a to 102 c are attached to the jig 103. Such a hand 902 holds the gripping tool 901. In addition, retroreflective markers 901a to 901e are attached to the gripper 901, as in the first embodiment. The gripping tool 901 is different from the wand 104 shown in FIG. 1A, and the arrangement relationship of the retroreflective markers arranged is also different. This is as described above.

  Here, it is assumed that the position and orientation relationship between the hand 902 and the gripper 901 and the position and orientation relationship between the hand 101 and the wand 104 are substantially the same.

  In FIG. 9B, 903 is a hand model arranged with the position and orientation of the hand 902, and 950 is a gripping tool model arranged with the position and orientation of the gripping tool 901. Here, although the position and orientation relationship between the hand 902 and the gripping tool 901 is substantially the same as the position and orientation relationship between the hand 101 and the wand 104, the position and orientation of the gripping tool 901 is not the virtual object 107, A gripper model 950 is disposed.

  In the present embodiment, a process for generating a composite image in which an image in a virtual space in which a hand model corresponding to the state of a user's hand is arranged in the position and orientation of the user's hand is superimposed on an image in the real space is shown in FIGS. The processing according to the flowchart of FIG. 6 is modified as follows.

  First, in the flowchart of FIG. 4, in step S402, the position / posture information of the gripping tool is acquired, and the gripping tool is identified based on the positional relationship of each retroreflective marker attached to the gripping tool.

  In steps S403 and S404, processing is performed based on “specified range data” unique to the identified gripping tool and “correspondence between hand state and attributes” unique to the identified gripping tool. That is, the storage unit 308 registers data of a specified range for each type of gripping tool. In step S505 of FIG. 5, determination is made using data of the specified range corresponding to the gripping tool identified in step S402. Process. Therefore, even if the position and orientation relationship between the hand and the gripping tool is the same, the determination result of the hand state may differ depending on the type of the gripping tool.

  In the flowchart of FIG. 6, in step S601, an attribute corresponding to the determined hand state is determined, but the attribute corresponding to the determined hand state may also be different for each type of gripping tool. That is, in the storage unit 308, attributes associated with various hand state information are registered in association with each other, but such association differs for each type of gripping tool.

  FIG. 10 is a diagram showing types of hand states that can be identified, attributes according to hand states, and functions of model data for each type of wand as a gripping tool.

  For example, when the type of the gripping tool identified in step S402 is “wand 1”, in step S505, it is determined whether the hand is in any one of states 1 to 3. become. For example, “the hand state is state 1” means that the position and orientation relationship obtained in step S503 is within the specified range indicated by the specified range data corresponding to state 1. Further, “the hand state is state 2” means that the position and orientation relationship obtained in step S503 is within the specified range indicated by the specified range data corresponding to state 2.

  When the type of the gripping tool identified in step S402 is “wand 3”, in step S505, it is determined whether the hand is in the state 1 or 2 and is neither. . As described above, the types of hand states to be identified and the number thereof may be different depending on the types of gripping tools identified.

  If the type of the gripping tool identified in step S402 is “wand 1” and the hand state is determined to be “state 1” in step S505, the hand model to be placed in the hand position / posture. “Hand model B” is selected. Further, “tool model A” is selected as the gripping tool model to be arranged at the position and orientation of the gripping tool. Further, “pick a screw” is selected as the function of the model data.

  The “model data function” indicates the type of processing to be performed by the image processing apparatus 300 when a predetermined operation is performed with the gripper. For example, when it is determined that the type of the gripping tool identified in step S402 is “wand 1” and the state of the hand is “state 2” in step S505, a predetermined operation is performed with the gripping tool. Then, the image processing apparatus 300 performs a process of “checking interference”.

  Note that both the hand model and the gripper model need not always be arranged, and depending on the situation, only one of them may be arranged. For example, when the hand is in a goo state, the arrangement of the gripper model may be canceled.

  In each of the above-described embodiments, an optical reflection type real-time three-dimensional motion analysis system is used for position and orientation measurement of the HMD 350, hand, gripping tool, and the like. However, the position and orientation measurement method is not limited to this, and each of the above embodiments may acquire the position and orientation using any method.

  In each of the above embodiments, the position and orientation of the HMD 350, the hand, and the gripping tool are measured by one sensor system. However, the position and orientation may be measured by combining various methods. For example, a plurality of sensor systems may be used such that a gyro sensor is used in addition to the optical reflection type real-time three-dimensional motion analysis system.

  In each of the above embodiments, the position and orientation relationship between the HMD 350 and the hand is used to determine the hand state. However, the hand state may be obtained using other elements. For example, the operation state of the hand and the positional posture relationship between the hand and the gripping tool may be obtained by a pressure sensor attached to the gripping tool.

  In addition to the position and orientation relationship between the HMD 350 and the hand, for example, a position and orientation relationship between the hand and the mock and a position and orientation relationship between the mock and the mock may be used.

  The first to third embodiments may be used in appropriate combination.

[Fourth Embodiment]
Each unit configuring the image processing apparatus 300 illustrated in FIG. 3 has been described as being configured by hardware in the first to third embodiments. However, some of them may be realized by software. In this case, a general PC (personal computer) may be used as the image processing apparatus 300 and the computer may execute the software.

  FIG. 11 is a diagram illustrating a hardware configuration of a computer applicable to the image processing apparatus 300. Note that the hardware configuration of the computer is not limited to this, and various modifications can be considered.

  The CPU 1101 controls the entire computer using programs and data stored in the RAM 1102 and the ROM 1103 and executes the above-described processes performed by the image processing apparatus 300.

  The RAM 1102 has an area for temporarily storing programs and data loaded from the external storage device 1106 and data received from the HMD 350 and the position / orientation sensor 304 via the I / F (interface) 1107. Further, the RAM 1102 has a work area used when the CPU 1101 executes each process. That is, the RAM 1102 can provide various areas as appropriate.

  The ROM 1103 stores computer setting data, a boot program, and the like.

  The operation unit 1104 is configured by a keyboard, a mouse, and the like, and can input various instructions to the CPU 1101 when operated by a computer operator.

  The display unit 1105 is configured by a CRT, a liquid crystal screen, or the like, and can display a processing result by the CPU 1101 using an image, characters, or the like.

  The external storage device 1106 is a large-capacity information storage device represented by a hard disk drive device. The external storage device 1106 stores an OS (operating system) and programs and data for causing the CPU 1101 to execute the above-described processes performed by the image processing apparatus 300.

  The program includes a program for causing the CPU 1101 to execute the above-described processes described as being performed by the image composition unit 303, the image generation unit 307, the model data attribute setting unit 306, and the state determination unit 305. In addition, the data includes the data group described as being held by the storage unit 308. In other words, the external storage device 1106 also functions as the storage unit 308. Note that the storage unit 308 may be a function of the RAM 1102.

  An I / F 1107 is used to connect the HMD 350 and the position / orientation sensor 304 to the computer. The image processing apparatus 300 receives an image of the real space captured by the imaging unit 301 or transmits a composite image to the display unit 309 via the I / F 1107. That is, the I / F 1107 also functions as the captured image capturing unit 302. Further, the image processing apparatus 300 receives the position / orientation information output from the position / orientation sensor 304 via the I / F 1107. Information received via the I / F 1107 is sent to the RAM 1102 and the external storage device 1106.

  A bus 1108 connects the above-described units.

[Other Embodiments]
Needless to say, the object of the present invention can be achieved as follows. That is, a recording medium (or storage medium) in which a program code of software that realizes the functions of the above-described embodiments is recorded is supplied to the system or apparatus. Needless to say, such a storage medium is a computer-readable storage medium. Then, the computer (or CPU or MPU) of the system or apparatus reads and executes the program code stored in the recording medium. In this case, the program code itself read from the recording medium realizes the functions of the above-described embodiment, and the recording medium on which the program code is recorded constitutes the present invention.

  Further, by executing the program code read by the computer, an operating system (OS) or the like running on the computer performs part or all of the actual processing based on the instruction of the program code. Needless to say, the process includes the case where the functions of the above-described embodiments are realized.

  Furthermore, it is assumed that the program code read from the recording medium is written in a memory provided in a function expansion card inserted into the computer or a function expansion unit connected to the computer. After that, based on the instruction of the program code, the CPU included in the function expansion card or function expansion unit performs part or all of the actual processing, and the function of the above-described embodiment is realized by the processing. Needless to say.

  When the present invention is applied to the recording medium, program code corresponding to the flowchart described above is stored in the recording medium.

It is a figure which shows the user's hand which observes the mixed reality space which superimposed the virtual space on the real space, and the holding tool used as the object which this hand holds. It is a figure which shows the virtual object displayed superimposed on a user's hand, and the virtual object displayed superimposed on a holding tool. It is a figure which shows the virtual object displayed superimposed on a user's hand, and the virtual object displayed superimposed on a holding tool. It is a block diagram which shows the function structure of the system which concerns on the 1st Embodiment of this invention. It is a flowchart of the process for producing | generating the synthesized image which superimposed the image of the virtual space which arrange | positioned the hand model according to the state of a user's hand with the position and orientation of a user's hand with the image of real space. It is a flowchart which shows the detail of the process in step S403. It is a flowchart which shows the detail of the process in step S404. It is a figure which shows the user's hand which observes the mixed reality space which superimposed the virtual space on the real space, and the holding tool used as the object which this hand holds. It is a figure which shows the virtual object displayed superimposed on a user's hand, and the virtual object displayed superimposed on a holding tool. It is a figure which shows the user's hand which observes the mixed reality space which superimposed the virtual space on the real space, and the holding tool used as the object which this hand holds. It is a figure which shows the virtual object displayed superimposed on a user's hand, and the virtual object displayed superimposed on a holding tool. It is a figure which shows the user's hand which observes the mixed reality space which superimposed the virtual space on the real space, and the holding tool used as the object which this hand holds. It is a figure which shows the virtual object displayed superimposed on a user's hand, and the virtual object displayed superimposed on a holding tool. It is a figure which shows the kind of hand state which can be discriminate | determined for every kind of wand as a holding tool, the attribute according to the state of a hand, and the function of model data. 2 is a diagram illustrating a hardware configuration of a computer applicable to the image processing apparatus 300. FIG.

Claims (8)

Means for obtaining the position and orientation of the user's viewpoint;
Means for acquiring the position and orientation of the user's hand;
Means for acquiring a position and orientation of a gripping tool as a target to be gripped by the hand;
A first placement means for placing a virtual object imitating the hand in a virtual space at the position and orientation of the hand;
A second placement means for placing a virtual object imitating the gripping tool in the virtual space at the position and orientation of the gripping tool;
Means for generating an image of the virtual space in which a virtual object is arranged by the first arrangement means and the second arrangement means based on the position and orientation of the viewpoint;
Output means for outputting an image of the virtual space,
The first arrangement means includes
Calculating means for obtaining a position and orientation relationship between the position and orientation of the hand and the position and orientation of the gripping tool;
A virtual object corresponding to the position and orientation relationship is selected as a target to be placed in the virtual space from among virtual objects representing each of a plurality of types of hand states that are held in advance as virtual objects that imitate the hand. An image processing apparatus comprising: selection means for performing   The first arrangement means obtains a distance between the position of the hand and the position of the gripping tool, and when the obtained distance is equal to or less than a predetermined distance, the calculation means and the selection means The image processing apparatus according to claim 1, wherein the process is executed. Furthermore, it comprises means for acquiring an image of the real space that can be seen from the viewpoint,
The image processing apparatus according to claim 1, wherein the output unit outputs the virtual space image in a state of being combined with the real space image.   The image processing apparatus according to claim 1, wherein the second arrangement unit arranges a virtual object imitating the gripping tool corresponding to the positional posture relationship in the virtual space. The virtual object corresponding to the position and orientation relationship differs for each type of gripping tool,
The image processing apparatus according to claim 1, wherein the selection unit selects a virtual object corresponding to the position and orientation relationship based on a type of a gripping tool as a target to be gripped by the hand. Obtaining the position and orientation of the user's viewpoint;
Obtaining a position and orientation of the user's hand;
Obtaining a position and orientation of a gripping tool as a target to be gripped by the hand;
A first placement step of placing a virtual object imitating the hand in a virtual space at the position and orientation of the hand;
A second arrangement step of arranging a virtual object imitating the gripping tool in the virtual space at the position and orientation of the gripping tool;
Generating an image of the virtual space in which a virtual object is arranged in the first arrangement step and the second arrangement step based on the position and orientation of the viewpoint;
An output step of outputting an image of the virtual space,
The first arrangement step includes
A calculation step for obtaining a position and orientation relationship between the position and orientation of the hand and the position and orientation of the gripping tool;
A virtual object corresponding to the position and orientation relationship is selected as a target to be placed in the virtual space from among virtual objects representing each of a plurality of types of hand states that are held in advance as virtual objects that imitate the hand. An image processing method comprising: a selection step.   A program for causing a computer to execute each step constituting the image processing method according to claim 6.   A computer-readable storage medium storing the program according to claim 7.

Images