CN117369649B - Virtual reality interaction system and method based on proprioception - Google Patents
Virtual reality interaction system and method based on proprioception Download PDFInfo
- Publication number
- CN117369649B CN117369649B CN202311647877.5A CN202311647877A CN117369649B CN 117369649 B CN117369649 B CN 117369649B CN 202311647877 A CN202311647877 A CN 202311647877A CN 117369649 B CN117369649 B CN 117369649B
- Authority
- CN
- China
- Prior art keywords
- user
- virtual
- humanoid
- interaction
- menu
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/011—Arrangements for interaction with the human body, e.g. for user immersion in virtual reality
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/048—Interaction techniques based on graphical user interfaces [GUI]
- G06F3/0481—Interaction techniques based on graphical user interfaces [GUI] based on specific properties of the displayed interaction object or a metaphor-based environment, e.g. interaction with desktop elements like windows or icons, or assisted by a cursor's changing behaviour or appearance
- G06F3/0482—Interaction with lists of selectable items, e.g. menus
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/44—Arrangements for executing specific programs
- G06F9/451—Execution arrangements for user interfaces
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/764—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using classification, e.g. of video objects
- G06V10/765—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using classification, e.g. of video objects using rules for classification or partitioning the feature space
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
- G06V40/20—Movements or behaviour, e.g. gesture recognition
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F2203/00—Indexing scheme relating to G06F3/00 - G06F3/048
- G06F2203/01—Indexing scheme relating to G06F3/01
- G06F2203/011—Emotion or mood input determined on the basis of sensed human body parameters such as pulse, heart rate or beat, temperature of skin, facial expressions, iris, voice pitch, brain activity patterns
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- General Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Human Computer Interaction (AREA)
- Software Systems (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Health & Medical Sciences (AREA)
- General Health & Medical Sciences (AREA)
- Multimedia (AREA)
- Psychiatry (AREA)
- Social Psychology (AREA)
- Artificial Intelligence (AREA)
- Computing Systems (AREA)
- Databases & Information Systems (AREA)
- Evolutionary Computation (AREA)
- Medical Informatics (AREA)
- User Interface Of Digital Computer (AREA)
Abstract
The invention discloses a virtual reality interaction system and method based on proprioception, and relates to the technical field of virtual reality interaction. The human interface construction module is configured to construct a corresponding virtual human menu in the virtual environment according to individual characteristics of a user, and assign human interaction operation to the virtual human menu; the human body action recognition module is configured to recognize the slapping body part of the user and recognize the human body gesture, and provides a data source for real-time interaction; and the real-time interaction module is configured to trigger corresponding humanoid interaction operation according to the recognition result of the human action recognition module and obtain corresponding interaction feedback. The invention constructs the humanoid menu associated with the human body structure, simplifies the complex process of the feedback from the visual content to the brain of the user, and enables the user to accurately perceive the body position corresponding to the action and interaction by utilizing the proprioception, thereby more naturally interacting with the virtual environment.
Description
Technical Field
The invention relates to the technical field of virtual reality interaction, in particular to a virtual reality interaction system and method based on proprioception.
Background
In recent years, virtual reality technology has been rapidly developed and widely used in many fields. Virtual environments are becoming more and more immersive and realistic. However, how to interact naturally and easily in a virtual reality environment remains a challenging problem. At present, although the virtual space interaction uses a plurality of modes such as a keyboard, a mouse, voice, even eyeball motion, electroencephalogram and the like, the air click interaction is still the most mainstream interaction mode of menu selection.
The air click interaction mainly adopts a controller or a gesture to realize menu selection operation. The method based on the controller is mainly realized by light rays emitted by the controller in the virtual scene. When a ray passes through a menu item, the user can select it by pressing a trigger on the controller. Gesture-based interaction methods enable a user to interact naturally with a virtual environment without requiring additional controllers by recognizing and interpreting gestures of the user. Thus, the over-the-air click interactions can be categorized as "intuitively clicking on what you see" interaction style. The user is provided with a relatively immersive VR experience to some extent because the user can always observe the current virtual environment throughout the selection process. However, the controller-based approach requires additional learning of the method of use of the different controllers or the setting of the same controller in different VR applications, increasing the cognitive burden, making the interaction process more complex and cumbersome. In contrast, gesture-based interactions are more natural than controller-based interactions. However, manipulating the finger in the air, especially for long or complex tasks, can lead to user fatigue, and lack of haptic feedback limits the realism of the VR experience.
In order to enhance the feel of real haptic feedback, some work has proposed body-based interaction methods. It attaches the UI interaction menu directly to the user's body, allowing the user to interact with the UI by physically touching the menu. Holonens also designs hand menus, allowing users to quickly invoke virtual arm menu interfaces, and are very easy to display and hide. Not only is a realistic haptic feedback added, but the advantage of "intuitively clicking what you see" is also retained. However, when the user interacts with the body menu, the immersive experience is broken because the user needs to look at his own body to locate the interactive menu. In addition, long lifting of the arm or head can also increase fatigue and discomfort.
Both of the above methods require the user to see a virtual or real hand in the virtual space, or use a virtual handle as a metaphor for the hand, to improve the accuracy of the menu selection interaction: when the user selects the interactive menu item, the motion of the hand becomes a continuous process, and the brain sends out instructions according to visual feedback to continuously adjust the motion path of the finger so as to accurately position the interactive menu item. But this process not only increases the brain reaction time during the interaction process, but also affects the immersion of the user VR experience. Therefore, how to realize more intuitive air click interaction and simplify the complex process from visual content to brain feedback of the user becomes a technical problem to be solved in the prior art.
Disclosure of Invention
Aiming at the defects existing in the prior art, the invention aims to provide a virtual reality interaction system and method based on proprioception, which construct a humanoid menu associated with human body structure, simplify the complex process of feedback from visual content to brain of a user, and enable the user to accurately perceive the body position corresponding to actions and interactions by utilizing proprioception so as to perform more natural interactions with a virtual environment.
In order to achieve the above object, the present invention is realized by the following technical scheme:
the first aspect of the present invention provides a virtual reality interaction system based on proprioception, comprising:
the human interface construction module is configured to construct a corresponding virtual human menu in the virtual environment according to individual characteristics of a user, and assign human interaction operation to the virtual human menu;
the human body action recognition module is configured to recognize the slapping body part of the user and recognize the human body gesture, and provides a data source for real-time interaction;
and the real-time interaction module is configured to trigger corresponding humanoid interaction operation according to the recognition result of the human action recognition module and obtain corresponding interaction feedback.
Further, the humanoid interface construction module includes a virtual humanoid menu module configured to:
obtaining image information of a user according to user capturing equipment, and obtaining position information of head, foot and hand joint points of a human body, so as to calculate height information of the user;
and generating a virtual humanoid interaction interface with the visual height similar to the user as a virtual humanoid menu by utilizing the association between the body coordinate system of the user and the virtual scene coordinate system.
Further, the human interface construction module includes an assignment module configured to:
dividing human-shaped interaction areas on the virtual human-shaped menu according to biological characteristics of human bodies;
binding specific interactive operation to a humanoid interactive area in the virtual humanoid menu;
and storing the association between the virtual humanoid menu and the real human body part in a virtual-real body part mapping table according to the bound interactive operation.
Further, the human-shaped interaction region is divided into a head region, an upper arm region, a lower arm region, a chest region, an upper abdomen region and a lower abdomen region according to biological characteristics of a human body.
Further, the human motion recognition module comprises a model training module for training a motion recognition model, and recognition of the user motion is realized by using the trained motion recognition model.
Still further, the model training module is configured to:
acquiring a real-time slapping position of a user;
and processing the real-time slapping position by using the trained action recognition model, and judging the humanoid interaction area slapped by the current user.
Still further, the model training module is further configured to:
acquiring a marked known data set, and dividing the marked known data set into a training set and a testing set;
training the support vector machine by using a training set;
and evaluating and adjusting the model obtained by training by using the test set to obtain the action recognition model.
Further, according to the recognition result of the human motion recognition module, the mapping table of the virtual and the real body parts is retrieved, so that the corresponding human shape interaction operation is triggered.
Further, the interactive feedback includes visual feedback, auditory feedback, and tactile feedback.
The second aspect of the present invention provides a virtual reality interaction method based on proprioception, comprising the steps of:
constructing a corresponding virtual humanoid menu in the virtual environment according to individual characteristics of a user, and assigning humanoid interactive operation to the virtual humanoid menu;
the method comprises the steps of identifying the body part of a user slapping and identifying the human body gesture, and providing a data source for real-time interaction;
and triggering corresponding humanoid interactive operation according to the recognition result of the human action recognition module, and obtaining corresponding interactive feedback.
The one or more of the above technical solutions have the following beneficial effects:
the invention discloses a virtual reality interaction system and a virtual reality interaction method based on proprioception, wherein an interaction interface is constructed into a human-shaped structure, and the interaction interface is laid out according to biological characteristics of human bodies, so that positions of interaction elements in the interaction interface are in one-to-one correspondence with positions of human body parts in the real world, and a process of naturally interacting with own bodies in the physical world is simulated through proprioceptive interaction. The user can lightly press or touch familiar body landmarks without consciously positioning virtual interactive elements, so that the interactive feeling is more visual. The proprioceptive-based interactions of the present invention do not require the user to visually browse or search menu items and can initiate interactions faster, thus such interactions reduce the cognitive load and reaction time required to select options.
The invention also simplifies the navigation and selection process in the virtual menu, potentially generates more efficient and productive interaction, and improves interaction efficiency. The user can seamlessly engage in interactions without continuing to focus on the vision on the menu or control, thus maintaining a higher level of immersion in the virtual environment.
The interaction technology provided by the invention is not only suitable for a virtual reality system based on a head-mounted display device (HMD), but also suitable for an immersive mixed reality system (such as CAVE) based on projection, and has higher universality.
Additional aspects of the invention will be set forth in part in the description which follows and, in part, will be obvious from the description, or may be learned by practice of the invention.
Drawings
The accompanying drawings, which are included to provide a further understanding of the invention and are incorporated in and constitute a part of this specification, illustrate embodiments of the invention and together with the description serve to explain the invention.
FIG. 1 is a diagram illustrating an interactive interface and an interactive manner according to a first embodiment of the present invention.
FIG. 2 is a functional architecture diagram of a proprioception-based virtual reality interaction system according to a first embodiment of the invention;
FIG. 3 is a virtual human interface according to a first embodiment of the present invention;
FIG. 4 is a virtual humanoid menu viewed by a VR glasses when a user performs a flick task in accordance with one embodiment of the present invention;
FIG. 5 is a flow chart of an interaction process of a proprioception-based virtual reality interaction system according to a first embodiment of the invention;
fig. 6 is a schematic diagram of a scene image and a user gesture after a user enters a questionnaire system according to the first embodiment of the present invention;
FIG. 7 is a schematic diagram showing a user touching his abdomen to click the button "No" on the mermaid abdomen in the first embodiment of the present invention;
FIG. 8 is a schematic diagram of a user touching his forearm to click the button "Next" on the mermaid forearm in accordance with a first embodiment of the invention;
fig. 9 is a schematic diagram of a scene image and a user gesture after a user enters a cognitive training system according to the first embodiment of the present invention;
fig. 10 is a schematic diagram of a user touching his or her arm to click a button on the mermaid arm in the first embodiment of the present invention.
Detailed Description
It should be noted that the following detailed description is exemplary and is intended to provide further explanation of the invention. Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs.
It should be noted that, in the embodiments of the present invention, related data such as the gesture and the position of the user is referred to, and when the above embodiments of the present invention are applied to specific products or technologies, user permission or consent needs to be obtained.
It is noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of exemplary embodiments according to the present invention. As used herein, the singular is also intended to include the plural unless the context clearly indicates otherwise, and furthermore, it is to be understood that the terms "comprises" and/or "comprising" when used in this specification are taken to specify the presence of stated features, steps, operations, devices, components, and/or combinations thereof;
embodiment one:
the first embodiment of the invention provides a virtual reality interaction system based on proprioception, and provides a virtual reality interaction mode-what you see is what you beat-based on proprioception-for the problem that in the prior art, the virtual interaction process is not intuitive enough and is not immersed enough. The interactive interface is constructed into a human-shaped structure, and the interactive interface is laid out according to the biological characteristics of the human body, so that the positions of the interactive elements in the interactive interface correspond to the positions of the human body parts in the real world one by one. For example, the arm is composed of an upper arm and a forearm, and the arm region of the humanoid menu may be divided into two interactable portions corresponding to the positions of the upper arm and the forearm of the real arm, respectively. When the user sees the upper arm part of the humanoid menu presented in front of eyes, the user can naturally slap the upper arm position of the real arm without switching the viewpoint, and the interactive control of the virtual menu is realized.
As shown in fig. 1, the present embodiment is illustrated in an interactive manner in a helmet-based virtual reality system. The virtual interface is laid out according to the biological characteristics of the human body, and the right half part in fig. 1 is the virtual interface presented in front of the eyes of the user, so that the positions of the menus of the virtual interface are in one-to-one correspondence with the human body parts in the real world. The user roams the virtual scene through the virtual reality glasses, and interactive control of the virtual menu is realized by beating the physical body part corresponding to the virtual interface by means of proprioception. In the system, the slapping position of the user is required to be identified, and the invention takes an RGB-D camera as an example to track and identify the slapping position of the user. The present embodiment is described by taking a storm magic mirror as an example, but is not limited to such VR glasses.
The present embodiment provides a continuous immersive experience based on proprioception, which is the perception of a person's body position, posture and motion. In the embodiment, the interactive elements in the interactive interface in the air in the virtual environment are rearranged into the human body shape, so that a user intuitively perceives that the elements on the human body interface are corresponding body parts. That is, the interactive elements in the human interface form a one-to-one mapping with each part of the user's body, and the user can drive the selection of the corresponding interactive elements in the human interface by tapping the corresponding body parts, enhancing the avatar experience and the sense of presence of the user in the virtual world. The system comprises human interface construction, human action recognition and real-time interaction tasks, and as shown in fig. 2, the human interface construction process comprises the steps of calculating the height of a user, generating human interaction areas, dividing the human interaction areas, assigning human interaction operations and constructing a virtual-real body part mapping table. The human body action recognition process comprises the steps of counting the distribution of the slapping points of different interaction areas of the human-shaped interface, and judging the body part slapped by the current user hand. The real-time interaction task process comprises the steps of retrieving a virtual-real body part mapping table, triggering corresponding interaction operation, and carrying out visual feedback, auditory feedback or tactile feedback according to the interaction operation. In particular, visual feedback is information provided by the visual perception of the user. In real-time interactive tasks, visual elements such as images, charts, animations, etc., may be presented to a user through a display screen, projector, virtual reality device, or other visual device.
Auditory feedback is information provided by the auditory perception of a user, and may be achieved by means of sound, sound effects, speech synthesis, and the like.
And haptic feedback, wherein in a virtual reality environment, interaction with a virtual object is simulated through force feedback equipment, so that a user feels the texture and resistance of the object. In this embodiment, the haptic feedback refers to a haptic sensation obtained by the user using an interactive manner of beating the body, so as to enhance the interactive experience of the user.
The above functions are realized by the following modules:
and the humanoid interface construction module is configured to construct a corresponding virtual humanoid menu in the virtual environment according to individual characteristics of the user, and assign humanoid interaction operation to the virtual humanoid menu.
The human body action recognition module is configured to recognize a slapping body part of a user and recognize a human body gesture and provides a data source for real-time interaction.
And the real-time interaction module is configured to trigger corresponding humanoid interaction operation according to the recognition result of the human action recognition module and obtain corresponding interaction feedback.
The human interface construction module comprises a virtual human menu module which is used for constructing a corresponding virtual human menu in the virtual environment according to individual characteristics of a user:
s101, obtaining the position information of the head, the foot and the hand joint point of the human body according to the image information of the user obtained by the user capturing equipment, so as to calculate the height information of the user. In this embodiment, the user capture device is exemplified by an RGB-D camera.
S102, according to the height information of the user, a virtual humanoid interactive interface with the visual height similar to the user is generated as a virtual humanoid menu by utilizing the association between the body coordinate system of the user and the virtual scene coordinate system. And then assigning human interactive operation to the virtual human menu.
The human interface construction module further comprises a virtual human menu module for assigning human interaction operations to the virtual human menu:
s103, dividing human-shaped interaction areas on the virtual human-shaped menu according to biological characteristics of human bodies, so that the positions of interaction elements in the interaction interface correspond to the positions of human body parts in the real world. In this embodiment, considering the challenge of maintaining the body balance during the interaction of the user, only the division of the upper body is considered, the upper body of the human body is divided into 8 regions corresponding to different body parts, and as shown in fig. 3, the human-shaped interaction region is divided into a head region 1, upper arm regions 2 and 3, lower arm regions 4 and 5, a chest region 6, an upper abdomen region 7 and a lower abdomen region 8 according to the biological characteristics of the human body. Each region represents a portion of the human body, and the regions represent the head, upper arm, lower arm, chest, upper abdomen, and lower abdomen, respectively, of the human body.
S104, binding specific interaction operation to the humanoid interaction area in the virtual humanoid menu.
In a specific embodiment, the interactive operation corresponding to the "head" in the virtual interface is designated, which means "return to main menu". The operation of the other interaction areas is defined by the user according to the interaction purpose. In order to enrich the interactive operation of the interface and the complex logic structure of the menu, the system stores multilevel menu information in a tree structure. Each row in the file representing the node has information of each node recorded in detail. The record of each node comprises the following details of the interaction menu number, the interaction menu name and the parent node number of the interaction menu in the humanoid menu.
And the interactive system reads the content in the file configuration human interface when running. Thanks to this organized file format, the hierarchical structure of the interface can be more easily expressed, which also simplifies the understanding and management of the connections and characteristics between each node. Therefore, the multi-level menu options are accurately presented in the user interface, and the menu information can be quickly arranged and managed. This eases the navigation interface and improves the user experience.
S105, the association between the virtual humanoid menu and the real human body part is stored in the virtual-real body part mapping table according to the bound interactive operation, as shown in table 1, each row in the table contains the number, the main body part name, the position in the virtual space, the operation (the operation referring to this menu). And designating the interactive operation corresponding to the head in the virtual interface, namely returning to the main menu, and defining the operation of other interactive areas by a user.
TABLE 1 virtual-to-real body part mapping table example
The human body action recognition part is mainly used for recognizing the slapping body part of the user and the human body posture. Specifically, the user's slapping body part is identified, providing a source of data for the real-time interactive portion. The human motion recognition module comprises a model training module which is used for training a motion recognition model and realizing recognition of the user motion by using the trained motion recognition model. The recognition of the user action by using the trained action recognition model specifically comprises the following steps:
acquiring a real-time slapping position of a user: and counting the distribution of the slapping points of different interaction areas of the human-shaped interface, namely counting the positions of the slapping points of the appointed part of the body.
Judging the body part where the user's hand is slapped in real time: and processing the real-time slapping position by using the trained action recognition model, and judging the humanoid interaction area slapped by the current user.
In this embodiment, the motion recognition model is described by taking a Support Vector Machine (SVM) as an example. The specific steps of training the action recognition model are as follows:
s201, acquiring a marked known data set, and dividing the marked known data set into a training set and a testing set.
In a specific embodiment, 6 users were recruited, 2 of which were men and 4 of which were women. Wherein, the user characteristics are: age 22-28 years (average age 24 years, standard deviation 2.098), body height 155-183 cm (average height 166 cm, standard deviation 10.658), weight 44-85 kg (average weight 61 kg, standard deviation 13.711). Moreover, the dominant hand of all users is the right hand. Participants were required to wear virtual reality glasses in experiments to view the virtual humanoid interface presented in front of their eyes. Their task is to click on the corresponding area of the body with the right hand, as instructed, to complete the menu selection in the humanoid interface. Each menu requires 50 choices, recording the right hand position of each click, i.e. the body part of the user, and the position of the menu in the virtual interface. For example, as shown in fig. 4, when a certain user performs the selection tasks of three menus T1, T6, and T7, data of the corresponding body touch position is collected.
S202, training the support vector machine by using the training set.
And S203, evaluating and adjusting the model obtained by training by using the test set to obtain an action recognition model.
In a specific embodiment, the multi-classification model SVM is trained based on the data collected as described above. During the training of the model, the menu item is selected as the dependent variable to be predicted with the right hand position and height data as the independent variables. The data are classified using SVM and 8 SVM classifiers are trained according to 8 parts of the human body. To this end, each corresponding partial data set is divided into an 80% training set and a 20% test set. The SVM under each condition is trained using a training set. To avoid overfitting the data, 3 replicates of 10 fold cross-validation were used and a 20% test set was predicted to evaluate the quality of the SVM.
Finally, classifying the body parts slapped by the hands of the user in real time by using the trained SVM model, calculating the probability of each body part, and then selecting the body part with the highest probability as the interaction area of the current hand.
And in the real-time interaction module, the virtual and real body part mapping table is retrieved according to the identification result of the human body action identification module, so that corresponding human-shaped interaction operation is triggered, and real-time feedback in visual sense, auditory sense and touch sense is provided. In the process of interacting with the virtual interface, the user's proprioception is utilized to perform rapid and intuitive slapping operation, and the interactive control of the virtual interface is completed.
Fig. 5 is a flowchart of a system interaction process according to the present embodiment. The method comprises the steps of detecting user gesture and position data and triggering human menu interaction. The human-shaped menu interaction process comprises the steps of selecting a menu to be selected by tapping a corresponding part of a body, deducing a menu area clicked by a user by using a trained SVM model, triggering interaction operation with a virtual human-shaped menu by retrieving a virtual-real body part mapping table, and realizing interaction with the virtual human-shaped menu according to the process.
In one particular embodiment, once the humanoid interface is established, the user may use it for an interactive experience. The process begins with loading structural data of a menu into the system to build a humanoid interface. Subsequently, the tracking sensor is activated. The user then wears cardboard 3D glasses so that they can be immersed in a virtual 3D environment. In this environment, the user can interact directly with the human interface by tapping his body under the guidance of the motion tracking sensor. Such interactions occur subconsciously and naturally, enhancing the overall user experience. The method comprises the following specific steps:
s301: gesture and position data of a user when roaming in a virtual environment are detected in real time, including position data of hand joints, head joints and foot joints. Wherein, the height value of the user is calculated by utilizing the joint information of the head and the foot;
s302: it is determined whether to trigger a humanoid menu interaction. If the gesture of the user is to lift the right hand, the constructed humanoid menu is presented in front of the user, suspended in the air, and the step S303 is skipped, otherwise, the step S301 is skipped;
s303: selecting a desired menu by tapping on a corresponding part of the body;
s304: and deducing a menu area clicked by the user by using the trained SVM model. Specifically, the obtained user right hand position and height information is used as input of an SVM multi-classification model, and is output as an interaction area label selected by a user (the label is empty and clicked, and indicates that the label is not clicked);
s305: triggering the interaction operation with the virtual humanoid menu by retrieving the virtual-real body part mapping table;
s506: it is determined whether the user ends the interaction, if the user' S gesture is to raise the left hand, the interaction ends, otherwise, step S301 is skipped.
In steps S302 and S306, the method for determining the user gesture is as follows:
in S302, the user is identified to lift the right hand, requiring the user' S right hand elbow to be higher than the right shoulder D 1 *H/H m The wrist of the right hand is higher than the elbow D of the right hand 2 *H/H m Rice, wherein D 1 And D 2 As distance parameter, H is the height of the user (in meters), H m Is of standard height, Y wristR Representing the height of the user's right wrist, Y elbowR Representing the height of the user's right elbow, Y shoulderR The height of the right shoulder of the user is represented, namely the following two conditions are required to be met:
，
。
in S506, the user is identified to lift the left hand, requiring the user' S left hand elbow to be higher than the left shoulder D 1 *H/H m Rice, the left hand wrist is higher than the left hand elbow D 2 *H/H m Rice, wherein D 1 And D 2 As distance parameter, H is the height of the user (in meters), H m Is of standard height, Y wristL Representing the height of the left wrist of the user, Y elbowL Representing the height of the left elbow of the user, Y shoulderL The height of the left shoulder of the user is represented, namely the following two conditions are required to be met:
，
。
considering that the arm distance of the user is approximately proportional to the height of the user, the distance required by this embodiment is proportional to the height of the user. In consideration of the fact that a user with standard height (the data are collected and counted in the human motion recognition part) lifts the right hand, the wrist is 0.15m higher than the elbow and 0.2m higher than the shoulder, in order to reduce the difficulty of user interaction, the requirement is properly reduced on the premise that other motion recognition conflicts are not caused, and D is taken as the scheme in the embodiment 1 = 0.17m，D 2 = 0.13m，H m =1.7m。Y wristR 、Y elbowR 、Y shoulderR 、Y wristL 、Y elbowL 、Y shoulderL Are obtained by the user motion capture device.
The user operates the virtual menu in the head-mounted virtual reality device by tapping on his own body part. The virtual menu gives good visual feedback of the "highlighting", "clicking" animation and tactile feedback of the user's own body. At t=0s, the virtual menu in the head-mounted virtual reality device is completed to be constructed. At t=3 s, the user lifts the right hand and the server begins to recognize the action of tapping the interface button. At t=7s, the user right hand moves to the "chest" button, and the virtual menu button "corresponding to" chest "is highlighted". At t=8s, the user has stopped for more than 0.5s on the chest, completed the click on "chest", and the menu goes to the next layer in the head-mounted virtual reality device. At t=10s, the user right hand moves down, passing the "upper abdomen" button, and the virtual menu button "highlighted" corresponding to "upper abdomen". At t=10.3 s, the user moves right hand to the "lower abdomen" button, and the virtual menu button "highlighted" corresponding to "lower abdomen". At t=11 s, the user has stopped at the chest for more than 0.5s, completing a click of "lower abdomen". The virtual menu button corresponding to the lower abdomen is clicked, and the menu in the head-mounted virtual reality device enters the next layer.
This example also demonstrates a comparison of the proposed method with the way in which fingers are manipulated in the air and the way in which arms are based. The user selects a menu by right finger direction to operate a virtual menu in the head-mounted virtual reality device. The virtual menu can give visual feedback of the "highlighting", "clicking" animation, but cannot provide corresponding tactile feedback. At t=0s, the virtual menu in the head-mounted virtual reality device is completed to be constructed. At t=3s, the user lifts the right hand and the server begins to identify the right hand finger pointing (the red aperture on the virtual menu shows the location of the user's finger pointing). At t=8s, the user moves the right hand so that the finger points to the "a" button, which is "highlighted". At t=9s, the user has stopped for more than 0.5s at the "a" button, and has completed clicking the "a" button, and the menu in the head-mounted virtual reality device enters the next layer. At t=12 s, the user right hand moves down to the "a2" button, the "a2" button "being highlighted. At t=13 s, the user has stopped for more than 0.5s at the "a2" button, has completed clicking the "a2" button, and the menu in the head-mounted virtual reality device enters the next layer. Manipulating the finger in the air, especially for long or complex tasks, can lead to user fatigue (due to the need to stay in the selected menu to prevent misconnection) and lack of haptic feedback limits the realism of the VR experience.
The user operates a virtual menu in the head-mounted virtual reality device by tapping his or her arm to select the menu. The virtual menu gives good visual feedback of the "highlighting", "clicking" animation and tactile feedback of the user's own body. At t=0s, the virtual menu in the head-mounted virtual reality device is completed to be constructed. At t=3 s, the user lifts the right hand and the server begins to track the left and right hand positions of the user so that the virtual left arm follows the left arm of the user. At t=7s, the user's right hand moves to the "left wrist" position and the virtual menu button "highlight" of the "left wrist" position. At t=8s, the user has stopped for more than 0.5s at the "left wrist" position, and has completed clicking the "left wrist" button, and the menu in the head-mounted virtual reality device goes to the next layer. At t=10s, the user line of sight moves up, observing the change in virtual scene. At t=12 s, the user's line of sight moves down to the "left arm" menu. At t=15s, the user right hand moves to the "left hand forearm" position and the virtual menu button "highlights" of the "left hand forearm" position. At t=16s, the user has stopped for more than 0.5s at the "left-hand forearm" position, and completed clicking of the "left-hand forearm" button, with the menu in the head-mounted virtual reality device going to the next level. At t=18s, the user line of sight moves up, observing a change in the virtual scene. When a user interacts with a body menu, the immersive experience is broken because the user needs to look at his body to locate the interactive menu. In addition, long lifting of the arm or head can also increase fatigue and discomfort.
As shown in fig. 6, 7 and 8, is a questionnaire system based on Augmented Reality (AR) technology. The following is a specific interaction process:
AR glasses: the user wears AR glasses that can superimpose virtual information over the user's real field of view.
Observation options: the user views the individual options in the questionnaire through AR glasses, which are presented in virtual form in their field of view.
Selecting and filling: the user uses the method proposed by the present disclosure to select and fill out questionnaires in an AR environment. This may include selecting answers, filling in text, or performing other questionnaire-related operations.
Virtual humanoid menu: to assist the user in conducting a questionnaire, a virtual humanoid menu is attached to the mermaid as a visual cue for the user. The user can realize quick, accurate and convenient questionnaire operation by looking at a specific area on the virtual humanoid menu.
RGB-D camera Kinect: in the system, the RGB-D camera Kinect is placed directly in front of the user, about 2.2 meters from the user. This camera is used to capture body posture information of the user so that the user can interact with the virtual questionnaire through gestures.
The AR-based questionnaire system provides a more visual and interactive way for questionnaire investigation, so that a user can interact with the questionnaire content more easily, and the user experience and the accuracy of the questionnaire are improved.
Fig. 6 shows a scene picture and a user gesture after the user enters the questionnaire system, fig. 7 shows the user touching his abdomen to click a button "No" on the mermaid abdomen, the option of the questionnaire is modified to "No", and fig. 8 shows the user touching his forearm to click a button "Next" on the mermaid forearm to enter the Next question.
As shown in fig. 9 and 10, is a CAVE-environment-based cognitive training system. The following is a specific interaction process:
immersive scene: the user enters the CAVE environment and can see an immersive submarine world scene, so that the user is full of visual details and immersion.
And (3) target selection: in a subsea scenario, a fish may appear and be selected as a target by the user. Subsequently, the fish starts to move, and more identical fish will appear and move together.
Target identification: the task of the user is to identify and select their targets during the course of the fish movements. However, the task is not easy because the system may introduce other creatures and sounds as disturbing factors.
A plurality of checkpoints: the system provides a total of 6 checkpoints. Of these, 3 are simple modes, and the user needs to select a target from among 3 fish. The other 3 are difficulty modes, requiring the selection of targets from the 6 fish. In the difficulty mode, the fish moves more chaotic, and the interference factors increase.
Race time: the user needs to complete the task in a total of game time, limited to 7 minutes.
Virtual humanoid menu: to ensure the user's immersion, the mermaid statue is set as a virtual humanoid menu, located in front of the user. Different parts represent different buttons, and a user can stay for 0.5 seconds at the corresponding part of the virtual humanoid menu by using the user's own hand so as to realize clicking of the buttons.
The system aims to provide an interesting cognitive training mode and challenges the attention and cognitive ability of a user through an immersive experience. The Kinect camera is placed in front of the user for capturing body posture information of the user, enabling the user to interact with the virtual environment through gestures.
Fig. 9 shows a scene and a user gesture after the user enters the cognitive training system, and fig. 10 shows the user touching his or her arm to click a button on the mermaid arm to select the fish on the mermaid arm.
Through the above example verification, it can be seen that in the virtual reality interaction system based on proprioception, the user can lightly press or touch familiar body landmarks without consciously positioning virtual interaction elements, so that the interaction feel is more visual. The proprioceptive-based interactions of the present invention do not require the user to visually browse or search menu items and can initiate interactions faster, thus such interactions reduce the cognitive load and reaction time required to select options. The invention also simplifies the navigation and selection process in the virtual menu, potentially generates more efficient and productive interaction, and improves interaction efficiency. The user can seamlessly engage in interactions without continuing to focus on the vision on the menu or control, thus maintaining a higher level of immersion in the virtual environment.
The interaction technology provided by the invention is not only suitable for a virtual reality system based on a head-mounted display device (HMD), but also suitable for an immersive mixed reality system (such as CAVE) based on projection, and has higher universality.
Embodiment two:
the second embodiment of the invention provides a virtual reality interaction method based on proprioception, which comprises the following steps:
constructing a corresponding virtual humanoid menu in the virtual environment according to individual characteristics of a user, and assigning humanoid interactive operation to the virtual humanoid menu;
the method comprises the steps of identifying the body part of a user slapping and identifying the human body gesture, and providing a data source for real-time interaction;
and triggering corresponding humanoid interactive operation according to the recognition result of the human action recognition module, and obtaining corresponding interactive feedback.
The steps involved in the second embodiment correspond to those of the first embodiment, and reference is made to the relevant description of the first embodiment for the implementation manner.
It will be appreciated by those skilled in the art that the modules or steps of the invention described above may be implemented by general-purpose computer means, alternatively they may be implemented by program code executable by computing means, whereby they may be stored in storage means for execution by computing means, or they may be made into individual integrated circuit modules separately, or a plurality of modules or steps in them may be made into a single integrated circuit module. The present invention is not limited to any specific combination of hardware and software.
While the foregoing description of the embodiments of the present invention has been presented in conjunction with the drawings, it should be understood that it is not intended to limit the scope of the invention, but rather, it is intended to cover all modifications or variations within the scope of the invention as defined by the claims of the present invention.
Claims (8)
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311647877.5A CN117369649B (en) | 2023-12-05 | 2023-12-05 | Virtual reality interaction system and method based on proprioception |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311647877.5A CN117369649B (en) | 2023-12-05 | 2023-12-05 | Virtual reality interaction system and method based on proprioception |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN117369649A CN117369649A (en) | 2024-01-09 |
| CN117369649B true CN117369649B (en) | 2024-03-26 |
Family
ID=89391333
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202311647877.5A Active CN117369649B (en) | 2023-12-05 | 2023-12-05 | Virtual reality interaction system and method based on proprioception |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN117369649B (en) |
Citations (13)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102640086A (en) * | 2009-12-04 | 2012-08-15 | 微软公司 | Sensing mechanical energy to adapt the body for data entry |
| CN103529930A (en) * | 2013-01-04 | 2014-01-22 | 深圳市中兴移动通信有限公司 | Menu indexing method and device based on somatosensory recognition and terminal with device based on somatosensory recognition |
| CN105404449A (en) * | 2015-07-21 | 2016-03-16 | 浙江传媒学院 | Hierarchically extendable multi-pie somatosensory menu and syntax-directed identification method therefor |
| CN106951072A (en) * | 2017-03-06 | 2017-07-14 | 南京航空航天大学 | On-screen menu body feeling interaction method based on Kinect |
| CN107291240A (en) * | 2017-06-29 | 2017-10-24 | 华南理工大学 | A kind of virtual reality multilevel menu exchange method based on brain-computer interface |
| CN109496331A (en) * | 2016-05-20 | 2019-03-19 | 奇跃公司 | Contextual awareness of user interface menus |
| KR20200072621A (en) * | 2018-12-12 | 2020-06-23 | 주식회사 바이널아이 | System for an interactive contents support using social community function and virtual reality supporting function |
| WO2022170650A1 (en) * | 2021-02-09 | 2022-08-18 | 南京微纳科技研究院有限公司 | Contactless human-machine interaction system and method |
| WO2022179279A1 (en) * | 2021-02-26 | 2022-09-01 | 华为技术有限公司 | Interaction method, electronic device, and interaction system |
| CN115497149A (en) * | 2022-10-21 | 2022-12-20 | 浙江吉利控股集团有限公司 | A music interaction method for car cockpit |
| CN116301328A (en) * | 2023-01-21 | 2023-06-23 | 浙江大学 | Virtual reality equipment interaction identification system and control equipment based on arm knocking |
| CN117032453A (en) * | 2023-08-03 | 2023-11-10 | 深想教育科技(深圳)有限公司 | Virtual reality interaction system for realizing mutual recognition function |
| CN117130518A (en) * | 2023-08-25 | 2023-11-28 | 北京正负无限科技有限公司 | Control display method, head display device, electronic device and readable storage medium |
Family Cites Families (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US9910497B2 (en) * | 2006-02-08 | 2018-03-06 | Oblong Industries, Inc. | Gestural control of autonomous and semi-autonomous systems |
| IT1399456B1 (en) * | 2009-09-11 | 2013-04-19 | Sr Labs S R L | METHOD AND APPARATUS FOR THE USE OF GENERIC SOFTWARE APPLICATIONS THROUGH EYE CONTROL AND INTERACTION METHODS IS APPROPRIATE. |
| US20120229509A1 (en) * | 2011-03-07 | 2012-09-13 | Liu Guangsong | System and method for user interaction |
| US11106273B2 (en) * | 2015-10-30 | 2021-08-31 | Ostendo Technologies, Inc. | System and methods for on-body gestural interfaces and projection displays |
| CN109344755B (en) * | 2018-09-21 | 2024-02-13 | 广州市百果园信息技术有限公司 | Video action recognition method, device, equipment and storage medium |
| WO2021145878A1 (en) * | 2020-01-16 | 2021-07-22 | Von Clausewitz Systems Llc | Mobile application platform projected on a secondary display with intelligent gesture interactions |
-
2023
- 2023-12-05 CN CN202311647877.5A patent/CN117369649B/en active Active
Patent Citations (13)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102640086A (en) * | 2009-12-04 | 2012-08-15 | 微软公司 | Sensing mechanical energy to adapt the body for data entry |
| CN103529930A (en) * | 2013-01-04 | 2014-01-22 | 深圳市中兴移动通信有限公司 | Menu indexing method and device based on somatosensory recognition and terminal with device based on somatosensory recognition |
| CN105404449A (en) * | 2015-07-21 | 2016-03-16 | 浙江传媒学院 | Hierarchically extendable multi-pie somatosensory menu and syntax-directed identification method therefor |
| CN109496331A (en) * | 2016-05-20 | 2019-03-19 | 奇跃公司 | Contextual awareness of user interface menus |
| CN106951072A (en) * | 2017-03-06 | 2017-07-14 | 南京航空航天大学 | On-screen menu body feeling interaction method based on Kinect |
| CN107291240A (en) * | 2017-06-29 | 2017-10-24 | 华南理工大学 | A kind of virtual reality multilevel menu exchange method based on brain-computer interface |
| KR20200072621A (en) * | 2018-12-12 | 2020-06-23 | 주식회사 바이널아이 | System for an interactive contents support using social community function and virtual reality supporting function |
| WO2022170650A1 (en) * | 2021-02-09 | 2022-08-18 | 南京微纳科技研究院有限公司 | Contactless human-machine interaction system and method |
| WO2022179279A1 (en) * | 2021-02-26 | 2022-09-01 | 华为技术有限公司 | Interaction method, electronic device, and interaction system |
| CN115497149A (en) * | 2022-10-21 | 2022-12-20 | 浙江吉利控股集团有限公司 | A music interaction method for car cockpit |
| CN116301328A (en) * | 2023-01-21 | 2023-06-23 | 浙江大学 | Virtual reality equipment interaction identification system and control equipment based on arm knocking |
| CN117032453A (en) * | 2023-08-03 | 2023-11-10 | 深想教育科技(深圳)有限公司 | Virtual reality interaction system for realizing mutual recognition function |
| CN117130518A (en) * | 2023-08-25 | 2023-11-28 | 北京正负无限科技有限公司 | Control display method, head display device, electronic device and readable storage medium |
Non-Patent Citations (7)
| Title |
|---|
| An interactive VR system based on full-body tracking and gesture recognition;Zeng Xia等;《Optical Design And Testing Vii》;20161031(第第10021期期);的1-7页 * |
| Hierarchical Menu Selection with a Body-Centered Remote Interface;Bossavit, Benoit等;《Interacting With Computers》;20130801;第26卷(第5期);第389-402页 * |
| MGP: a monitoring-based VR interactive mode to support guided practice;Wang Yu等;《Visual Computer》;20230701;第39卷(第8期);第3387-3401页页 * |
| 一种基于Kinect的虚拟现实姿态交互工具;鲁明;王真水;田元;李琳;;系统仿真学报;20130908(第09期);第2124-2130页 * |
| 基于实时视频感知的虚拟体育交互系统;陆启迪等;《计算机系统应用》;20230331;第32卷(第3期);第125-132页 * |
| 基于虚拟化身的增强现实交互技术研究及应用;张振宁;《中国优秀硕士学位论文全文数据库 (信息科技辑)》;20200615;第I138-1148页 * |
| 虚拟现实中实时交互方法研究;盖伟;《中国博士学位论文全文数据库 (信息科技辑)》;20180315(第3期);第I138-36页 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN117369649A (en) | 2024-01-09 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US11741917B2 (en) | Eclipse cursor for mixed reality displays | |
| JP7411133B2 (en) | Keyboards for virtual reality display systems, augmented reality display systems, and mixed reality display systems | |
| Spittle et al. | A review of interaction techniques for immersive environments | |
| CN119960596A (en) | Device, method and graphical user interface for content application | |
| CN112424736B (en) | Machine Interaction | |
| CN116909407B (en) | Touch display screen panoramic interaction method and control system based on virtual reality | |
| TW202105129A (en) | Artificial reality systems with personal assistant element for gating user interface elements | |
| KR20240134054A (en) | Selecting virtual objects in a three-dimensional space | |
| Bowman | Principles for the design of performance-oriented interaction techniques | |
| US11998798B2 (en) | Virtual guided fitness routines for augmented reality experiences | |
| Gao | Key technologies of human–computer interaction for immersive somatosensory interactive games using VR technology | |
| Hernoux et al. | A seamless solution for 3D real-time interaction: design and evaluation | |
| US20240077986A1 (en) | Opportunistic adaptive tangible user interfaces for use in extended reality environments | |
| CN117369649B (en) | Virtual reality interaction system and method based on proprioception | |
| CN109144598A (en) | Electronics mask man-machine interaction method and system based on gesture | |
| CN117742479A (en) | Man-machine interaction method, device, equipment and medium | |
| Wu et al. | An empirical practice of design and evaluation of freehand interaction gestures in virtual reality | |
| US20250238991A1 (en) | System and method for authoring context-aware augmented reality instruction through generative artificial intelligence | |
| Piumsomboon | Natural hand interaction for augmented reality. | |
| Pietroszek | 3D Pointing with Everyday Devices: Speed, Occlusion, Fatigue | |
| Kim | Tangible user interface for CAVE based on augmented reality technique | |
| Babu | Focus AR: A study of Focus-based Mobile AR App in Industrial Training | |
| Nordin | SMART VIRTUAL ENVIRONMENT FOR DESIGNERS WITH MOTOR LIMITATIONS | |
| Uras | Designing for Spatial Computing: Exploring Natural and Traditional Interfaces in a VR Moodboarding Tool | |
| JP2024171644A (en) | Action visualization device, action visualization method, and program |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |