CN115373513A - Desktop interaction system, realization method and storage medium - Google Patents

Abstract

The invention discloses a desktop interactive system, a realization method and a storage medium, wherein the interactive system comprises: the interactive pen comprises at least three keys, wherein each key has a pressing state and a releasing state, and the keys are used for receiving pressing and releasing operations of a user and sending the pressing and releasing operations to the interactive equipment; and the interaction equipment is used for performing grabbing, rotating, zooming and returning operations on the model according to the pressing and releasing operations. Therefore, various static models and skeleton models in the content developed by the user operation UE4 can be realized through the interactive pen and the interactive equipment, and the user experience can be improved.

Description

A desktop interactive system, implementation method, and storage medium

technical field

The invention relates to the field of data interaction, in particular to a desktop interaction system, an implementation method, and a storage medium.

Background technique

Virtual reality technology (virtual reality, VR), also known as spiritual technology, is a new practical technology developed in the 20th century. Virtual reality technology includes computer, electronic information, and simulation technology. Its basic implementation method is to simulate a virtual environment by computer to give people a sense of environmental immersion. With the continuous development of social productivity and science and technology, all walks of life have increasingly strong demand for VR technology, VR technology has also made great progress, and gradually become a new field of science and technology.

The interactive pen is the core component to realize the manipulation of the virtual scene, in which the spatial coordinates, posture and pen pointing of the interactive pen are the key data to realize the precise control of the virtual scene. How to interact with the interactive device through the interactive pen is an urgent problem to be solved.

Contents of the invention

In view of the above deficiencies in the prior art, the object of the present invention is to provide a desktop interactive system, an implementation method, and a storage medium. The desktop interactive system is a set of products that can be applied in many occasions such as education and teaching. Through interactive pens and interactive devices, users can operate various static models and skeleton models in the content developed by UE4.

The first aspect of the present invention provides a desktop interactive system, the interactive system includes: an interactive pen, the interactive pen includes at least three buttons, wherein each of the buttons has two states of pressing and releasing, the The button is used to receive the user's pressing and releasing operations, and send the pressing and releasing operations to the interactive device; the interactive device is used to grasp the model according to the pressing and releasing operations , rotate, scale, and return operations.

Optionally, in combination with the first aspect, in a possible implementation manner, the interactive pen includes a first button, and the first button is used to control the grasping model; the interactive device is used to follow the Perform ray detection in the direction pointed by the interactive pen, and store the hit result returned by the ray detection as a first variable; when the interactive device obtains from the interactive pen that the first button is pressed, the The interaction device judges whether the first variable is valid; if the first variable is valid, the interaction device grabs the model corresponding to the ray detection hit result.

Optionally, in combination with the first aspect, in a possible implementation manner, the interaction device is further configured to, when it is judged that the first variable is invalid, capture all the models in the interaction frame.

Optionally, in combination with the first aspect, in a possible implementation manner, the interactive pen is further configured to obtain its own rotation angle when the second button is pressed, and send the rotation angle to the The interaction device; the interaction device is further configured to rotate the captured model according to the rotation angle.

Optionally, in combination with the first aspect, in a possible implementation manner, the interactive pen is further configured to obtain its own coordinate position and the change amount of the coordinate position when the second button is pressed, wherein , the change amount of the coordinate position takes the first direction as the positive direction, and the direction opposite to the first direction is the reverse direction, and sends the change amount of the coordinate position to the interaction device; the interaction device is further configured to map the variation of the coordinate position to the virtual coordinates of the captured model, and scale the captured model according to the variation of the coordinate position.

Optionally, in combination with the first aspect, in a possible implementation manner, the interactive pen further includes a third button, and when the third button is clicked, the interactive device performs a return menu operation; when the When the third button is long pressed, the interactive device performs data calibration.

The second aspect of the present invention provides a desktop interaction implementation method, which is applied to the desktop interaction system described in the first aspect and any possible implementation manner of the first aspect. The method includes: receiving the pressing and releasing operation of the button by the user, and sending the pressing and releasing operation to the interactive device; performing grabbing, rotating, zooming and returning operations on the model according to the pressing and releasing operation.

With reference to the second aspect, in a possible implementation manner, the interactive pen includes a first button, and the first button is used to control the grasping model; perform ray detection along the direction pointed by the interactive pen, and Store the hit result returned by the ray detection as a first variable; when it is acquired that the first key is pressed, judge whether the first variable is valid; when the first variable is valid, set the ray The model corresponding to the detection hit result is grabbed.

With reference to the second aspect, in a possible implementation manner, the method further includes: when it is judged that the first variable is invalid, capturing all models in the interaction frame.

With reference to the second aspect, in a possible implementation manner, the method further includes: when the second key is pressed, the interactive pen obtains its own rotation angle, and sends the rotation angle to the interactive device; the interaction device rotates the captured model according to the rotation angle.

With reference to the second aspect, in a possible implementation manner, the method further includes: when the second button is pressed, the interactive pen obtains its own coordinate position and the change amount of the coordinate position, wherein the The change amount of the coordinate position takes the first direction as the positive direction, and the direction opposite to the first direction is the reverse direction, and sends the change amount of the coordinate position to the interactive device; the interactive device sends The change amount of the coordinate position is mapped to the virtual coordinates of the captured model, and the captured model is scaled according to the change amount of the coordinate position.

The third aspect of the present invention also provides a non-volatile computer-readable storage medium, which is characterized in that the non-volatile computer-readable storage medium stores computer-executable instructions, and the computer-executable instructions are controlled by one or When multiple processors execute, the one or multiple processors can be made to execute the above desktop interaction implementation method.

The invention discloses a desktop interactive system, its realization method, and a storage medium. The interactive system includes: an interactive pen, and the interactive pen includes at least three buttons, wherein each of the buttons has two states of pressing and releasing , the button is used to receive the user's press and release operations, and send the press and release operations to the interactive device; the interaction device is used to control the model according to the press and release operations Grab, rotate, scale, and return operations. In this way, users can operate various static models and skeleton models in the content developed by UE4 through interactive pens and interactive devices, which can improve user experience.

Description of drawings

The present invention will be further described below in conjunction with accompanying drawing and embodiment, in the accompanying drawing:

Fig. 1 is a schematic structural diagram of an interactive pen provided by the present invention;

Fig. 2 is a schematic diagram of adding an IMUPenComponent component provided by the present invention;

Fig. 3 is a schematic diagram of events provided by an IMUPenComponent component provided by the present invention;

Fig. 4 is a schematic diagram of a desktop interactive system provided by the present invention;

FIG. 5 is a flowchart of a method for implementing desktop interaction provided by the present invention;

Fig. 6 is a schematic structural diagram of an interactive device provided by the present invention.

Detailed ways

The embodiment of the present application provides a ground filtering method and related equipment, aiming at improving the reliability of ground filtering through multiple filtering.

The technical solutions in the embodiments of the present application will be described below with reference to the drawings in the embodiments of the present application. In this application, "at least one" means one or more, and "multiple" means two or more. "And/or" describes the association relationship of associated objects, indicating that there may be three types of relationships, for example, A and/or B, which can mean: A exists alone, A and B exist simultaneously, and B exists alone, where A, B can be singular or plural. The character "/" generally indicates that the contextual objects are an "or" relationship. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one item (piece) of a, b or c can represent: a, b, c, a and b, a and c, b and c or a and b and c, wherein a, b and c can be It can be single or multiple. It should be noted that "at least one item (item)" can also be interpreted as "one item (item) or multiple items (item)".

It should be noted that, in this application, words such as "exemplary" or "for example" are used as examples, illustrations or illustrations. Any embodiment or design described herein as "exemplary" or "for example" is not to be construed as preferred or advantageous over other embodiments or designs. Rather, the use of words such as "exemplary" or "such as" is intended to present related concepts in a concrete manner.

The first, second, etc. descriptions that appear in the embodiments of this application are only for illustration and to distinguish the description objects. Any limitations of the examples.

This application is mainly to explain the 3D desktop interactive system formed by the interactive pen and interactive equipment. The 3D desktop interactive system is a set of products that can be applied to various occasions such as education and teaching. By using the matching interactive pen and other equipment, users can operate Various static models and skeleton models in the content developed by UE4. In order to facilitate the use of users and develop new scenes on this basis, we designed and developed a basic interactive logic framework based on the UE4 rendering engine and interactive pen.

Please refer to Figure 1, Figure 1 provides a schematic diagram of an interactive pen, in Figure 1, the interactive pen 1 includes at least three buttons, the three buttons can provide button information for the UE4 engine, each button has a press and Release both states. Among them, the basic interaction mode set by the present invention is:

The first button 11: used to control the capture model, select a certain model to capture the model, select other blank areas to capture all the models.

The second button 12: used to rotate and scale the model, grabbing the blank area is regarded as all of the zoom and rotation.

The third button 13: Click to return to the menu, long press to provide data calibration to other software.

Next, the key classes involved in the desktop interactive system provided by the present invention are described:

BP_DeskController: The class corresponding to the interactive pen function.

BP_InteractableActor: Basic interactive object class, which realizes basic interactive functions

BP_InteractionHelper: The core class of the interaction framework, used to assist in interactive operations such as grasping and rotation.

IInteractive: interface, all interactive classes (interactive objects, function buttons) must implement this interface. CheckCanPickUp is one of the interface functions. The base class of the interactive object BP_InteractableActor returns true, and the base class of the button returns false.

The core idea of this interaction framework is to use an Acotr (BP_InteractionHelper) as an auxiliary tool to attach the model item that the user wants to operate to the Actor, and then call the interaction method written on the Actor to operate the Actor to realize the interaction in the scene. Manipulation of interactive objects.

Next, the implementation of the interaction logic framework is described:

S1: Create a new class named "BP_DeskController" as a class that interfaces with the interactive pen.

S2: Add an IMUPenComponent component to the "BP_DeskController", which can obtain the key state of the interactive pen, such as the state that the first key 11 is pressed but not released. Please refer to Figure 2 and Figure 3, Figure 2 provides a schematic diagram of adding an IMUPenComponent component. Fig. 3 provides a schematic diagram of events provided by the IMUPenComponent component.

Please refer to FIG. 4 . FIG. 4 provides a schematic diagram of a desktop interactive system. The schematic diagram of the desktop interactive system may include an interactive pen 1 and an interactive device 2 . The interactive device 2 may include devices capable of displaying various static models and skeleton models, such as a tablet computer, a mobile phone, or a 3D sand table.

As mentioned above, the interactive pen 1 may include at least three buttons, wherein each of the buttons has two states of pressing and releasing, and the buttons are used to receive the pressing and releasing operations of the user, and Press and release operations are sent to the interacting device 2 . The interaction device 2 is configured to perform grabbing, rotating, zooming and returning operations on the model according to the pressing and releasing operations. The operations of grabbing the model, rotating the model, scaling the model and returning are described below:

1. For the realization of the grabbing function:

The first button 11 included in the interactive pen 1 is used to control the grasping model. The interactive device 2 is used to perform ray detection along the direction pointed by the interactive pen 1, and store the hit result returned by the ray detection as a first variable; when the interactive device 2 obtains from the interactive pen 1 that the first button 11 is pressed When pressing down, the interactive device 2 judges whether the first variable is valid; if the first variable is valid, the interactive device 2 grabs the model corresponding to the ray detection hit result. If the first variable is invalid, grab all the models in the interaction frame.

Specifically, the interactive device 2 uses "BP_DeskController" to perform ray detection along the direction pointed by the interactive pen, and always returns the ray hit result, and stores the result as the variable "TraceHit". When the first button 11 is pressed, it is first judged whether the HitActor in "TraceHit" is valid.

If it is valid, then judge whether the HitActor implements the "IInteractive" interface. This interface defines basic grabbing and zooming interface events for interaction, and is implemented in the base class "BP_InteractableActor" of interactive objects. All models and other objects in the scene that can be interacted with by the user are interactive objects, and they only need to inherit the "BP_InteractableActor" class to interact with the interactive pen 1. Grab the model the ray is pointing at.

If it is invalid, or does not inherit the "IInteractive" interface, it is determined that the interactive object has not been hit. In this interaction framework, this is regarded as grabbing all interactive objects in the scene. First call the UE4 engine API: GetAllActorsWithInterface to get all the Actors that inherit the "IInteractive" interface in the scene, traverse these Actors and call the function CheckCanPickUp() in the interface, if the return is true, the Actor is grabable, call UE4 engine API: AttachActorToActor attaches (adsorbs, attaches) all graspable Actors to BP_InteractionHelper, and operates BP_InteractionHelper to realize the function of grasping all interactive objects.

2. For the realization of the rotation function:

When the second button 12 of the interactive pen 1 is pressed, obtain its own rotation angle, and send the rotation angle to the interaction device 2; the interaction device 2 is also used to The grabbed model is rotated. It should be noted that the rotation function needs to be based on the grabbing function, and the grabbed model can be rotated only after the model is grabbed.

Specifically, whether you are operating on a single model or multiple models, you need to attach the interactive object to BP_InteractionHelper before operating BP_InteractionHelper. Rotation is to call the UE4 engine API: SetActorRotation and assign the rotation of the interactive pen itself to BP_InteractionHelper. That is to say, the degree of rotation of the book rotated by the interactive pen 1 is the same as that of the grasped model.

3. For the realization of the zoom function:

When the second key 12 is pressed, the interactive pen 1 can obtain its own coordinate position and obtain the change amount of the coordinate position, wherein, the change amount of the coordinate position takes the first direction as the positive direction, so as to be consistent with the The direction opposite to the first direction is the opposite direction, and the change amount of the coordinate position is sent to the interaction device; the interaction device 2 is also used to map the change amount of the coordinate position to the captured In the virtual coordinates of the model, the captured model is scaled according to the change amount of the coordinate position.

Specifically, non-linear scaling for interaction models of different sizes can be implemented. Get the Actor first and call its internal API: GetBounds. This API returns the actor's bounding sphere radius. Use UE4's curve tool to make a non-linear scaling curve, the type is CurveFloat. Get the Actor's initial radius value MeshBoundsRadius in the BeginPlay event of the base class BP_InteractableActor of the interactive model. Then call the engine API: SetActorScale3D to scale when scaling is required, and the scaling parameters are obtained from the scaling curve according to the MeshBoundsRadius to obtain the corresponding scaling ratio. If it is the overall zoom, traverse all interaction models, and use the largest radius as MeshBoundsRadius.

The actual position and rotation of the interactive pen 1 can be mapped to the virtual world of the UE4 engine. For example, moving the interactive pen 1 forward is regarded as zooming out, and dragging backward is regarded as zooming in.

4. Return to operation.

The third key 13 is single-clicked to return to the menu, and long-pressed is used to provide other software for data calibration.

The desktop interaction system provided by the present application realizes that the UE4 content based on the 3D desktop system can realize multiple interaction modes for the models in the UE4 content scene through the interactive pen.

In view of the above desktop interaction system, the present application further provides a desktop interaction method, which is applied to the above desktop interaction system. See Figure 5, the method includes:

S301. Receive a user's press and release operations on a button, and send the press and release operations to an interactive device.

S302. Perform grabbing, rotating, zooming and returning operations on the model according to the pressing and releasing operations.

The following is an overall description of step S301 and step S302:

The interactive pen includes a first button, and the first button is used to control the grasping model; perform ray detection along the direction pointed by the interactive pen, and store the hit result returned by the ray detection as a first variable ; When it is acquired that the first button is pressed, judge whether the first variable is valid; when the first variable is valid, capture the model corresponding to the ray detection hit result.

When it is judged that the first variable is invalid, grab all the models in the interaction frame.

When the second button is pressed, the interactive pen obtains its own rotation angle, and sends the rotation angle to the interaction device; rotate.

When the second button is pressed, the interactive pen acquires its own coordinate position and the change amount of the coordinate position, wherein, the change amount of the coordinate position takes the first direction as the positive direction, so as to be consistent with the first direction The direction opposite to the direction is the opposite direction, and the change of the coordinate position is sent to the interaction device; the interaction device maps the change of the coordinate position to the virtual coordinate of the captured model , scaling the captured model according to the change amount of the coordinate position.

FIG. 6 is a schematic structural diagram of an interactive device provided by an embodiment of the present application. The interactive device 400 may have relatively large differences due to different configurations or performances, and may include one or more processors (cemtrml processimgumits, CPU) 410 ( For example, one or more processors) and memory 420, one or more storage media 430 (such as one or more mass storage devices) for storing application programs 433 or data 432. Wherein, the memory 420 and the storage medium 430 may be temporary storage or persistent storage. The program stored in the storage medium 430 may include one or more modules (not shown in the figure), and each module may include a series of instruction operations on the interactive device 400 . Furthermore, the processor 410 may be configured to communicate with the storage medium 430 , and execute a series of instruction operations in the storage medium 430 on the interaction device 400 .

The interactive device 400 may also include one or more power sources 440, one or more wired or wireless network interfaces 430, one or more input and output interfaces 460, and/or, one or more operating systems 431, such as Wimdows Serve, Nmc OS X, Umix, Limux, FreeBSD, and more. Those skilled in the art can understand that the structure of the interactive device shown in FIG. 6 does not constitute a limitation on the data processing device, and may include more or less components than shown in the figure, or combine some components, or arrange different components.

The present application also provides a computer-readable storage medium. The computer-readable storage medium may be a non-volatile computer-readable storage medium. The computer-readable storage medium may also be a volatile computer-readable storage medium. Instructions are stored in the computer-readable storage medium, and when the instructions are run on the computer, the computer is made to execute the steps of the method for implementing desktop interaction.

Those skilled in the art can clearly understand that for the convenience and brevity of the description, the specific working process of the above-described system, device and unit can refer to the corresponding process in the foregoing method embodiment, which will not be repeated here.

If the integrated unit is realized in the form of a software function unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or part of the contribution to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium , including several instructions to make a computer device (which may be a personal computer, a server, or a network device, etc.) execute all or part of the steps of the methods described in the various embodiments of the present application. The aforementioned storage media include: U disk, mobile hard disk, read-only memory (remd-omly nenory, RON), random access memory (rmmdon mccess nenory, RMN), magnetic disk or optical disc, etc., which can store program codes. .

In the embodiments provided in the present application, it should be understood that the disclosed methods can be implemented in other ways without exceeding the spirit and scope of the present application. The present embodiment is only an exemplary example and should not be taken as a limitation, and the specific content given should not limit the purpose of the present application. For example, some features may be ignored, or not implemented.

The technical means disclosed in the solutions of this application are not limited to the technical means disclosed in the above embodiments, but also include technical solutions composed of any combination of the above technical features. It should be pointed out that those skilled in the art can make some improvements and modifications without departing from the principles of the present application, and these improvements and modifications are also regarded as the protection scope of the present application.

The above is a detailed introduction to the desktop interactive system, implementation method and storage medium provided by the embodiment of the present application. In this paper, specific examples are used to illustrate the principle and implementation of the present application. The description of the above embodiment is only for To help understand the method and its core idea of this application; at the same time, for those of ordinary skill in the art, according to the idea of this application, there will be changes in the specific implementation and application scope. In summary, the content of this specification It should not be construed as a limitation of the application. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that: they can still modify the technical solutions described in the aforementioned embodiments, or perform equivalent replacements for some of the technical features; and these The modification or replacement does not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the various embodiments of the present application.

Claims (10)

1. A desktop interaction system, the interaction system comprising:

the interactive pen comprises at least three keys, wherein each key has a pressing state and a releasing state, and the keys are used for receiving pressing and releasing operations of a user and sending the pressing and releasing operations to the interactive equipment;

the interaction device is used for performing grabbing, rotating, zooming and returning operations on the model according to the pressing and releasing operations;

the interactive system further comprises an interactive logic frame which is used for building a class called BP _ DeskController as a class for being in butt joint with the interactive pen, and adding an IMUPenComponent component into the BP _ DeskController, wherein the component can acquire the key state of the interactive pen.

2. The system of claim 1, wherein the interactive pen comprises a first button, the first button being configured to control a grabbing model;

the interactive device is used for performing ray detection along the direction pointed by the interactive pen and storing a hit result returned by the ray detection as a first variable;

when the interactive equipment acquires that the first key is pressed down from the interactive pen, the interactive equipment judges whether the first variable is valid;

and if the first variable is valid, the interaction equipment captures a model corresponding to the ray detection hit result.

3. The system of claim 2,

and the interactive equipment is also used for capturing all models in the interactive frame when the first variable is judged to be invalid.

4. The system of claim 2 or 3,

the interactive pen is further used for acquiring the rotation angle of the interactive pen when the second key is pressed down and sending the rotation angle to the interactive equipment;

the interaction device is further used for rotating the grabbed model according to the rotation angle.

5. The system of claim 2 or 3,

the interactive pen is further configured to acquire a coordinate position of the interactive pen when the second key is pressed, acquire a variation of the coordinate position, wherein the variation of the coordinate position takes a first direction as a positive direction and a direction opposite to the first direction as a negative direction, and send the variation of the coordinate position to the interactive device;

the interactive device is further configured to map the variation of the coordinate position to the virtual coordinate of the captured model, and scale the captured model according to the variation of the coordinate position.

6. The system of claim 1, wherein the interactive pen further comprises a third button, and when the third button is clicked, the interactive device performs a return menu operation; and when the third key is pressed for a long time, the interactive device executes data calibration.

7. A desktop interaction implementation method is applied to the desktop interaction system of any one of claims 1 to 6, and comprises the following steps:

receiving pressing and releasing operations of a user on the keys, and sending the pressing and releasing operations to the interactive equipment;

and performing grabbing, rotating, zooming and returning operations on the model according to the pressing and releasing operations.

8. The method of claim 7, wherein the interactive pen comprises a first button for controlling a grabbing model;

performing ray detection along the direction pointed by the interactive pen, and storing a hit result returned by the ray detection as a first variable;

when the first key is pressed, judging whether the first variable is valid;

and when the first variable is effective, capturing a model corresponding to the ray detection hit result.

9. The method of claim 8, further comprising:

and when the first variable is judged to be invalid, capturing all models in the interactive frame.

10. A non-transitory computer-readable storage medium storing computer-executable instructions that, when executed by one or more processors, cause the one or more processors to perform the method of any one of claims 7-9.

Images