CN109408128B - Split AR (augmented reality) device communication method and AR device - Google Patents

Abstract

The invention discloses a split AR device communication method and an AR device, wherein the communication method comprises the following steps: virtualizing the AR head-mounted display into a plurality of standard devices; the controller detects that the communication interface has external equipment inserted, and sends an enumeration request to the external equipment; the AR head-mounted display feeds back equipment information to the controller, wherein the equipment information at least comprises information of all standard equipment; the controller searches a corresponding device driver from a local support device list according to the device information, and loads the driver; the controller communicates with each standard device. According to the split type AR device, the controller and the battery are separated from the head-mounted display, so that the weight of the head-mounted display is greatly reduced, the head-mounted display is lighter and lighter, the AR head-mounted display is virtualized into a plurality of standard devices in a communication mode of the head-mounted display and the controller, the two parts can directly communicate with each other, the data transmission rate requirement can be met, and the real-time performance of data transmission is guaranteed.

Description

Split AR device communication method and AR device

technical field

The present invention relates to the technical field of electronic devices, and in particular, to a communication method for a split AR device and an AR device.

Background technique

At present, the mainstream head-mounted AR devices on the market mainly use the structure of an all-in-one machine to package the controller, battery and display screen as a whole. This all-in-one machine has a complex structure and heavy weight. The user wears both the main unit and the display screen on the head, which is too bulky, and has a strong sense of oppression on the user's head, which seriously affects the user's sensory effect.

SUMMARY OF THE INVENTION

In order to solve the technical problems of complex structure and inconvenient wearing of the existing integrated head-mounted AR device, the present invention proposes a communication method for a split AR device, which can solve the above problems.

In order to solve the above-mentioned technical problems, the present invention adopts the following technical solutions to be realized:

A method for communicating with a split-type AR device, the split-type AR device includes an AR head-mounted display and a controller set separately, the AR head-mounted display and the controller are connected through a data cable, and the split-type AR device The communication method includes the following steps:

Virtualize the AR head-mounted display into several standard devices;

The controller detects that an external device is inserted into the communication interface, and sends an enumeration request to the external device;

The AR head-mounted display feeds back device information to the controller, where the device information at least includes information of all standard devices;

The controller searches for the corresponding device driver from the local supported device list according to the device information, and loads the driver;

The controller communicates with various standard devices.

Further, the communication interface between the AR head-mounted display and the controller is a USB interface.

Further, the AR head-mounted display includes two cameras, each camera is virtualized as a standard USB device, each standard USB device communicates with the controller through a USB endpoint, and the camera data is transmitted using the USBUVC protocol.

Further, the two cameras are RGB Camera and TOF Camera respectively.

Further, the AR head-mounted display includes several sensors, and all the sensors are virtualized as a standard USB device, the standard USB device communicates with the controller using a USB endpoint, and the sensor data is transmitted through the USB hid raw protocol.

Further, the control signal and the sensor data share the same USB endpoint for transmission.

Further, the controller monitors the communication interface for monitoring the plugging and unplugging status of the external device.

Further, when each standard device packages the data sent to the controller, a time stamp is added at the same time, and when the controller parses the data, the data of all standard devices is synchronized according to the time stamp.

Further, the device information includes device descriptors, configuration descriptors, interface descriptors and endpoint descriptors, and the controller loads a driver according to the device information, and communicates with each standard device through corresponding endpoints and interfaces .

The present invention also proposes an AR device, including an AR head-mounted display and a controller that are arranged separately, the AR head-mounted display and the controller are connected through a data cable, and the AR head-mounted display at least includes a display screen, A camera and a sensor, the controller is connected to a battery, and the AR device communicates according to the communication method for a split AR device described in any of the preceding items.

Compared with the prior art, the advantages and positive effects of the present invention are: the split AR device of the present invention separates the controller and the battery from the head-mounted display, and the head-mounted display will be greatly reduced without the above-mentioned devices. The weight of the AR head-mounted display makes the head-mounted display lighter. The communication method between the head-mounted display and the controller is by virtualizing the AR head-mounted display into several standard devices. The controller loads the corresponding drivers for each standard device, and the two parties can communicate directly. There is no need to re-encapsulate the data of the sensor and camera, and it is fully compatible with the android software. Each standard device is used to transmit different data, which can meet the data transmission rate requirements, ensure the real-time data transmission, and prevent jamming.

Other features and advantages of the present invention will become more apparent upon reading the detailed description of the embodiments of the present invention in conjunction with the accompanying drawings.

Description of drawings

In order to explain the embodiments of the present invention or the technical solutions in the prior art more clearly, the following briefly introduces the accompanying drawings that need to be used in the description of the embodiments or the prior art. Obviously, the accompanying drawings in the following description are only These are some embodiments of the present invention. For those of ordinary skill in the art, other drawings can also be obtained according to these drawings without creative efforts.

1 is a schematic block diagram of an embodiment of a split AR device proposed by the present invention;

FIG. 2 is a schematic diagram of a signal transmission direction in an embodiment of a communication method for a split AR device proposed by the present invention.

Detailed ways

The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, but not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

Embodiment 1. This embodiment proposes a communication method for a split AR device. As shown in FIG. 1 , the split AR device includes an AR head-mounted display and a controller that are arranged separately. The AR head-mounted display and the controller pass through The data line connection, the split AR device communication method includes the following steps:

The AR head-mounted display is virtualized into several standard devices; in this embodiment, the head-mounted display is composed of the following main modules: the left and right display screens are used to display the augmented reality picture; the two cameras are RGB camera and TOF camera, among them, RGB camera is used to shoot real-time scene, TOF camera is used to measure the depth of field to locate the spatial position of the object; acceleration sensor, gyroscope, gravity sensor, backlight sensor and proximity sensor, among which, acceleration sensor, gyroscope, gravity The sensor is integrated in the IMU chip to measure the user's own position information; the backlight sensor and the proximity sensor are integrated in the ALS+PS chip, and the speaker and mic are used to input and output voice signals respectively. Since the AR head-mounted display has many modules In this solution, by virtualizing the module into several standard devices, a standard interface can be used to communicate with the control module using a standard protocol, without the need to repackage the data of each module of the head-mounted display, which can achieve real-time transmission and processing of users. spatial location and scene information. Achieve better augmented reality effects.

In this solution, the head-mounted display and the controller use traditional communication interfaces to connect and communicate: the data transmission scheme between the head-mounted display and the controller requires the development of both ends, the controller is the host side, and the head-mounted display is the device side .

After the head-mounted display and the controller are powered on, as shown in Figure 2, when the head-mounted display and the controller are connected by a data cable, the controller detects that an external device is inserted into the communication interface, and sends an enumeration request to the external device;

The AR head-mounted display feeds back the device information to the controller according to the received enumeration request, and the device information includes at least the information of all virtual standard devices;

The controller searches for the corresponding device driver from the local supported device list according to the device information fed back by the AR head-mounted display, and loads the driver;

Since the controller respectively loads the driver programs for each standard device, the controller and each standard device can communicate directly through the communication interface.

In the split AR device of this embodiment, the battery is arranged on the controller end and is connected to the controller to supply power. Due to the heavy weight of the battery, by separating the controller and the battery from the head-mounted display, it will greatly reduce the energy consumption. The weight of the head-mounted display makes the head-mounted display lighter, thus solving the technical problem that the existing one-piece head-mounted display has a strong sense of pressure on the user's head and affects the wearing effect. The head-mounted display communicates with the controller Method By virtualizing the AR head-mounted display into several standard devices, the controller loads the corresponding drivers for each standard device, and the two parties can communicate directly without re-encapsulating the data of the sensor and camera, and fully compatible with android software. , each standard device is used to transmit different data, which can meet the data transmission rate requirements, ensure the real-time data transmission, and prevent jamming.

The USB interface is widely used in communication, supports hot plugging, and is convenient to plug and unplug. As a preferred embodiment, the communication interface between the AR head-mounted display and the controller in this embodiment is a USB interface. Among them, the controller can choose Qualcomm 845 chip. The head-mounted display uses the 6-axis sensor and camera to obtain the corresponding data in real time, and then transmits the obtained data to the controller through the usb. The controller calculates the user's spatial geographic location in real time through the 6DOF algorithm, and then integrates the user's spatial information into In a specific scene, it is then transferred to the head-mounted display through USB to realize augmented reality.

The camera in the head-mounted display needs to transmit the largest amount of data. In this embodiment, each camera is virtualized as a standard USB device. Each standard USB device communicates with the controller through a USB endpoint, and the camera data uses USB UVC protocol for transmission.

The AR head-mounted display includes several sensors, and all the sensors are virtualized as a standard USB device. The standard USB device uses a USB endpoint to communicate with the controller, and the sensor data is transmitted through the USB hid raw protocol.

In this solution, the basis for virtualizing each module in the head-mounted display into a standard device is based on the data type and data volume it transmits, and the USB UVC protocol and USB hid raw protocol used in the transmission between the standard device and the controller are both The standard protocol at the bottom of the USB, when using the USB communication interface to transmit data, there is no need to re-encapsulate the data of the sensor and the camera, and it is fully compatible and supported with android software.

Using the USB communication interface to communicate the head-mounted display and the controller can be compatible with the charger of the traditional one-piece head-mounted display, and the traditional charger can be used to charge it.

That is, the modules in the head-mounted display in this embodiment are virtualized into three standard devices, and three USB endpoints (endpoints) are used between the corresponding head-mounted display and the controller to output data, and each standard device uses one endpoint to transmit data. , 2 endpoints are used to transmit camera data of UVC class, and 1 endpoint is used to transmit sensor data of hid raw class.

For example, three standard devices respectively use three USB endpoints EP0, EP1, EP2 to transmit data:

EP0: used to transmit sensor data, including 6DOF and control commands, the data volume is expected to be 48kbs-22Mbps.

EP1: Used to transmit RGB Camera data, 1080/30fps or 1080/60fps preview, the image format is MJPEG, and the data volume is 300-995Mbs.

EP2: used to transmit TOF Camera data at 60fps.

Since the data amount of the control signal is small, in this embodiment, the control signal and the sensor data share the same USB endpoint for transmission. There is no need to re-occupy an endpoint for the control signal, which saves the channel and reduces the occupation of the bandwidth.

After the controller is powered on, it also includes the step of monitoring the communication interface in real time, so as to monitor the plugging and unplugging state of the external device. If the external device is not detected, it will continue to monitor. If the external device is detected, it will send a command to the external device to establish a connection with the head-mounted display, and load the corresponding driver (UVC and hid raw) according to the USB enumeration. . If the head-mounted display responds normally, there will be usb communication between the controller and the head-mounted display, and the display will send sensor and camera data to the controller. The controller performs normal data processing.

In order to ensure the synchronization of sensor data and camera data, each standard device adds a timestamp when packaging the data sent to the controller. When the controller parses the data, it synchronizes the data of all standard devices according to the timestamp.

The AR head-mounted display feeds back the device information to the controller, the device information includes device descriptors, configuration descriptors, interface descriptors and endpoint descriptors. The controller loads the driver according to the configuration descriptor in the device information, and according to The interface description conforms to the endpoint descriptor and communicates with each standard device through the corresponding endpoint and interface.

Embodiment 2, this embodiment proposes an AR device, as shown in FIG. 1 , including an AR head-mounted display and a controller that are arranged separately. The AR head-mounted display and the controller are connected through a data cable, and the AR head-mounted display at least It includes a display screen, a camera and a sensor. The display screen includes two left and right, which are used to display augmented reality images; two cameras are RGB camera and TOF camera. to measure the depth of field to locate the spatial position of the object; there are 6 sensors, namely acceleration sensor, gyroscope, gravity sensor, backlight sensor and proximity sensor, used to measure the user's own position information; the head-mounted display also includes speaker and mic, respectively Used to input and output voice signals, the battery is set at the same end as the controller, and the battery is connected to the controller for powering it. By separating the controller and the battery from the head-mounted display, the head-mounted display will be greatly reduced The weight of the display makes the head-mounted display lighter, and the AR device communicates according to the communication method of the split AR device described in the first embodiment. For details, please refer to the first embodiment, which will not be repeated here.

Of course, the above description is not intended to limit the present invention, and the present invention is not limited to the above examples. Changes, modifications, additions or substitutions made by those of ordinary skill in the art within the scope of the present invention should also belong to protection scope of the present invention.

Claims (8)

1. A split-type AR device communication method, wherein the split-type AR device comprises an AR head-mounted display and a controller set separately, and the AR head-mounted display and the controller are connected by a data cable, The split AR device communication method includes the following steps: Virtualize the AR head-mounted display into several standard devices according to the type of transmitted data; The controller monitors the communication interface for monitoring the plugging state of the external device; the controller detects that an external device is inserted into the communication interface, and sends an enumeration request to the external device; The AR head-mounted display feeds back device information to the controller, and the device information includes at least the information of all standard devices; when each standard device packages the data sent to the controller, a timestamp is added at the same time; The controller searches for the corresponding device driver from the local supported device list according to the device information, and loads the driver; the controller loads the driver according to the device information, and communicates with each standard device through the corresponding endpoint communicate with the interface; and when the controller parses the data, synchronizes the data of all standard devices according to the timestamp; The controller uses standard interfaces and standard protocols to communicate with various standard devices. 2 . The method for communicating with a split AR device according to claim 1 , wherein the communication interface between the AR head-mounted display and the controller is a USB interface. 3 . 3. The split AR device communication method according to claim 2, wherein the AR head-mounted display comprises two cameras, each camera is virtualized as a standard USB device, and each standard USB device passes through a The USB endpoint communicates with the controller, and the camera data is transmitted using the USB UVC protocol. 4. The split AR device communication method according to claim 3, wherein the two cameras are RGB Camera and TOF Camera respectively. 5 . The split AR device communication method according to claim 2 , wherein the AR head-mounted display comprises several sensors, and all the sensors are virtualized as a standard USB device, and the standard USB device adopts a USB endpoint to communicate with all the sensors. 6 . The controller communicates with the controller, and the sensor data is transmitted through the USB hid raw protocol. 6 . The method for communicating with a split AR device according to claim 5 , wherein the control signal and the sensor data share the same USB endpoint for transmission. 7 . 7 . The method for communicating with a split AR device according to claim 1 , wherein the device information includes a device descriptor, a configuration descriptor, an interface descriptor, and an endpoint descriptor. 8 . 8. An AR device, comprising an AR head-mounted display and a controller that are arranged separately, the AR head-mounted display and the controller are connected by a data cable, and the AR head-mounted display at least comprises a display screen , a camera and a sensor, the controller is connected to a battery, and the AR device communicates according to the communication method for a split AR device according to any one of claims 1-7.

Images