CN115494776A - AR (augmented reality) -glasses-based equipment control method, device, equipment and medium - Google Patents

Abstract

The embodiment of the invention discloses an AR glasses-based equipment control method, an AR glasses-based equipment control device, AR glasses-based equipment and an AR glasses-based medium. The method comprises the following steps: acquiring controlled equipment in a visual field, carrying out object recognition on the controlled equipment, and determining a target equipment identifier of the controlled equipment; determining a target hologram of the controlled equipment in the hologram database according to the target equipment identifier, and displaying the target hologram in a visual field; and when the target virtual button in the target hologram is detected to be selected, controlling the controlled equipment according to the control flow corresponding to the target virtual button. The method can realize virtual control on reality and improve the portability of equipment control.

Description

A device control method, device, device and medium based on AR glasses

technical field

The present invention relates to the field of augmented reality technology, in particular to a device control method, device, device and medium based on AR glasses.

Background technique

In the handling of materials, equipment such as intelligent industrial bodies, intelligent forklifts, and intelligent industrial knives are usually used.

In the prior art, these intelligent industrial devices usually implement signal transmission and real-time communication at the bottom layer through computers. Therefore, when controlling the intelligent industrial equipment, a computer must be equipped, resulting in poor portability of the control of the intelligent industrial equipment. Moreover, the intelligent industrial equipment is controlled through the bottom layer, and the control process is not intuitive.

Contents of the invention

The present invention provides a device control method, device, device and medium based on AR glasses, so as to control reality through virtual reality and improve the portability of device control.

According to one aspect of the present invention, a device control method based on AR glasses is provided, the method includes: acquiring a controlled device within the field of view, performing object recognition on the controlled device, and determining the target of the controlled device Equipment Identity;

According to the target device identification, determine the target hologram of the controlled device in the hologram database, and display the target hologram within the field of view;

When it is detected that the target virtual button in the target hologram is selected, the controlled device is controlled according to the control process corresponding to the target virtual button.

According to another aspect of the present invention, a device control device based on AR glasses is provided, the device comprising:

A device identification determination module, configured to obtain a controlled device within the field of view, perform object recognition on the controlled device, and determine a target device identification of the controlled device;

A target hologram display module, configured to determine the target hologram of the controlled device in the hologram database according to the target device identification, and display the target hologram within the field of view;

The controlled device control module is configured to control the controlled device according to the control process corresponding to the target virtual button when it is detected that the target virtual button in the target hologram is selected.

According to another aspect of the present invention, a kind of AR glasses is provided, and the AR glasses include:

at least one processor; and

a memory communicatively coupled to the at least one processor; wherein,

The memory stores a computer program that can be executed by the at least one processor, and the computer program is executed by the at least one processor, so that the at least one processor can execute the method described in any embodiment of the present invention. A device control method based on AR glasses.

According to another aspect of the present invention, a computer-readable storage medium is provided, the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement any of the embodiments of the present invention when executed. Device control method based on AR glasses.

According to the technical solution of the embodiment of the present invention, by acquiring the controlled device in the field of view and performing object recognition on the controlled device, the target device ID of the controlled device is determined; according to the target device ID, the ID of the controlled device is determined in the hologram database Target hologram, and display the target hologram in the field of vision; when the target virtual button in the target hologram is detected to be selected, the controlled device is controlled according to the control process corresponding to the target virtual button, solving the problem of virtual reality control problems and improve the portability of device control.

It should be understood that the content described in this section is not intended to identify key or important features of the embodiments of the present invention, nor is it intended to limit the scope of the present invention. Other features of the present invention will be easily understood from the following description.

Description of drawings

In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings that need to be used in the description of the embodiments will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present invention. For those skilled in the art, other drawings can also be obtained based on these drawings without creative effort.

Fig. 1a is a flow chart of a device control method based on AR glasses according to Embodiment 1 of the present invention;

Fig. 1b is a schematic diagram of a hologram provided according to Embodiment 1 of the present invention;

Fig. 1c is a schematic diagram showing a hologram according to Embodiment 1 of the present invention;

FIG. 2 is a flow chart of a device control method based on AR glasses provided according to Embodiment 2 of the present invention;

FIG. 3 is a schematic structural diagram of a device control device based on AR glasses provided according to Embodiment 3 of the present invention;

FIG. 4 is a schematic structural diagram of an electronic device implementing a device control method based on AR glasses according to an embodiment of the present invention.

detailed description

In order to enable those skilled in the art to better understand the solutions of the present invention, the following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only It is an embodiment of a part of the present invention, but not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by persons of ordinary skill in the art without making creative efforts shall fall within the protection scope of the present invention.

It should be noted that the terms "first", "second", and "target" in the description and claims of the present invention and the above drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequentially. It is to be understood that the data so used are interchangeable under appropriate circumstances such that the embodiments of the invention described herein can be practiced in sequences other than those illustrated or described herein. Furthermore, the terms "comprising" and "having", as well as any variations thereof, are intended to cover a non-exclusive inclusion, for example, a process, method, system, product or device comprising a sequence of steps or elements is not necessarily limited to the expressly listed instead, may include other steps or elements not explicitly listed or inherent to the process, method, product or apparatus.

Embodiment one

Figure 1a is a flow chart of a device control method based on AR glasses according to Embodiment 1 of the present invention. This embodiment is applicable to the situation of controlling devices through VR glasses, and the method can be controlled by a device control device based on AR glasses. For execution, the device control device based on AR glasses may be implemented in the form of hardware and/or software, and the device control device based on AR glasses may be configured in VR glasses. As shown in Figure 1a, the method includes:

Step 110, acquire the controlled device within the field of view, perform object recognition on the controlled device, and determine the target device identifier of the controlled device.

Wherein, the controlled device may be an intelligent industrial device. For example, the controlled device may be a handling industrial intelligent body, a scanning industrial intelligent body, and a functional industrial intelligent body. The controlled equipment can be binocular photogrammetry tracking automatic guided vehicle (Automated Guided Vehicle, AGV), laser scanning AGV, industrial intelligent forklift, tool or tool measuring tool transport vehicle, etc.

Specifically, binocular photogrammetry tracking AGV is a device that can obtain images of the object being photographed in real time; by solving the 2D photos obtained by photogrammetry, high-precision 3D coordinate values of the measurement points are obtained. Binocular photogrammetry tracking AGV can be mainly composed of camera, lens, light source and AGV trolley. A laser scanning AGV can be an end effector for laser data collection. Sign points containing high-reflectivity glass beads can be arranged on the actuator, and the AGV can be tracked by binocular photogrammetry to implement feedback on the position information of the actuator. The actuator can obtain the shape data of the measured object through the triangular laser principle, and track the AGV feedback through photogrammetry. The pose information of the measured object is converted and stitched into a whole. Through the cooperation of binocular photogrammetry tracking AGV and laser scanning AGV, the scanning of target equipment can be realized. Industrial intelligent forklifts can be responsible for the intelligent handling of large parts and tooling. The tool and measuring tool transport vehicle can be responsible for the intelligent distribution of tool and measuring tools.

Exemplarily, the industrial smart forklift can be controlled through VR glasses to realize the intelligent handling of large parts and tooling; Control binocular photogrammetry tracking AGV and laser scanning AGV to realize scanning of large parts. Controlling the device through VR glasses can free the user's hands and improve the user's freedom in controlling the device.

In the embodiment of the present invention, the object recognition of the controlled device by the VR glasses may be realized through an object recognition algorithm. The process of specific object recognition can be to scan the controlled device through VR glasses, obtain the feature data of the controlled device, compare the feature data, and determine which device is the controlled device, that is, determine the controlled device. Target device ID. Wherein, the target device identifier may be a device name of the controlled device. The embodiment of the present invention does not limit the specific object recognition algorithm.

Step 120: Determine the target hologram of the controlled device in the hologram database according to the target device identifier, and display the target hologram within the field of view.

Wherein, the hologram database may be a pre-generated database containing holograms of various devices. In the hologram database, each hologram may be stored according to the device identification. Specifically, the hologram database can be stored in the cloud. VR glasses can access cloud data through the network to obtain target holograms. Fig. 1b is a schematic diagram of a hologram provided according to Embodiment 1 of the present invention. As shown in Figure 1b, the hologram can be the control panel of the device. For example, virtual buttons with the following information can be included in the hologram: device name, communication status, industrial agent model, device status/running status, device parameters, and control button areas, etc.

Fig. 1c is a schematic diagram showing a hologram according to Embodiment 1 of the present invention. As shown in Figure 1c, within the field of view of the VR glasses, the controlled device can be displayed, and holograms can be displayed on both sides of the controlled device. Specifically, the hologram may be composed of two parts. The first part of the hologram may be displayed on the left side of the controlled device; the other part of the hologram may be displayed on the right side of the controlled device. The content of each part of the hologram can be set according to actual needs, which is not limited in this embodiment of the present invention.

In an optional implementation of the embodiment of the present invention, displaying the target hologram within the field of view includes: acquiring real-time location information of the controlled device, and positioning the target hologram in the field of view of the controlled device according to the real-time location information sides.

Wherein, the real-time location information of the controlled device may be the current location information fed back by the controlled device to the VR glasses in real time. Alternatively, the real-time location information of the controlled device may be location information predicted by the VR glasses in real time according to the received current location information. Positioning the entire system on both sides of the field of view of the controlled device can be achieved through a Simultaneous Localization and Mapping (SLAM) positioning algorithm. Using the SLAM positioning algorithm can complete the positioning of the corresponding hologram in the real space.

The core process of SLAM positioning algorithm mainly includes three steps: preprocessing, matching and map fusion. Specifically, the preprocessing can be to obtain the environmental information of the location through the lidar or other sensors; then optimize the raw data of the lidar, eliminate some problematic data, or perform filtering. Matching is a very critical step. Matching can be to search the point cloud data of the current local environment for the corresponding position on the established map. The quality of the matching has a direct impact on the accuracy of the SLAM map construction. In the SLAM positioning process, it is necessary to match and stitch the point cloud currently collected by the lidar into the original map. Map fusion can be to splice this round of new data from lidar into the original map, and finally complete the update of the map.

The target hologram corresponding to the controlled device can be continuously positioned on both sides of the field of view of the controlled device through the SLAM positioning algorithm, and the specific effect can be shown in Figure 1c. When the controlled device in the VR glasses changes, the corresponding hologram can be changed synchronously. That is, when the industrial intelligent forklift can be seen in the VR glasses, the hologram corresponding to the industrial intelligent forklift can be displayed; when the tool and measuring tool transport vehicle can be seen in the VR glasses, the hologram corresponding to the tool and tool measuring tool transport vehicle can be displayed. When the VR glasses turn back and you can see the industrial smart forklift, according to the SLAM positioning algorithm, it is not necessary to identify the industrial smart forklift, but directly display the hologram corresponding to the industrial smart forklift according to the positioning of the industrial smart forklift.

Step 130, when it is detected that the target virtual button in the target hologram is selected, control the controlled device according to the control flow corresponding to the target virtual button.

Among them, communication can be established between the VR glasses and the controlled device. VR glasses can have the highest authority of the controlled device. AR glasses can establish communication with the controlled device based on communication protocols such as HTTP protocol, using unity web related development technology. Specifically, by accessing and communicating with the IP address of the controlled device, the calibration of the get and post command information addresses can be completed in the low-end code part. Then, the communication between the VR glasses and the controlled device is realized according to the content of the header file in a unified format, such as a json file.

Based on the communication mechanism between VR glasses and the controlled device, developers can complete the setting and writing of intelligent industrial equipment control commands in the background code script, that is, realize the control process; edit the virtual holographic interface, that is, realize the layout of virtual buttons ; And mount virtual-real interactive scripts on relevant controls. Finally, an event connection is established between the background communication system and control code and the front-end interactive virtual hologram control.

In practical applications, the operator can intuitively view the relevant information of the controlled device through the hologram, and based on the communication established between the VR glasses and the controlled device, the operator can control the controlled device according to the virtual button in the hologram. Issue instructions to control the controlled device.

Specifically, the AR glasses can control the hologram through the ray detection algorithm, so as to realize the information query or interactive control based on the hologram, and control the controlled device. Among them, the VR glasses can detect the movement of the operator's head or glasses through the ray detection algorithm, and detect whether the movement touches the virtual button of the hologram. When the VR glasses detect that the operator touches the target virtual button in the hologram, the control process corresponding to the target virtual button can be executed.

Specifically, in an optional implementation manner of the embodiment of the present invention, when it is detected that the target virtual button in the target hologram is selected, the controlled device is controlled according to the control process corresponding to the target virtual button, including: when detecting When the virtual button corresponding to the moving equipment to the target detection point in the target hologram is selected, according to the real-time position information and the target position information of the target detection point, a device movement instruction is generated; the device movement instruction is sent to the controlled device, so that The controlled device moves to the target detection point according to the device movement instruction.

Among them, one of the more important functions of the controlled device is tooling handling, that is, the controlled device can move the tooling to the target detection point to achieve high-precision detection of large parts in the tooling in a specific environment. VR glasses can determine the movement path to move the controlled device to the target detection point according to the real-time position information of the controlled device and the target position information of the target detection point, and can generate corresponding device movement instructions according to the movement path to control the controlled device .

In an optional implementation manner of the embodiment of the present invention, controlling the controlled device according to the control flow corresponding to the target virtual button includes: controlling the controlled device according to the control flow corresponding to the target virtual button based on the 5GCPE network.

Among them, AR glasses can realize communication with the controlled device based on the 5G network environment. Specifically, the AR glasses can use 5G Customer Premise Equipment (CPE) to realize communication with the controlled device. The 5GCPE network can perform secondary relay (enhancement) on wireless signals (such as WIFI) to extend the coverage of wireless signals. Through the 5GCPE network, the stability of the network can be improved, the normal operation of the controlled device can be guaranteed, the delay can be reduced, and the control of the device can be kept in a stable state.

In the technical solution of this embodiment, the target device identification of the controlled device is determined by acquiring the controlled device in the field of view and performing object recognition on the controlled device; according to the target device identification, the target of the controlled device is determined in the hologram database hologram, and display the target hologram in the field of view; when the target virtual button in the target hologram is detected to be selected, the controlled device is controlled according to the control process corresponding to the target virtual button, which solves the problem of virtual reality. For control issues, viewing device information through AR glasses can free your hands and improve the portability of device control; replacing abstract codes with intuitive holograms to control devices can reduce the technical level requirements for operators and the cost of device control. Threshold, so as to avoid spending a long time on personnel training and reduce costs; and, through the hologram of VR glasses to intervene in the control process of the device, the device can be controlled intuitively, improving work efficiency and convenience of operation.

Embodiment two

Fig. 2 is a flow chart of a device control method based on AR glasses according to Embodiment 2 of the present invention. This embodiment is a further refinement of the above-mentioned technical solution. Various alternatives in one embodiment are combined. As shown in Figure 2, the method includes:

Step 210: Obtain the controlled device within the field of view, perform object recognition on the controlled device, and determine the target device identifier of the controlled device.

In an optional implementation manner of the embodiment of the present invention, before obtaining the controlled device in the field of view, performing object recognition on the controlled device, and determining the device identifier of the controlled device, it further includes: performing at least one device to be identified Object recognition, determine the device identification of the device to be identified; obtain the control application that matches the device to be identified, and generate a hologram of the device to be identified according to the control application; store the hologram according to the device identification of the device to be identified, and generate a hologram database .

Among them, the VR glasses can realize the object recognition of the device to be recognized through the object recognition algorithm, extract the object features of the device to be recognized, and store them. Specifically, the storage may be performed according to the device identification of the device to be identified. Different devices to be identified may have different control applications. Among them, the control application can be to check the device name, check the communication status, check the industrial agent model, check the status/running status of the equipment, check the parameters of the equipment or move the equipment, etc. Corresponding virtual buttons can be generated according to different control applications. Virtual buttons can be displayed in the hologram. The corresponding control application can be realized by mounting the virtual-real interaction script on the virtual button.

Step 220: Determine the target hologram of the controlled device in the hologram database according to the target device identifier.

Step 230, acquiring real-time location information of the controlled device, and positioning the target hologram on both sides of the field of view of the controlled device according to the real-time location information.

In an optional implementation of the embodiment of the present invention, obtaining the real-time location information of the controlled device includes: obtaining the current location information, moving speed and timestamp information corresponding to the current location information of the controlled device; , timestamp information, and moving speed to predict the real-time location information of the controlled device.

Wherein, in the Json file of the interaction between the controlled device and the VR glasses, the current location information, moving speed, and timestamp information corresponding to the current location information of the controlled device may be included. The current location information may include coordinate information and posture information of the device. The movement speed may include the angular velocity, the linear velocity, etc. of the movement of the device. VR glasses can determine the time difference between file transfers based on real-time time information and timestamp information. The moving position information of the device during the file transfer can be determined according to the file transfer time difference and the moving speed. According to the location information of the equipment movement, the current location information can be compensated, and the real-time location information of the controlled equipment can be predicted. Thereby, the accuracy of the real-time location information of the controlled equipment is improved, the communication delay is prevented from causing inaccurate location information of the controlled equipment, and the reliability of the control is improved.

In an optional implementation manner of the embodiment of the present invention, positioning the target hologram on both sides of the field of view of the controlled device according to the real-time position information includes: performing coordinate transformation on the real-time position information to generate VR glasses coordinates of the controlled device; According to the coordinates of the VR glasses, the controlled device is displayed in the field of view, and the target hologram is positioned on both sides of the field of view of the controlled device.

Wherein, the coordinates in the location information of the controlled device and the coordinates in the VR glasses belong to different coordinate systems, and the origins and directions of the two coordinates are different. Therefore, coordinate conversion is required to convert the real-time position information of the controlled device into the coordinates in the VR glasses coordinate system, so as to realize the accurate display of the controlled device. Specifically, a mathematical method may be used to realize the coordinate conversion. For example, the coordinate transformation is realized by using the RT matrix. Wherein, the RT matrix may be a matrix including two aspects of translation and rotation. R can be a rotation matrix, that is, the Euler angles are expanded into values in the form of a matrix, and these values can be obtained by multiplying and adding the sin or cos of the angles. T can be a displacement transformation vector.

Step 240, when it is detected that the virtual button corresponding to the moving tool in the target hologram to the target detection point is selected, generate a device movement instruction according to the real-time position information and the target position information of the target detection point.

Step 250, sending the device movement instruction to the controlled device, so that the controlled device moves to the target detection point according to the device movement instruction.

In an optional implementation of the embodiment of the present invention, sending the device movement instruction to the controlled device, so that the controlled device moves to the target detection point according to the device movement instruction, includes: based on the 5GCPE network, sending the device movement instruction to The controlled device, so that the controlled device moves to the target detection point according to the device movement instruction.

In the technical solution of this embodiment, the target device identification of the controlled device is determined by acquiring the controlled device in the field of view and performing object recognition on the controlled device; according to the target device identification, the target of the controlled device is determined in the hologram database Hologram; obtain the real-time position information of the controlled device, and position the target hologram on both sides of the field of view of the controlled device according to the real-time position information; when the moving tool in the target hologram is detected to the virtual button corresponding to the target detection point When selected, according to the real-time location information and the target location information of the target detection point, a device movement command is generated; the device movement command is sent to the controlled device, so that the controlled device moves to the target detection point according to the device movement command, which solves the problem of virtual pair For realistic control problems, viewing device information through AR glasses can free your hands and improve the portability of device control; controlling the device through intuitive holograms instead of abstract codes can reduce the technical level requirements for operators, and the equipment The threshold of control can avoid spending a long time on personnel training and reduce costs; moreover, through the hologram of VR glasses intervening in the control process of the equipment, the equipment can be controlled intuitively, improving work efficiency and convenience of operation.

According to the technical solution of the embodiment of the present invention, AR glasses are used for object recognition and corresponding holograms are generated at the same time. After the communication between the AR glasses and the industrial smart device is completed, the operator can click the virtual button in the VR glasses to check the related equipment. give command. Using the device control method based on AR glasses provided by the present invention, the operator can not only intuitively obtain various information of the device from the hologram to improve work efficiency; but also directly control single or multiple industrial smart devices through AR glasses, There is no need to carry computers or other mobile devices; operators only need to wear lightweight AR glasses to view relevant information about smart industrial devices, which greatly improves the freedom of hands during work. At the same time, it can avoid the need for traditional AR glasses to require users to install control applications (apps) on mobile phones or other mobile terminals, and send instructions to control industrial smart devices through the app, resulting in the need for communication between AR glasses and industrial smart devices. A mobile terminal is used to issue instructions, causing the operator to continuously use the mobile terminal to send instructions during work, reducing work efficiency.

Embodiment three

Fig. 3 is a schematic structural diagram of a device control device based on AR glasses provided according to Embodiment 3 of the present invention. As shown in FIG. 3 , the device includes: a device identification determination module 310 , a target hologram display module 320 and a controlled device control module 330 . in:

A device identification determination module 310, configured to obtain a controlled device within the field of view, perform object recognition on the controlled device, and determine a target device identification of the controlled device;

The target hologram display module 320 is used to determine the target hologram of the controlled device in the hologram database according to the target device identification, and display the target hologram within the field of view;

The controlled device control module 330 is configured to control the controlled device according to the control process corresponding to the target virtual button when it is detected that the target virtual button in the target hologram is selected.

Optionally, the device also includes:

Yet another device identification determination module, used to perform object identification on at least one device to be identified and determine the device to be identified before acquiring the controlled device within the field of view, performing object identification on the controlled device, and determining the device identification of the controlled device equipment identification;

A hologram generating module, configured to acquire a control application matching the device to be identified, and generate a hologram of the device to be identified according to the control application;

The hologram database generation module is configured to store the hologram according to the device identification of the device to be identified, and generate a hologram database.

Optionally, the target hologram display module 320 includes:

The target hologram display unit is configured to obtain real-time position information of the controlled device, and position the target hologram on both sides of the field of view of the controlled device according to the real-time position information.

Optionally, the target hologram display unit includes:

An information acquisition subunit, configured to acquire the current location information, moving speed, and time stamp information corresponding to the current location information of the controlled device;

The real-time location information prediction subunit is used to predict the real-time location information of the controlled device according to the current location information, time stamp information, and moving speed.

Optionally, the target hologram display unit includes:

The VR glasses coordinate generating subunit is used to convert the coordinates of the real-time position information to generate the VR glasses coordinates of the controlled device;

The target hologram display subunit is used to display the controlled device in the field of view according to the coordinates of the VR glasses, and position the target hologram on both sides of the field of view of the controlled device.

Optionally, the controlled device control module 330 includes:

The device movement instruction generation unit is used to generate a device movement instruction according to the real-time position information and the target position information of the target detection point when it is detected that the virtual button corresponding to the moving tool in the target hologram to the target detection point is selected;

The device movement instruction sending unit is configured to send the device movement instruction to the controlled device, so that the controlled device moves to the target detection point according to the device movement instruction.

Optionally, the controlled device control module 330 includes:

The controlled device control unit is used to control the controlled device according to the control process corresponding to the target virtual button based on the 5GCPE network.

The device control device based on AR glasses provided in the embodiments of the present invention can execute the device control method based on AR glasses provided in any embodiment of the present invention, and has corresponding functional modules and beneficial effects for executing the method.

Embodiment four

FIG. 4 shows a schematic structural diagram of an electronic device 10 that can be used to implement an embodiment of the present invention. Electronic device is intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other suitable computers. Electronic devices may also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices (eg, helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions, are by way of example only, and are not intended to limit implementations of the inventions described and/or claimed herein.

As shown in FIG. 4, the electronic device 10 includes at least one processor 11, and a memory connected in communication with the at least one processor 11, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., wherein the memory stores There is a computer program executable by at least one processor, and the processor 11 can operate according to a computer program stored in a read-only memory (ROM) 12 or loaded from a storage unit 18 into a random access memory (RAM) 13. Various appropriate actions and processes are performed. In the RAM 13, various programs and data necessary for the operation of the electronic device 10 are also stored. The processor 11 , ROM 12 , and RAM 13 are connected to each other through a bus 14 . An input/output (I/O) interface 15 is also connected to the bus 14 .

Multiple components in the electronic device 10 are connected to the I/O interface 15, including: an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a magnetic disk, an optical disk etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, and the like. The communication unit 19 allows the electronic device 10 to exchange information/data with other devices through a computer network such as the Internet and/or various telecommunication networks.

Processor 11 may be various general and/or special purpose processing components having processing and computing capabilities. Some examples of processor 11 include, but are not limited to, central processing units (CPUs), graphics processing units (GPUs), various dedicated artificial intelligence (AI) computing chips, various processors that run machine learning model algorithms, digital signal processing processor (DSP), and any suitable processor, controller, microcontroller, etc. The processor 11 executes various methods and processes described above, such as a device control method based on AR glasses.

In some embodiments, the AR glasses-based device control method can be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as the storage unit 18 . In some embodiments, part or all of the computer program may be loaded and/or installed on the electronic device 10 via the ROM 12 and/or the communication unit 19 . When the computer program is loaded into the RAM 13 and executed by the processor 11, one or more steps of the above-described AR glasses-based device control method can be executed. Alternatively, in other embodiments, the processor 11 may be configured in any other appropriate way (for example, by means of firmware) to execute the AR glasses-based device control method.

Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips Implemented in a system of systems (SOC), load programmable logic device (CPLD), computer hardware, firmware, software, and/or combinations thereof. These various embodiments may include being implemented in one or more computer programs executable and/or interpreted on a programmable system including at least one programmable processor, the programmable processor Can be special-purpose or general-purpose programmable processor, can receive data and instruction from storage system, at least one input device, and at least one output device, and transmit data and instruction to this storage system, this at least one input device, and this at least one output device an output device.

Computer programs for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus, so that the computer program causes the functions/operations specified in the flowcharts and/or block diagrams to be implemented when executed by the processor. A computer program may execute entirely on the machine, partly on the machine, as a stand-alone software package partly on the machine and partly on a remote machine or entirely on the remote machine or server.

In the context of the present invention, a computer readable storage medium may be a tangible medium which may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus or device. A computer readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. Alternatively, a computer readable storage medium may be a machine readable signal medium. More specific examples of machine-readable storage media would include one or more wire-based electrical connections, portable computer discs, hard drives, random access memory (RAM), read only memory (ROM), erasable programmable read only memory (EPROM or flash memory), optical fiber, compact disk read only memory (CD-ROM), optical storage, magnetic storage, or any suitable combination of the foregoing.

In order to provide interaction with the user, the systems and techniques described herein can be implemented on an electronic device having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display)) for displaying information to the user. monitor); and a keyboard and pointing device (eg, a mouse or a trackball) through which the user can provide input to the electronic device. Other kinds of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and can be in any form (including Acoustic input, speech input or, tactile input) to receive input from the user.

The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., as a a user computer having a graphical user interface or web browser through which a user can interact with embodiments of the systems and techniques described herein), or including such backend components, middleware components, Or any combination of front-end components in a computing system. The components of the system can be interconnected by any form or medium of digital data communication, eg, a communication network. Examples of communication networks include: local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

A computing system can include clients and servers. Clients and servers are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by computer programs running on the respective computers and having a client-server relationship to each other. The server can be a cloud server, also known as a cloud computing server or a cloud host. It is a host product in the cloud computing service system to solve the problems of difficult management and weak business expansion in traditional physical hosts and VPS services. defect.

It should be understood that steps may be reordered, added or deleted using the various forms of flow shown above. For example, each step described in the present invention may be executed in parallel, sequentially, or in a different order, as long as the desired result of the technical solution of the present invention can be achieved, there is no limitation herein.

The above specific implementation methods do not constitute a limitation to the protection scope of the present invention. It should be apparent to those skilled in the art that various modifications, combinations, sub-combinations and substitutions may be made depending on design requirements and other factors. Any modifications, equivalent replacements and improvements made within the spirit and principles of the present invention shall be included within the protection scope of the present invention.

Claims (10)

1. A device control method based on AR glasses, characterized in that, comprising: Obtain the controlled device within the field of view, and perform object recognition on the controlled device, and determine the target device identifier of the controlled device; According to the target device identification, determine the target hologram of the controlled device in the hologram database, and display the target hologram within the field of view; When it is detected that the target virtual button in the target hologram is selected, the controlled device is controlled according to the control process corresponding to the target virtual button. 2. The method according to claim 1, characterized in that before acquiring the controlled device within the field of view, performing object recognition on the controlled device, and determining the device identifier of the controlled device, further comprising: Perform object recognition on at least one device to be recognized, and determine the device identifier of the device to be recognized; Acquire a control application that matches the device to be identified, and generate a hologram of the device to be identified according to the control application; The hologram is stored according to the device identification of the device to be identified to generate a hologram database. 3. The method according to claim 1, wherein displaying the target hologram within the field of view comprises: Acquire real-time location information of the controlled device, and position the target hologram on both sides of the field of view of the controlled device according to the real-time location information. 4. The method according to claim 3, wherein obtaining the real-time location information of the controlled device comprises: Obtaining the current location information, moving speed, and time stamp information corresponding to the current location information of the controlled device; Predict real-time location information of the controlled device according to the current location information, the time stamp information, and the moving speed. 5. The method according to claim 3, wherein positioning the target hologram on both sides of the field of view of the controlled device according to the real-time position information comprises: performing coordinate transformation on the real-time position information to generate VR glasses coordinates of the controlled device; According to the coordinates of the VR glasses, the controlled device is displayed in the field of view, and the target hologram is positioned on both sides of the field of view of the controlled device. 6. The method according to claim 3, wherein when it is detected that the target virtual button in the target hologram is selected, the controlled device is controlled according to the control process corresponding to the target virtual button ,include: When it is detected that the virtual button corresponding to the moving tool in the target hologram to the target detection point is selected, a device movement instruction is generated according to the real-time position information and the target position information of the target detection point; Sending the device movement instruction to the controlled device, so that the controlled device moves to a target detection point according to the device movement instruction. 7. The method according to claim 1, wherein controlling the controlled device according to the control process corresponding to the target virtual button includes: Based on the 5GCPE network, the controlled device is controlled according to the control process corresponding to the target virtual button. 8. A device control device based on AR glasses, characterized in that it comprises: A device identification determination module, configured to obtain a controlled device within the field of view, perform object recognition on the controlled device, and determine a target device identification of the controlled device; A target hologram display module, configured to determine the target hologram of the controlled device in the hologram database according to the target device identification, and display the target hologram within the field of view; The controlled device control module is configured to control the controlled device according to the control process corresponding to the target virtual button when it is detected that the target virtual button in the target hologram is selected. 9. An AR glasses, characterized in that the AR glasses include: at least one processor; and a memory communicatively coupled to the at least one processor; wherein, The memory stores a computer program executable by the at least one processor, the computer program is executed by the at least one processor, so that the at least one processor can perform any one of claims 1-7 The device control method based on AR glasses. 10. A computer-readable storage medium, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the method described in any one of claims 1-7 when executed. A device control method based on AR glasses.

Images