CN114419469A - Target identification method and device, AR device and readable storage medium - Google Patents

Abstract

The present disclosure provides a target recognition method, apparatus, AR device and readable storage medium. The method includes: when a target recognition task is received, obtaining ambient light intensity; when the ambient light intensity is less than a preset value, obtaining two-dimensional image data and depth image data of a target object; based on the two The dimensional image data and the depth image data are used to obtain the recognition result of the target recognition task.

Description

Target recognition method, device, AR device and readable storage medium

technical field

Embodiments of the present disclosure relate to the field of identification technologies, and more particularly, to a target identification method, apparatus, AR device, and readable storage medium.

Background technique

Augmented Reality (AR) devices can use AR technology to simulate and simulate virtual information such as text, images, 3D models, and multimedia generated by computers, and then map them to the real world to realize the fusion of virtual information and the real world.

An RGB camera is usually installed in an AR device for capturing images. The RGB camera can obtain two-dimensional image information of an object, and the processor calculates the two-dimensional image information through a specific algorithm to realize target recognition.

However, in a low-light environment, the information amount of the two-dimensional image information collected by the RGB camera will be reduced, which will lead to a reduction in the target recognition rate. Therefore, it is necessary to propose a new object recognition method.

SUMMARY OF THE INVENTION

An object of the embodiments of the present disclosure is to provide a target recognition method, so as to improve the target recognition rate.

According to a first aspect of the embodiments of the present disclosure, there is provided a target recognition method, which is applied to an AR device, and the method includes:

In the case of receiving the target recognition task, obtain the ambient light intensity;

Obtaining two-dimensional image data and depth image data of the target object when the ambient light intensity is less than a preset value;

The recognition result of the target recognition task is obtained based on the two-dimensional image data and the depth image data.

Optionally, after obtaining the ambient light intensity, the method further includes:

When the ambient light intensity is greater than the preset value, acquiring two-dimensional image data of the target object;

The recognition result of the target recognition task is obtained based on the two-dimensional image data.

Optionally, obtaining the recognition result of the target recognition task based on the two-dimensional image data and the depth image data, including:

The two-dimensional image data and the depth image data are processed based on the first target recognition algorithm to obtain a recognition result of the target recognition task.

Optionally, obtaining the recognition result of the target recognition task based on the two-dimensional image data, including:

The two-dimensional image data is processed based on the second target recognition algorithm to obtain a recognition result of the target recognition task.

According to a second aspect of the embodiments of the present disclosure, there is also provided a target recognition device, which is arranged in an AR device, and the device includes:

an acquisition module, configured to acquire the ambient light intensity when the target recognition task is received; when the ambient light intensity is less than a preset value, acquire two-dimensional image data and depth image data of the target object;

A recognition module, configured to obtain a recognition result of the target recognition task based on the two-dimensional image data and the depth image data.

Optionally, the obtaining module is also used for:

When the ambient light intensity is greater than the preset value, acquiring two-dimensional image data of the target object;

The recognition module is further configured to obtain a recognition result of the target recognition task based on the two-dimensional image data.

Optionally, the identification module is specifically used for:

The two-dimensional image data and the depth image data are processed based on the first target recognition algorithm to obtain a recognition result of the target recognition task.

Optionally, the identification module is specifically used for:

The two-dimensional image data is processed based on the second target recognition algorithm to obtain a recognition result of the target recognition task.

According to a third aspect of the embodiments of the present disclosure, an AR device is further provided, including an ambient light sensor, an RGB camera, and a time-of-flight sensor, and the AR device further includes:

memory for storing executable computer instructions;

a processor, configured to execute the target identification method according to any one of the first aspects of the embodiments of the present disclosure according to the control of the executable computer instructions;

Wherein, the ambient light sensor, the RGB camera and the time-of-flight sensor are all connected to the processor.

According to a fourth aspect of the embodiments of the present disclosure, there is also provided a readable storage medium, where a program or an instruction is stored on the readable storage medium, and when the program or instruction is executed by a processor, the first embodiment of the present disclosure is implemented The steps of the target recognition method of any one of the aspects.

One beneficial effect of the embodiments of the present disclosure is that the ambient light intensity can be acquired when the target recognition task is received; and the two-dimensional image data and depth of the target object can be acquired when the ambient light intensity is less than a preset value. image data; obtaining the recognition result of the target recognition task based on the two-dimensional image data and the depth image data. In this way, the target object can be identified based on the two-dimensional image data and the depth image data when the ambient light intensity is less than the preset value, so as to avoid the fact that the amount of information of the target object collected when the light intensity is low is relatively high. The target recognition rate is reduced due to less, and the target recognition rate is significantly improved.

Other features and advantages of the present specification will become apparent from the following detailed description of exemplary embodiments of the present specification with reference to the accompanying drawings.

Description of drawings

The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments of this specification and, together with the description, serve to explain the principles of this specification.

1 is a block diagram of a hardware configuration of an AR device according to an embodiment of the present disclosure;

FIG. 2 shows a schematic flowchart of a target recognition method according to an embodiment of the present disclosure;

FIG. 3 is a schematic flowchart of a building process of a recognition model according to an embodiment of the present disclosure;

4 is a schematic diagram of a convolutional neural network of a recognition model used by the target recognition method according to an embodiment of the present disclosure;

5 is a schematic block diagram of a target identification device according to an embodiment of the present disclosure;

FIG. 6 is a schematic diagram of a hardware structure of an AR device according to an embodiment.

Detailed ways

Various exemplary embodiments of the present disclosure will now be described in detail with reference to the accompanying drawings. It should be noted that the relative arrangements of components and steps, numerical expressions and numerical values set forth in these embodiments do not limit the scope of the embodiments of the present disclosure unless specifically stated otherwise.

The following description of at least one exemplary embodiment is merely illustrative in nature and is in no way intended to limit the disclosure, its application or uses in any way.

Techniques, methods, and apparatus known to those of ordinary skill in the relevant art may not be discussed in detail, but where appropriate, such techniques, methods, and apparatus should be considered part of the specification. .

In all examples shown and discussed herein, any specific values should be construed as illustrative only and not limiting. Accordingly, other instances of the exemplary embodiment may have different values.

It should be noted that like numerals and letters refer to like items in the following figures, so once an item is defined in one figure, it does not require further discussion in subsequent figures.

<Hardware configuration>

FIG. 1 is a block diagram of a hardware configuration of an AR device 1000 according to an embodiment of the present disclosure.

As shown in FIG. 1 , the AR device 1000 may be, for example, AR glasses, etc., which is not limited in this embodiment of the present disclosure. In one embodiment, as shown in FIG. 1 , the AR device 1000 may include a processor 1100 , a memory 1200 , an interface device 1300 , a communication device 1400 , a display device 1500 , an input device 1600 , a microphone 1700 and a speaker 1800 .

The processor 1100 may include, but is not limited to, a central processing unit CPU, a microprocessor MCU, and the like. The memory 1200 includes, for example, a ROM (Read Only Memory), a RAM (Random Access Memory), a nonvolatile memory such as a hard disk, and the like. The interface device 1300 includes, for example, various bus interfaces, such as a serial bus interface (including a USB interface), a parallel bus interface, and the like. The communication device 1400 is capable of, for example, wired or wireless communication. The display device 1500 is, for example, a liquid crystal display, an LED display, a touch display, or the like. The input device 1600 includes, for example, a touch screen or the like. The microphone 1700 may be used to input voice information. The speaker 1800 may be used to output voice information.

In this embodiment, the memory 1200 of the AR device 1000 is used to store instructions for controlling the processor 1100 to operate to implement or support the implementation of the target recognition method according to any embodiment. A skilled person can design instructions according to the solutions disclosed in this specification. How the instruction controls the processor to operate is well known in the art, so it will not be described in detail here.

Those skilled in the art should understand that although a plurality of devices of the AR device 1000 are shown in FIG. 1 , the AR device 1000 in this embodiment of the present disclosure may only involve some of the devices, or may also include other devices. Not limited.

<Method Example>

FIG. 2 shows a schematic flowchart of a target recognition method according to an embodiment of the present disclosure, and the target recognition method can be implemented by, for example, the AR device shown in FIG. 1 .

As shown in FIG. 2 , the target identification method provided by this embodiment may include the following steps 2100 to 2300 .

Step 2100: Obtain the ambient light intensity when the target recognition task is received.

In this step, the ambient light intensity may be acquired by an ambient light sensor provided in the AR device.

Step 2200: Obtain two-dimensional image data and depth image data of the target object when the ambient light intensity is less than a preset value.

It should be noted that the preset value is the corresponding value when the AR device is under a certain illumination intensity, and the recognition rate of the target object recognition based only on the two-dimensional image data is lower than the recognition rate based on the two-dimensional image data and the depth image data. Light intensity value.

In this step, after the ambient light intensity is acquired, the acquired ambient light intensity may be compared with a preset value. If the acquired ambient light intensity is less than a preset value, acquire the two-dimensional image data of the target object and the depth image data. In order to improve the target recognition rate in low light environment.

It can be understood that, because the measurement principle of the time-of-flight sensor is to emit infrared light of a specific range of wavelengths, after being reflected by the object for a period of time, it is received by the sensor, according to the time difference or phase between the light emission and the return to the sensor after being reflected by the object. difference, by converting to distance data, the image depth information of the object can be obtained. Therefore, in this embodiment, the depth image data may be acquired by a time-of-flight sensor provided in the AR device, and the two-dimensional image data may be acquired by an RGB camera provided in the AR device .

It can be understood that, after obtaining the ambient light intensity, if the obtained ambient light intensity is greater than the preset value, only the two-dimensional image data of the target object needs to be obtained without controlling the flight time. The sensor obtains the depth data of the target object, so that the device power consumption of the AR device can be reduced as much as possible while ensuring the recognition rate of the target object.

Step 2300: Obtain a recognition result of the target recognition task based on the two-dimensional image data and the depth image data.

Specifically, when the ambient light intensity is less than a preset value, it may be determined to use the first target recognition algorithm when performing the target recognition task; when the ambient light intensity is greater than the preset value, it may be determined A second target recognition algorithm is used when performing the target recognition task.

That is to say, when the recognition result of the target recognition task is obtained based on the two-dimensional image data and the depth image data, the two-dimensional image data and the depth image data may be processed specifically based on the first target recognition algorithm to obtain The recognition result of the target recognition task. When obtaining the recognition result of the target recognition task based on the image data, specifically, the two-dimensional image data may be processed based on the second target recognition algorithm to obtain the recognition result of the target recognition task.

It should be noted here that, as shown in FIG. 3 , both the first target recognition algorithm and the second target recognition algorithm are implemented based on the recognition model, and the construction process of the recognition model may refer to the following steps 3100 to 3100 3300:

Step 3100: Construct a training data set based on the mapping relationship between the input image data and the correct target recognition result.

Step 3200: Learning a large amount of training data in the training data set through a deep learning algorithm, and outputting a recognition result.

Step 3300: Compare the recognition result with the corresponding target recognition result. If there is an error between the recognition result and the target recognition result, adjust the parameters in the deep learning algorithm to reduce the target recognition error and improve the accuracy of the target recognition algorithm.

In this way, by repeating the above steps 3200 and 3300, the recognition model is obtained.

Usually, deep learning uses the convolutional neural network algorithm, which is a multi-layer hierarchical neural network, including convolutional layers, pooling layers, and fully connected layers. As shown in Figure 4, the function of the convolution layer is to extract the features contained in the input data, which can be used for image filtering to extract the local feature information of the image. The function of the pooling layer is to enhance the robustness to the deviation of the input signal. Usually, the maximum value of the output value of the convolution layer is used as the output value of the pooling layer, and finally the output result of the deep learning is obtained through the fully connected layer.

In this embodiment, the specific implementation process of the convolutional neural network includes: initialization of the image filter in the convolutional layer; the image filter in the convolutional layer calculates the weighted sum of pixels of each point for the input data, and convolves the weighted sum. , save the result to the output data to the pooling layer; the pooler in the pooling layer selects a point in the output data of the image filter in the convolutional layer, searches the neighborhood, and selects the maximum value as the output data To the fully connected layer; the fully connected layer of the fully connected layer uses the input data to obtain the output target result through the transfer function according to the weight and threshold.

According to this embodiment, the ambient light intensity can be obtained when the target recognition task is received; when the ambient light intensity is less than a preset value, the two-dimensional image data and the depth image data of the target object can be obtained; The two-dimensional image data and the depth image data are used to obtain the recognition result of the target recognition task. In this way, the target object can be identified based on the two-dimensional image data and the depth image data when the ambient light intensity is less than the preset value, so as to avoid the fact that the amount of information of the target object collected when the light intensity is low is relatively high. The target recognition rate is reduced due to less, and the target recognition rate is significantly improved.

<Apparatus Example>

An embodiment of the present disclosure provides a target recognition apparatus, which is set in an AR device. As shown in FIG. 5 , the target recognition apparatus 5000 may include: an acquisition module 5100 and an identification module 5200 .

Wherein, the acquisition module 5100 is configured to acquire the ambient light intensity when the target recognition task is received; when the ambient light intensity is less than a preset value, acquire two-dimensional image data and depth image data of the target object.

The recognition module 5200 is configured to obtain the recognition result of the target recognition task based on the two-dimensional image data and the depth image data.

In one embodiment, the obtaining module 5100 is further configured to obtain the two-dimensional image data of the target object when the ambient light intensity is greater than the preset value; the identifying module 5200 is further configured to: : obtain the recognition result of the target recognition task based on the two-dimensional image data.

In one embodiment, the recognition module 5200 is specifically configured to: process the two-dimensional image data and the depth image data based on the first target recognition algorithm to obtain the recognition result of the target recognition task.

In one embodiment, the recognition module 5200 is specifically configured to: process the two-dimensional image data based on the second target recognition algorithm to obtain the recognition result of the target recognition task.

The target identification device in this embodiment can be used to implement the technical solutions of the above method embodiments, and the implementation principles and technical effects thereof are similar, and are not repeated here.

<Apparatus Example>

FIG. 6 is a schematic diagram of a hardware structure of an AR device according to an embodiment. As shown in FIG. 6 , the AR device 6000 may include an ambient light sensor 6100, an RGB camera 6200 and a time-of-flight sensor 6300. The AR device 6000 may further include: a memory 6400 for storing executable computer instructions; a processor 6500 , for executing the target recognition method according to the above embodiment according to the control of the executable computer instructions; wherein, the ambient light sensor 6100, the RGB camera 6200 and the flight time sensor 6300 are all connected with The processor 6500 is connected.

In another embodiment, the AR device 6000 may include the above target recognition apparatus 5000 .

In one embodiment, each module of the above target identification apparatus 5000 may be implemented by the processor 6500 running computer instructions stored in the memory 6400 .

<Computer-readable storage medium>

The embodiments of the present disclosure also provide a computer-readable storage medium, which stores computer instructions, and when the computer instructions are run by a processor, executes the detection method of the head-mounted display device provided by the embodiments of the present disclosure.

The present disclosure may be a system, method and/or computer program product. The computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for causing a processor to implement various aspects of the present disclosure.

A computer-readable storage medium may be a tangible device that can hold and store instructions for use by the instruction execution device. The computer-readable storage medium may be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. More specific examples (non-exhaustive list) of computer readable storage media include: portable computer disks, hard disks, random access memory (RAM), read only memory (ROM), erasable programmable read only memory (EPROM) or flash memory), static random access memory (SRAM), portable compact disk read only memory (CD-ROM), digital versatile disk (DVD), memory sticks, floppy disks, mechanically coded devices, such as printers with instructions stored thereon Hole cards or raised structures in grooves, and any suitable combination of the above. Computer-readable storage media, as used herein, are not to be construed as transient signals per se, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (eg, light pulses through fiber optic cables), or through electrical wires transmitted electrical signals.

The computer readable program instructions described herein may be downloaded to various computing/processing devices from a computer readable storage medium, or to an external computer or external storage device over a network such as the Internet, a local area network, a wide area network, and/or a wireless network. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and/or edge servers. A network adapter card or network interface in each computing/processing device receives computer-readable program instructions from a network and forwards the computer-readable program instructions for storage in a computer-readable storage medium in each computing/processing device .

Computer program instructions for carrying out operations of the present disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or instructions in one or more programming languages. Source or object code, written in any combination, including object-oriented programming languages, such as Smalltalk, C++, etc., and conventional procedural programming languages, such as the "C" language or similar programming languages. The computer readable program instructions may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer, or entirely on the remote computer or server implement. In the case of a remote computer, the remote computer may be connected to the user's computer through any kind of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (eg, using an Internet service provider through the Internet connect). In some embodiments, custom electronic circuits, such as programmable logic circuits, field programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), can be personalized by utilizing state information of computer readable program instructions. Computer readable program instructions are executed to implement various aspects of the present disclosure.

Aspects of the present disclosure are described herein with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the disclosure. It will be understood that each block of the flowchart illustrations and/or block diagrams, and combinations of blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer readable program instructions.

These computer readable program instructions may be provided to a processor of a general purpose computer, special purpose computer or other programmable data processing apparatus to produce a machine that causes the instructions when executed by the processor of the computer or other programmable data processing apparatus , resulting in means for implementing the functions/acts specified in one or more blocks of the flowchart and/or block diagrams. These computer readable program instructions can also be stored in a computer readable storage medium, these instructions cause a computer, programmable data processing apparatus and/or other equipment to operate in a specific manner, so that the computer readable medium on which the instructions are stored includes An article of manufacture comprising instructions for implementing various aspects of the functions/acts specified in one or more blocks of the flowchart and/or block diagrams.

Computer readable program instructions can also be loaded onto a computer, other programmable data processing apparatus, or other equipment to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other equipment to produce a computer-implemented process , thereby causing instructions executing on a computer, other programmable data processing apparatus, or other device to implement the functions/acts specified in one or more blocks of the flowcharts and/or block diagrams.

The flowchart and block diagrams in the Figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of instructions, which comprises one or more functions for implementing the specified logical function(s) executable instructions. In some alternative implementations, the functions noted in the blocks may occur out of the order noted in the figures. For example, two blocks in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It is also noted that each block of the block diagrams and/or flowchart illustrations, and combinations of blocks in the block diagrams and/or flowchart illustrations, can be implemented in dedicated hardware-based systems that perform the specified functions or actions , or can be implemented in a combination of dedicated hardware and computer instructions. It is well known to those skilled in the art that implementation in hardware, implementation in software, and implementation in a combination of software and hardware are all equivalent.

Various embodiments of the present disclosure have been described above, and the foregoing descriptions are exemplary, not exhaustive, and not limiting of the disclosed embodiments. Numerous modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The terminology used herein was chosen to best explain the principles of the embodiments, the practical application or technical improvement in the marketplace, or to enable others of ordinary skill in the art to understand the embodiments disclosed herein. The scope of the present disclosure is defined by the appended claims.

Claims (10)

1. A target identification method is applied to an AR device, and comprises the following steps:

under the condition of receiving a target identification task, acquiring the ambient light intensity;

under the condition that the ambient illumination intensity is smaller than a preset value, acquiring two-dimensional image data and depth image data of a target object;

and obtaining the recognition result of the target recognition task based on the two-dimensional image data and the depth image data.

2. The method of claim 1, wherein after obtaining the ambient light intensity, the method further comprises:

under the condition that the ambient illumination intensity is larger than the preset value, acquiring two-dimensional image data of the target object;

and obtaining the recognition result of the target recognition task based on the two-dimensional image data.

3. The method of claim 1, wherein obtaining the recognition result of the target recognition task based on the two-dimensional image data and the depth image data comprises:

and processing the two-dimensional image data and the depth image data based on a first target recognition algorithm to obtain a recognition result of the target recognition task.

4. The method of claim 2, wherein deriving the recognition result of the object recognition task based on the two-dimensional image data comprises:

and processing the two-dimensional image data based on a second target identification algorithm to obtain an identification result of the target identification task.

5. An object recognition apparatus provided in an AR device, the apparatus comprising:

the acquisition module is used for acquiring the ambient light intensity under the condition of receiving the target identification task; under the condition that the ambient illumination intensity is smaller than a preset value, acquiring two-dimensional image data and depth image data of a target object;

and the identification module is used for obtaining an identification result of the target identification task based on the two-dimensional image data and the depth image data.

6. The apparatus of claim 5, wherein the obtaining module is further configured to:

under the condition that the ambient illumination intensity is larger than the preset value, acquiring two-dimensional image data of the target object;

the identification module is further configured to: and obtaining the recognition result of the target recognition task based on the two-dimensional image data.

7. The apparatus of claim 5, wherein the identification module is specifically configured to:

and processing the two-dimensional image data and the depth image data based on a first target recognition algorithm to obtain a recognition result of the target recognition task.

8. The apparatus of claim 6, wherein the identification module is specifically configured to:

and processing the two-dimensional image data based on a second target identification algorithm to obtain an identification result of the target identification task.

9. An AR device comprising an ambient light sensor, an RGB camera, and a time-of-flight sensor, the AR device further comprising:

a memory for storing executable computer instructions;

a processor for executing the object recognition method according to any one of claims 1 to 4 under the control of the executable computer instructions;

wherein the ambient light sensor, the RGB camera, and the time of flight sensor are all connected to the processor.

10. A readable storage medium, on which a program or instructions are stored, which when executed by a processor, carry out the steps of the object identification method according to any one of claims 1 to 4.

Images