CN119355954A

CN119355954A - Optical element and preparation method thereof, eye tracking device, and near-eye display device

Info

Publication number: CN119355954A
Application number: CN202411433323.XA
Authority: CN
Inventors: 兰富洋; 关健; 何强; 王兆民
Original assignee: Meta Bounds Inc
Current assignee: Meta Bounds Inc
Priority date: 2024-10-14
Filing date: 2024-10-14
Publication date: 2025-01-24

Abstract

The embodiments of the present invention provide an optical element and a method for preparing the same, an eye tracking device, and a near-eye display device. A reference structure is formed on the optical element, and the optical element is capable of transmitting visible light. At least part of the infrared light reflected by the eye is reflected to the camera via the optical element, so that the image captured by the camera includes an eye pattern, and at least part of the infrared light reflected by the eye is irradiated to the reference structure, so that the image captured by the camera also includes the reference pattern, so that the line of sight of the eye can be determined based on the eye pattern and the reference pattern. The technical solution of the embodiments of the present invention aims to improve the installation flexibility of the infrared light source and the camera, so as to facilitate the structural design of the near-eye display device.

Description

Optical element, preparation method thereof, eye movement tracking device and near-eye display equipment

Technical Field

The invention relates to the technical field of sight tracking, in particular to an optical element, a preparation method thereof, an eye movement tracking device and near-eye display equipment.

Background

The eye tracking apparatus generally irradiates a human eye with a light source, and the image acquisition unit photographs a spot and an iris image formed by the human eye, thereby calculating a line of sight direction of the human eye from the spot and the iris image. When the eye tracking device is applied to near-to-eye display equipment of augmented reality (AugmentedReality, abbreviated as AR) glasses, since the positions of the light source and the image acquisition unit relative to human eyes need to meet certain conditions, the light source and the image acquisition unit are usually arranged on a glasses frame or a lens, however, if the light source and the image acquisition unit are arranged on the glasses frame, the volume of the glasses frame is increased, the appearance design of the AR glasses is not facilitated when the wearing is affected, and if the light source or the image acquisition unit is arranged on the lens, the vision of human eyes is blocked, and the viewing experience of the AR glasses is affected.

Disclosure of Invention

The embodiment of the invention provides an optical element, a preparation method thereof, an eye movement tracking device and near-eye display equipment, and aims to improve the installation flexibility of an infrared light source and a camera so as to facilitate the structural design of the near-eye display equipment.

In a first aspect, embodiments of the present invention provide an optical element having a reference structure formed thereon;

The optical element can transmit visible light, infrared light reflected by eyes is at least partially reflected to the camera through the optical element, so that an image acquired by the camera comprises an eye pattern, and the infrared light reflected by the eyes is at least partially irradiated to the reference structure, so that the image acquired by the camera also comprises a reference pattern, and the sight direction of the eyes can be determined according to the eye pattern and the reference pattern.

Optionally, the optical element comprises a non-reference structure and the reference structure, the non-reference structure having a reflectivity greater than a reflectivity of the reference structure;

The eye-reflected infrared light is reflected at least partially to the camera via the non-reference structure to cause the camera to capture an eye pattern.

Optionally, the optical element comprises a reflective film layer and a functional layer, wherein the functional layer is arranged on one side of the reflective film layer and is provided with a shading area corresponding to the reference pattern, and a part of the reflective film layer corresponding to the shading area and the shading area form the reference structure, or

The optical element comprises a reflective film, and the reference structure is a hole formed in the reflective film.

Optionally, the optical element comprises a holographic reflective film, which is exposed through a mask to form the reference structure.

Optionally, the reference pattern comprises at least one of a grid, an array pattern and a two-dimensional code.

In a second aspect, an embodiment of the present invention provides a method for manufacturing an optical element, including the steps of:

A reflecting layer is arranged on one side of the holographic medium layer, a mask plate is arranged on the other side of the holographic medium layer, and a shading area corresponding to the reference pattern is arranged on the mask plate;

And arranging a laser emitter in a camera preset area on one side of the mask plate, which is away from the holographic medium layer, wherein the laser emitter emits infrared laser towards the holographic medium layer, and the infrared laser is reflected by the reflecting layer to expose the holographic medium layer, so that the holographic medium layer forms the reference structure, and the holographic reflecting film is obtained.

In a third aspect, an embodiment of the present invention further provides a method for manufacturing an optical element, including the steps of:

A reflection layer is arranged on one side of the holographic medium layer, a mask plate is arranged in an eye preset area on the other side of the holographic medium layer, and the mask plate is provided with a reflection area corresponding to the reference pattern;

And arranging a laser emitter in a camera preset area on the other side of the holographic medium layer, wherein the laser emitter emits infrared laser towards the holographic medium layer, the infrared laser is reflected to the mask plate through the reflecting layer, part of the infrared laser is reflected to the reflecting layer through the reflecting area and is reflected to the camera preset area again through the reflecting layer so as to expose the holographic medium layer, and the holographic medium layer forms the reference structure to obtain the holographic reflecting film.

In a fourth aspect, an embodiment of the present invention provides an eye tracking device, including:

The infrared light source is used for emitting infrared light;

An optical element as recited in the first aspect, the infrared light being reflected to the optical element through the eye, the optical element being capable of transmitting visible light;

A camera to which the infrared light reflected by the eyes is reflected at least partially via the optical element such that an image acquired by the camera includes an eye pattern, and the infrared light reflected by the eyes is irradiated at least partially to the reference structure such that the image acquired by the camera also includes a reference pattern, and

And the processing module is electrically connected with the camera and is used for determining the sight line direction of the eyes according to the eye pattern and the reference pattern.

Optionally, the processing module is configured to:

Determining coordinates of pupils and/or irises in the eye patterns in the reference patterns according to the images, and taking the coordinates as coordinates to be matched;

and matching the coordinates to be matched with a pre-stored coordinate database, and determining the sight line direction corresponding to the matched pre-stored coordinates as the current sight line direction of the eyes.

Optionally, the processing module matches the coordinate to be matched with a pre-stored coordinate database, and when determining the line of sight direction corresponding to the matched pre-stored coordinate as the current line of sight direction of the eye, the processing module is configured to:

When the absolute value of the difference value between the coordinate to be matched and one of the pre-stored coordinates is smaller than or equal to a preset threshold value, the coordinate to be matched is matched with the pre-stored coordinate, and the sight line direction corresponding to the pre-stored coordinate is determined as the current sight line direction of the eyes;

And when the absolute value of the difference value between the coordinate to be matched and any one of the pre-stored coordinates in the pre-stored coordinate database is larger than the preset threshold value, interpolating the sight line direction corresponding to the pre-stored coordinates in the pre-stored coordinate database, and determining the interpolation sight line direction matched with the coordinate to be matched as the current sight line direction of the eyes.

Optionally, the processing module is further configured to:

And determining coordinates of pupils and/or irises in the eye pattern in the reference pattern according to the image when the eyes are observed along the appointed sight direction, and storing the coordinates as the pre-stored coordinates into the pre-stored coordinate database.

Optionally, the eye tracking device includes the picture frame, locates the lens of picture frame and with wearing parts that the picture frame is connected, optical element locates the lens, the camera is located wearing parts, the infrared light source is located the picture frame or wearing parts.

In a fifth aspect, embodiments of the present invention provide a near-eye display device comprising an eye tracking apparatus as described in the fourth aspect.

The embodiment of the invention provides an optical element, a preparation method thereof, an eye movement tracking device and near-eye display equipment, wherein a reference structure is formed on the optical element, so that when the optical element is applied to the eye movement tracking device, infrared light emitted by an infrared light source is transmitted to eyes, after the infrared light reflected by the eyes is transmitted to the optical element, at least part of the infrared light can be reflected to a camera through the optical element, so that an image acquired by the camera comprises eye patterns, and at least part of the image can irradiate the reference structure of the optical element, and the image acquired by the camera also comprises the reference patterns, thereby determining the sight line direction of the eyes according to the eye patterns and the reference patterns, and realizing eye movement tracking. In the process of realizing eye tracking, the optical element can change the transmission path of infrared light, so that the camera can acquire images comprising eye patterns and reference patterns, the installation positions of the infrared light source and the camera are improved more flexibly, the optical element can transmit visible light, the infrared light source, the camera and the optical element can be prevented from shielding the sight of a user, and the structural design of the near-eye display device with the eye tracking function is facilitated.

Drawings

In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for the description of the embodiments will be briefly described below, and it is obvious that the drawings in the following description are some embodiments of the present application, and other drawings may be obtained according to these drawings without inventive effort for a person skilled in the art.

Fig. 1 is a schematic structural diagram of an optical element according to an embodiment of the present invention;

Fig. 2 is a schematic structural diagram of an eye tracking device according to an embodiment of the present invention;

FIG. 3 is a diagram showing an infrared light transmission path when the eye tracking device according to the embodiment of the present invention works;

FIG. 4 is a schematic diagram of another embodiment of an eye tracking device;

FIG. 5 is a diagram showing an infrared light transmission path when the eye tracker according to the embodiment of the present invention is in operation;

FIG. 6 is a schematic diagram of an image captured by a camera according to an embodiment of the present invention;

FIG. 7 is a schematic diagram of an exposure process of a holographic dielectric layer according to an embodiment of the present invention;

fig. 8 is a schematic structural diagram of a mask plate according to an embodiment of the present invention;

FIG. 9 is a schematic diagram illustrating an exposure process of another holographic dielectric layer according to an embodiment of the present invention;

fig. 10 is a schematic diagram of a virtual image and an image acquired by a camera according to an embodiment of the present invention.

Detailed Description

The following description of the embodiments of the present invention will be made clearly and fully with reference to the accompanying drawings, in which it is evident that the embodiments described are some, but not all embodiments of the invention. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention.

The flow diagrams depicted in the figures are merely illustrative and not necessarily all of the elements and operations/steps are included or performed in the order described. For example, some operations/steps may be further divided, combined, or partially combined, so that the order of actual execution may be changed according to actual situations.

It is to be understood that the terminology used in the description of the invention herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. As used in this specification and the appended claims, the singular forms "a," "an," and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise.

Some embodiments of the invention are described in detail below with reference to the accompanying drawings. The following embodiments and features of the embodiments may be combined with each other without conflict.

Referring to fig. 1 to 3, an optical element 10 is provided, and a reference structure 10' is formed on the optical element 10. The optical element 10 is capable of transmitting visible light, infrared light reflected by the eye 100 is at least partially reflected to the camera 30 via the optical element 10, such that an image acquired by the camera 30 comprises an eye pattern, and infrared light reflected by the eye 100 is at least partially illuminated to the reference structure 10', such that an image acquired by the camera 30 further comprises a reference pattern, such that a line of sight direction of the eye 100 can be determined from the eye pattern and the reference pattern.

It will be appreciated that when the optical element 10 is applied to the eye tracking device, the eye tracking device includes the infrared light source 20 and the camera 30, the infrared light emitted from the infrared light source 20 is reflected to the optical element 10 via the eye 100, at least part of the infrared light reflected to the optical element 10 is reflected to the camera 30 via the optical element 10, so that an image obtained by collecting infrared light by the camera 30 includes an eye pattern, and at least part of the infrared light reflected to the optical element 10 is transmitted to the reference structure 10', the reference structure 10' can reflect, diffract or absorb the infrared light, so that an image obtained by collecting infrared light by the camera 30 includes a reference pattern, that is, the optical element 10 can change a transmission path of the infrared light reflected by the eye 100, so that the camera 30 can collect an image including the eye pattern and the reference pattern, thereby enabling the eye tracking device to determine a line of sight direction of the eye 100 according to the eye pattern and the reference pattern, and eye tracking is achieved. Since the optical element 10 can change the transmission path of the infrared light and can transmit the visible light, the installation positions of the infrared light source 20 and the camera 30 are more flexible under the condition that the optical element 10, the infrared light source 20 and the camera 30 are prevented from shielding the sight of a user, so that the structural design of the near-eye display device with the eye movement tracking function is facilitated.

Referring to fig. 2 to 5, an eye tracking device according to an embodiment of the invention includes an infrared light source 20, an optical element 10, a camera 30 and a processing module (not shown). The infrared light source 20 is used for emitting infrared light, the optical element 10 is the optical element described above, the infrared light is reflected to the optical element 10 through the eye 100, and the optical element 10 can transmit visible light. The infrared light reflected by the eye 100 is at least partially reflected to the camera 30 via the optical element 10 such that the image acquired by the camera 30 comprises an eye pattern, and the infrared light reflected by the eye 100 is at least partially illuminated to the reference structure 10' such that the image acquired by the camera 30 further comprises a reference pattern. The processing module is electrically connected to the camera 30, and the processing module is used for determining the sight line direction of the eyes 100 according to the eye pattern and the reference pattern.

In some embodiments, the eye tracking device includes a frame 40, a lens 60 disposed on the frame 40, and a wearing part connected to the frame 40, the optical element 10 is disposed on the lens 60, the camera 30 is disposed on the wearing part, and the infrared light source 20 is disposed on the frame 40 or the wearing part. It will be appreciated that the near-eye display device is generally provided with a lens frame 40, a lens 60 and a wearing part, when a user wears the near-eye display device, the lens 60 of the near-eye display device is generally positioned in front of the eyes 100, the optical element 10 can reflect infrared light through visible light, the optical element 10 can be stably arranged on the lens 60 without affecting the line of sight of the user, and can reflect infrared light reflected by the eyes 100 of the user, while the infrared light source 20 can be arranged on the lens frame 40 or the wearing part, the camera 30 can be arranged on the wearing part, the infrared light source 20 and the camera 30 are prevented from shielding the line of sight of the user, and the size of the lens frame 40 is controlled, and the influence on the aesthetic degree, wearing stability and wearing comfort of the near-eye display device due to the overlarge size of the lens frame 40 is avoided.

For example, the infrared light emitted from the infrared light source 20 can be directly transmitted to the eyes 100 of the user, or the infrared light emitted from the infrared light source 20 can be transmitted to the optical element 10 and then reflected to the eyes 100 of the user through the optical element 10.

For example, the eye tracking device is applied to AR glasses/VR glasses including a frame 40, a temple 50, a lens 60, and a nose pad 70. Wherein, the two legs 50 are respectively connected to two opposite sides of the frame 40, the two lenses 60 are disposed between the two legs 50 and spaced apart from each other on the frame 40, and the nose pad 70 is disposed between the two lenses 60 and the frame 40.

An exemplary arrangement of an eye tracking device in AR/VR glasses is shown in fig. 2. As shown in fig. 2, an infrared light source 20 may be provided on the nose piece 70 to be disposed toward the eyes 100 of the user. The optical element 10 is disposed on the lens 60, for example, on the inside of the lens 60 near the user's eye 100, or on the outside of the lens 60 away from the user's eye 100, or on the inside of the lens 60. The camera 30 is provided inside the temple 50 and toward the optical element 10.

Fig. 3 shows a transmission path of infrared light when the eye tracking device in fig. 2 is operated, and as shown in fig. 3, when the eye tracking device is operated, the infrared light source 20 emits infrared light toward the eyes 100 of the user, the infrared light is reflected by the eyes 100 of the user and then transmitted to the optical element 10, the optical element 10 reflects the infrared light to the camera 30 again, and the camera 30 collects and images the infrared light to obtain an image including an eye pattern and a reference pattern.

Another exemplary arrangement of the eye tracking device in AR/VR glasses is shown in fig. 4, in which the optical element 10 is disposed on the lens 60, and the infrared light source 20 and the camera 30 are disposed on the temple 50 and toward the optical element 10, as shown in fig. 4, so that the space of the nose pad 70 is advantageously saved, and the wiring from the temple 50 to the nose pad 70 is omitted, thereby facilitating the wiring of the infrared light source 20 and the camera 30.

Alternatively, the infrared light source 20 and the camera 30 may be disposed on the temple 50 at intervals or closely disposed on the temple 50. Preferably, the infrared light source 20 and the camera 30 are closely positioned to improve the accuracy of eye tracking and to achieve a more compact appearance.

Fig. 5 shows a transmission path of infrared light when the eye tracking device in fig. 4 is operated, and as shown in fig. 5, infrared light emitted from the infrared light source 20 is transmitted toward the optical element 10, reflected by the optical element 10 and transmitted to the eyes 100 of the user, the eyes 100 of the user are illuminated, then the infrared light is reflected by the eyes 100 of the user to the optical element 10, and reflected again by the optical element 10 and transmitted to the camera 30, so that the camera 30 can collect and image the infrared light, and an image including an eye pattern and a reference pattern is obtained.

Alternatively, the infrared light source 20 emits infrared light having a wavelength of 800nm to 1000nm, preferably with an infrared light having a wavelength of 850nm or 940 nm.

Illustratively, the infrared light source 20 may employ a point light source, and the infrared light source 20 occupies a small space.

For example, camera 30 may employ an infrared camera 30 that is matched to infrared light source 20, and camera 30 may respond and image only infrared light of a particular wavelength emitted by infrared light source 20.

It should be noted that the shape and size of the optical element 10 may be adjusted according to practical situations, for example, the shape and size of the lens 60. Alternatively, the optical element 10 may be square, circular, elliptical, or the like, and is not particularly limited herein.

Illustratively, the central region of the optical element 10 is located on the optical axis of the camera 30 to ensure the integrity of the image captured by the camera 30.

It will be appreciated that the reference pattern is formed in dependence on the optical element 10, and that the relative position between the various components of the eye tracker is unchanged when the user wears the near-eye display device, and the relative position between the eye tracker and the user's eye 100 is also unchanged, so that the position of the reference pattern in the image acquired by the camera 30 is also relatively fixed, and the relative positions of the pupil, iris and reference pattern in the eye pattern acquired by the camera 30 are different when the user's eye 100 rotates to observe in different directions of view, and thus the direction of view of the user can be determined from the relative positional relationship between the pupil, iris and reference pattern in the eye pattern.

In some embodiments, the reference pattern comprises at least one of a grid, an array pattern, and a two-dimensional code. It can be appreciated that the reference pattern and the eye pattern are superimposed, and when the reference pattern adopts a grid, an array pattern or a two-dimensional code, it can be ensured that the eye pattern can be accurately identified.

Further, each unit in the grid, the array pattern or the two-dimensional code is distributed in the longitudinal and transverse directions, so that the coordinates of the eye pattern in the reference pattern can be determined, and the accuracy of eye movement tracking can be ensured.

Illustratively, the array pattern may be a lattice as shown in fig. 6.

For example, as shown in fig. 6 (a), an image collected by the camera 30 when the eye 100 of the user is observed upward may be shown in fig. 6 (B), an image collected by the camera 30 when the eye 100 of the user is observed downward and leftward may be shown in fig. 6 (a) and (B), and it may be seen that the positions of pupils in the eye pattern in the reference pattern are different in fig. 6 (a) and (B), and the direction of the line of sight of the eye 100 of the user may be determined according to the positions of pupils in the eye pattern in the reference pattern.

In some embodiments, the optical element 10 includes a non-reference structure 10 "and a reference structure 10', the non-reference structure 10" having a reflectivity that is greater than the reflectivity of the reference structure 10'. Infrared light reflected by the eye 100 is at least partially reflected to the camera 30 via the non-reference structure 10 "such that the camera 30 captures an eye pattern. It will be appreciated that the reflectivity of the non-reference structure 10 "and the reference structure 10 'is for infrared light, the reflectivity of the non-reference structure 10" is R1, then 0< R1, the reflectivity of the reference structure 10' is R2, then 0.ltoreq.R2, and R2< R1. The infrared light reflected by the eye portion 100 is transmitted to the optical element 10, the rear portion is reflected to the camera 30 by the non-reference structure 10", the other portion irradiates to the reference structure 10' without reflection or with less reflection, so that the color difference exists in the image obtained by the camera 30 collecting the infrared light, and the areas of the eye pattern and the reference pattern can be divided according to the color difference, so that the sight line direction of the eye portion 100 can be determined according to the eye pattern and the reference pattern.

Preferably, the reflectivity of the non-reference structure 10 "and the reference structure 10' satisfy the relationship R2<0.5R1. When the above relation is satisfied, it is advantageous for the processing module to divide the area of the reference pattern more precisely after the image acquired by the camera 30 is acquired, so as to improve the accuracy of determining the direction of the eye 100 according to the eye pattern and the reference pattern.

In an alternative embodiment, the optical element 10 includes a reflective film layer and a functional layer, where the functional layer is disposed on one side of the reflective film layer and is provided with a light shielding region corresponding to the reference pattern, and a portion of the reflective film layer corresponding to the light shielding region and the light shielding region form a reference structure. It can be appreciated that the functional layer is disposed on one side of the reflective film layer, and the functional layer has a light shielding region corresponding to the reference pattern, and the light shielding region is used to block part or all of the infrared light, so as to reduce the reflectivity of the reference structure 10'.

Illustratively, the reflective film layer may be a dichroic mirror film layer or a holographic reflective film layer.

Illustratively, the functional layer may be formed using a crystalline material such as calcium fluoride, a liquid crystal material, or a film such as silicon dioxide.

For example, the light shielding area may be in a grid shape, an array shape or a two-dimensional code shape, and the reference pattern in the image collected by the camera 30 may be a corresponding grid, array pattern or two-dimensional code.

In another alternative embodiment, as shown in fig. 1, the optical element 10 includes a reflective film, and the reference structure 10' is a hole formed in the reflective film corresponding to the reference pattern. It can be understood that both visible light and infrared light can pass through the hole, and since infrared light cannot be reflected at the hole, part of infrared light cannot be reflected to the camera 30, so that the image collected by the camera 30 has a reference pattern corresponding to the hole.

The reflective film may be a dichroic mirror film or a holographic reflective film layer, for example.

For example, the holes may be grid-shaped, array-shaped or two-dimensional code-shaped, and the reference pattern in the image collected by the camera 30 may be a corresponding grid, array pattern or two-dimensional code.

In some embodiments, the optical element 10 includes a holographic reflective film that is exposed through a mask to form the reference structure 10'. It will be appreciated that the holographic medium layer 11 may be subjected to mask exposure by using laser light in the infrared band to obtain a holographic reflection film, so that the holographic reflection film records an interference hologram, and when infrared light reflected by the eye portion 100 irradiates the holographic reflection film, the holographic reflection film at least partially reflects the infrared light to the camera 30, so that an image collected by the camera 30 includes an eye pattern, and at least partially irradiates the reference structure, and then is diffracted and transmitted to the camera, so that an image collected by the camera 30 includes a reproduced reference pattern. In other words, the holographic reflection film can reflect infrared light and form a reference pattern on the camera 30, which is beneficial to controlling the volume of the optical element 10, and the holographic reflection film is convenient to prepare in a mask exposure mode, and the prepared holographic reflection film has small influence on visible light, which is beneficial to ensuring the display effect of the near-to-eye display device.

As shown in fig. 6 and 7, in an alternative embodiment, the preparation method of the holographic reflective film includes the steps of disposing a reflective layer 12 on one side of a holographic medium layer 11, disposing a mask plate 13 on the other side, disposing a light shielding region 131 corresponding to a reference pattern on the mask plate 13, disposing a laser emitter (not shown) on a camera preset region 10a on one side of the mask plate 13 facing away from the holographic medium layer 11, emitting infrared laser toward the holographic medium layer 11, and reflecting the infrared laser through the reflective layer 12 to expose the holographic medium layer 11, so that the holographic medium layer 11 forms a reference structure 10' to obtain the holographic reflective film.

It will be appreciated that the optical element 10 comprises a holographic reflective film which is exposed from the holographic dielectric layer 11. The mask plate 13 disposed at one side of the holographic medium layer 11 has a light shielding region 131 corresponding to the reference pattern, the infrared laser is blocked from being transmitted to the holographic medium layer 11, and other regions except the light shielding region 131 are light transmitting regions, so that the infrared laser can transmit and transmit to the holographic medium layer 11. When the laser emitter in the camera preset area 10a emits infrared laser to the holographic medium layer 11, a part of the infrared laser is blocked by the light shielding area 131, so that the position of the holographic medium layer 11 corresponding to the light shielding area 131 cannot record holograms and has no reflection function, the other part of the infrared laser can be transmitted to the holographic medium layer 11 and the reflection layer 12 through the light transmission area of the mask plate 13 and can be reflected by the reflection layer 12, and the infrared laser irradiated to the holographic medium layer 11 by the laser emitter and the infrared laser reflected by the reflection layer 12 interfere with each other in the holographic medium layer 11, so that the holographic medium layer 11 records holograms at positions corresponding to positions outside the light shielding area 131 and has reflection functions, exposure of the holographic medium layer 11 is realized, and the holographic reflection film is obtained. When the holographic reflective film obtained by exposing the holographic medium layer 11, the camera 30 and the infrared light source 20 perform eye tracking, the infrared light cannot be reflected to the camera 30 because the holographic reflective film portion has no reflection function, so that the image collected by the camera 30 includes the reference pattern.

In the camera preset area 10a, that is, the preset eye tracking device, the camera 30 is positioned relative to the holographic reflective film. Specifically, the laser emitter emits infrared laser from the light entrance pupil position of the camera 30, so that the infrared light reflected by the prepared holographic reflection film is collected by the camera 30 more, and the integrity of the image collected by the camera 30 is ensured.

Illustratively, the infrared laser emitted from the laser emitter is reflected by the reflecting layer 12 and then passes through the eye preset area 10b, that is, the position of the eye 100 relative to the holographic reflecting film when the eye tracking device is operated.

Illustratively, the light shielding region 131 may be in a grid shape, a two-dimensional code shape, or an array shape as shown in fig. 8.

Exemplary, as shown in fig. 7, the reflective layer 12 is attached to one side of the holographic medium layer 11, and the mask plate is attached to the other side of the holographic medium layer 11, so as to ensure that the laser can expose the holographic medium layer 11 more precisely, and improve the accuracy of the hologram that can be formed by the holographic medium layer 11, thereby improving the accuracy of eye tracking of the eye tracking device.

Alternatively, the holographic reflection film may be prepared first, then the holographic reflection film is disposed on the lens 60, or the holographic medium layer 11 may be directly disposed on the lens 60, and then the holographic medium layer 11 is exposed by using the reflection layer 12, the mask plate 13 and the laser emitter to prepare the holographic reflection film.

As shown in FIG. 8, in another alternative embodiment, the preparation method of the holographic reflective film comprises the steps of arranging a reflective layer 12 on one side of a holographic medium layer 11, arranging a mask plate 13 on an eye preset area 10b on the other side of the holographic medium layer 11, arranging a laser emitter on a camera preset area 10a on the other side of the holographic medium layer 11, emitting infrared laser towards the holographic medium layer 11, reflecting the infrared laser to the mask plate 13 through the reflective layer 12, reflecting part of the infrared laser to the reflective layer 12 through the reflective area, and reflecting the infrared laser to the camera preset area 10a again through the reflective layer 12 to expose the holographic medium layer 11, so that the holographic medium layer 11 forms a reference structure 10', and obtaining the holographic reflective film.

It will be appreciated that the optical element 10 comprises a holographic reflective film which is exposed from the holographic dielectric layer 11. The mask plate 13 disposed in the eye preset region 10b has a reflection region corresponding to the reference pattern, the reflection region being capable of reflecting the infrared laser light, and regions other than the reflection region reflecting less or no infrared laser light. The laser emitter in the camera preset area 10a emits infrared laser to the holographic medium layer 11, the infrared laser passes through the holographic medium layer 11 and then is transmitted to the reflecting layer 12 and is reflected by the reflecting layer 12 to the mask 13, only part of infrared light is reflected to the holographic medium layer 11 by the reflecting area and forms interference with the infrared laser irradiated to the holographic medium layer 11 by the laser emitter and the infrared laser reflected by the reflecting layer 12 in the holographic medium layer 11, so that the holographic medium layer 11 records holograms related to a reference pattern and has a reflecting function, exposure of the holographic medium layer 11 is realized, and a holographic reflecting film is obtained. When the holographic reflection film obtained by exposing the holographic medium layer 11, the camera 30 and the infrared light source 20 perform eye tracking, the holographic reflection film can reproduce holograms under the irradiation of infrared light, so that the pattern collected by the camera 30 includes a reference pattern, and at the same time, the infrared light reflected by the eye 100 is collected by the camera 30, so that the collected pattern also includes an eye pattern. And it is worth to say that, when the holographic reflective film obtained by the preparation method is used for eye tracking, when the camera 30 collects images, the virtual image corresponding to the eye pattern and the virtual image corresponding to the reference pattern can be located at the same or similar object distance relative to the camera 30, and the camera 30 can focus the eye pattern and the reference pattern clearly at the same time, so that the collected eye pattern and the collected reference pattern are clear, the sight line direction of the eye 100 is determined according to the eye pattern and the reference pattern, and the accuracy of eye tracking is improved.

For example, the reflective region of the mask plate may be a highly reflective/scattering region capable of reflecting/scattering laser light in the infrared band, while the other regions outside the reflective region are low reflective/scattering regions, reflecting/scattering less or not reflecting/scattering laser light in the infrared band.

For example, the reflective area of the mask 13 may be in a grid shape, an array shape, or a two-dimensional code shape.

In some embodiments, the processing module is configured to determine coordinates of a pupil and/or an iris in the eye pattern in the reference pattern according to the image, use the coordinates as coordinates to be matched, match the coordinates to be matched with a pre-stored coordinate database, and determine a line-of-sight direction corresponding to the pre-stored coordinates that are matched as the line-of-sight direction of the current eye 100.

It may be appreciated that, as described above, when the eye 100 of the user is observed along different directions of sight, the positions of the pupils and/or the irises in the eye pattern are different in the reference pattern, the processing module may be provided with a pre-stored coordinate database, where the pre-stored coordinate database includes a plurality of pre-stored coordinates, the pre-stored coordinates represent the coordinates of the pupils and/or the irises in the eye pattern in the reference pattern, and any pre-stored coordinate has a corresponding direction of sight, when the eye tracking device performs eye tracking, the coordinates of the pupils and/or the irises in the current eye pattern in the reference pattern may be determined according to the image acquired by the camera 30, at this time, the coordinates may be determined as the coordinates to be matched, and the coordinates to be matched may be matched with a plurality of pre-stored coordinates in the pre-stored coordinate database, if the matching is successful, the direction of sight corresponding to the matched coordinates may be used as the direction of sight of the current eye 100, so that the near-eye display device may be controlled accordingly.

Illustratively, coordinates of the pupil center in the eye pattern in the reference pattern are determined from the image to improve accuracy of eye movement tracking.

Illustratively, the absolute value of the difference between the pre-stored coordinates that match the coordinates to be matched and the coordinates to be matched is less than the absolute value of the difference between the other pre-stored coordinates and the coordinates to be matched.

Further, the processing module is used for matching the coordinate to be matched with the pre-stored coordinate database, determining the sight line direction corresponding to the matched pre-stored coordinate as the sight line direction of the current eye 100, when the absolute value of the difference value between the coordinate to be matched and a pre-stored coordinate is smaller than or equal to a preset threshold value, matching the coordinate to be matched with the pre-stored coordinate, determining the sight line direction corresponding to the pre-stored coordinate as the sight line direction of the current eye 100, and when the absolute value of the difference value between the coordinate to be matched and any pre-stored coordinate in the pre-stored coordinate database is larger than the preset threshold value, performing interpolation processing on the sight line direction corresponding to the pre-stored coordinate in the pre-stored coordinate database, and determining the interpolation sight line direction matched with the coordinate to be matched as the sight line direction of the current eye 100. It can be understood that when the absolute value of the difference between the coordinate to be matched and any pre-stored coordinate is greater than the preset threshold, that is, the distance between the coordinate to be matched and any pre-stored coordinate is greater, interpolation processing is performed on the line-of-sight direction corresponding to the pre-stored coordinate to obtain an interpolated line-of-sight direction corresponding to the coordinate closer to the current coordinate to be matched, and the interpolated line-of-sight direction is determined as the line-of-sight direction of the current eye 100, so that the accuracy of eye tracking is improved.

In some embodiments, the processing module is further configured to determine coordinates of pupils and/or irises in the eye pattern in the reference pattern from the image of the eye 100 when viewed along the specified line of sight direction, and store the coordinates as pre-stored coordinates to a pre-stored coordinates database. It can be appreciated that, when the eye tracking device is initially used, the eye 100 of the user can be guided to observe along the specified line of sight direction, the camera 30 is utilized to collect the image, the processing module determines the coordinates of the pupil and/or the iris of the eye pattern in the reference pattern according to the image after the image collected by the camera 30 is obtained, the determined coordinates are pre-stored coordinates, the pre-stored coordinates are stored in the pre-stored coordinates database, and the pre-stored coordinates correspond to the specified line of sight direction, so that the calibration of the line of sight direction is realized. When the eye tracking device is used for eye tracking subsequently, coordinates of pupils and/or irises in the eye pattern in the reference pattern can be determined according to the acquired image again when the eyes 100 of the user are observed along the unknown sight line direction, the determined coordinates are coordinates to be matched at the moment, and the sight line direction corresponding to the pre-stored coordinates matched with the coordinates to be matched is determined as the sight line direction of the current eyes 100 by matching the coordinates to be matched with the pre-stored coordinates.

For example, taking an example that the eye tracking device is applied to the near-eye display device, when the user initially uses the near-eye display device, the near-eye display device may display a virtual image, and guide the user's eye 100 to observe a specified position of the virtual image, so as to guide the user's eye 100 to observe along a specified line of sight direction, thereby calibrating the line direction. For example, the virtual image may be a lattice, a square array, a five-pointed star array, or the like. Taking the virtual image as a dot matrix for example, the color/size of a designated point in the dot matrix can be distinguished from the color/size of other points, or the designated point is enabled to flash, and the other points are continuously displayed, so that the designated point is distinguished from the other points, and the user is guided to look at the designated point, so that the user can observe along the designated line-of-sight direction. As shown in fig. 9, the direction of the line of sight when the eye 100 observes a specified point of the virtual image, and the image including the eye pattern and the reference pattern acquired by the camera are shown in fig. 9. When the eye 100 observes the specified point of the virtual image, the eye 100 observes along the specified line of sight direction, and the infrared light source 20 emits infrared light at the same time, so that the camera 30 can collect the image including the eye pattern and the reference pattern, and thus, when the user observes along the specified line of sight direction, the coordinates of the pupil and/or the iris of the eye pattern in the image in the reference pattern can be determined according to the image, and the coordinates are stored as pre-stored coordinates into the pre-stored coordinate database, and after the user is guided to sequentially look at all the specified points in the lattice, all the pre-stored coordinates are obtained, so as to complete the calibration of the line of sight direction.

The camera 30 may continuously collect multiple frames of images to obtain an image with the most stable position of the eye 100 of the user, and perform calibration of the specified line of sight direction by using the image, so as to improve the calibration accuracy and optimize the accuracy of eye tracking by the eye tracking device.

The embodiment of the invention also provides near-eye display equipment, which comprises the eye movement tracking device. It can be appreciated that the near-eye display device includes the eye tracking device as described above, which also has all the technical effects of avoiding blocking the user's line of sight, making the mounting positions of the infrared light source and the camera more flexible, and being beneficial to adjusting the mounting positions of the infrared light source and the camera according to the needs, so as to control the volume of the near-eye display device.

The embodiment of the invention also provides an eye movement tracking method of the eye movement tracking device, the eye movement tracking device comprises an infrared light source, an optical element and a camera, the infrared light source is used for emitting infrared light, the infrared light is reflected to the optical element through eyes, the optical element can transmit visible light, and the optical element is used for reflecting the infrared light so that the camera can acquire images comprising eye patterns and reference patterns. The eye tracking method includes steps S10 to S30.

S10, acquiring an image acquired by a camera.

S20, determining coordinates of pupils and/or irises in the eye pattern in the reference pattern according to the image, and taking the coordinates as coordinates to be matched.

And S30, matching the coordinates to be matched with a pre-stored coordinate database, and determining the sight line direction corresponding to the matched pre-stored coordinates as the sight line direction of the current eyes.

The infrared light reflected by the eyes is reflected to the camera again through the optical element, so that the camera can acquire an image comprising an eye pattern and a reference pattern, and the eye sight direction is determined according to the coordinates of the pupil and/or the iris of the eye pattern in the reference pattern in the image.

The method of the present application is operational with numerous general purpose or special purpose computer system environments or configurations. Such as a personal computer, a server computer, a hand-held or portable device, a tablet device, a multiprocessor system, a microprocessor-based system, a set top box, a programmable consumer electronics, a network PC, a minicomputer, a mainframe computer, a distributed computing environment that includes any of the above systems or devices, and the like. The application may be described in the general context of computer-executable instructions, such as program modules, being executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform particular tasks or implement particular abstract data types. The application may also be practiced in distributed computing environments where tasks are performed by remote processing devices that are linked through a communications network. In a distributed computing environment, program modules may be located in both local and remote computer storage media including memory storage devices.

Embodiments of the present application also provide a computer readable storage medium, on which a computer program is stored, where a method implemented when the computer program is executed by a processor may refer to various embodiments of an eye tracking method of a near-eye display device of the present application.

The computer readable storage medium may be an internal storage unit of the near-eye display device according to the foregoing embodiment, for example, a hard disk or a memory of the near-eye display device. The computer readable storage medium may also be an external storage device of the near-eye display device, such as a plug-in hard disk, a smart memory card (SMARTMEDIACARD, SMC), a secure digital (SecureDigital, SD) card, a flash memory card (FLASHCARD), etc. that are provided on the near-eye display device.

Those of ordinary skill in the art will appreciate that all or some of the steps, systems, functional modules/units in the apparatus, and methods disclosed above may be implemented as software, firmware, hardware, and suitable combinations thereof. In a hardware embodiment, the division between functional modules/units mentioned in the above description does not necessarily correspond to the division of physical components, for example, one physical component may have a plurality of functions, or one function or step may be cooperatively performed by several physical components. Some or all of the physical components may be implemented as software executed by a processor, such as a central processing unit, digital signal processor, or microprocessor, or as hardware, or as an integrated circuit, such as an application specific integrated circuit. Such software may be distributed on computer readable media, which may include computer storage media (or non-transitory media) and communication media (or transitory media). The term computer storage media includes both volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data, as known to those skilled in the art. Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital Versatile Disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium which can be used to store the desired information and which can be accessed by a computer. Furthermore, as is well known to those of ordinary skill in the art, communication media typically embodies computer readable instructions, data structures, program modules or other data in a modulated data signal such as a carrier wave or other transport mechanism and includes any information delivery media.

It should be understood that the term "and/or" as used in the present specification and the appended claims refers to any and all possible combinations of one or more of the associated listed items, and includes such combinations. It should be noted that, in this document, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or system that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or system. Without further limitation, an element defined by the phrase "comprising one does not exclude the presence of other like elements in a process, method, article, or system that comprises the element.

The foregoing embodiment numbers of the present invention are merely for the purpose of description, and do not represent the advantages or disadvantages of the embodiments. While the invention has been described with reference to certain preferred embodiments, it will be understood by those skilled in the art that various changes and substitutions may be made therein without departing from the spirit and scope of the invention as defined by the appended claims. Therefore, the protection scope of the invention is subject to the protection scope of the claims.

Claims

1. An optical element, wherein a reference structure is formed on the optical element;

2. The optical element of claim 1, wherein the optical element comprises a non-reference structure and the reference structure, the non-reference structure having a reflectivity that is greater than a reflectivity of the reference structure;

The eye-reflected infrared light is reflected at least partially to the camera via the non-reference structure to cause the camera to capture the eye pattern.

3. The optical element according to claim 2, wherein the optical element comprises a reflective film layer and a functional layer, the functional layer being provided on one side of the reflective film layer and provided with a light shielding region corresponding to the reference pattern, a portion of the reflective film layer corresponding to the light shielding region and the light shielding region forming the reference structure, or

4. The optical element of claim 1, wherein the optical element comprises a holographic reflective film that is exposed through a mask to form the reference structure.

5. The optical element of any one of claims 1-4, wherein the reference pattern comprises at least one of a grid, an array pattern, a two-dimensional code.

6. A method of producing the holographic reflective film of claim 4, comprising the steps of:

7. A method of producing the holographic reflective film of claim 4, comprising the steps of:

8. An eye tracking device, comprising:

The infrared light source is used for emitting infrared light;

An optical element as claimed in any one of claims 1 to 5, to which the infrared light is reflected through the eye, the optical element being capable of transmitting visible light;

9. The eye tracking device according to claim 8, wherein the processing module is configured to:

10. The eye tracking device according to claim 9, wherein the processing module is configured to match the coordinate to be matched with a pre-stored coordinate database, and determine, when a line-of-sight direction corresponding to the matched pre-stored coordinate is a current line-of-sight direction of the eye, to:

11. The eye tracking device according to claim 9, wherein the processing module is further configured to:

12. The eye tracking device according to claim 8, wherein the eye tracking device comprises a lens frame, a lens provided in the lens frame, and a wearing member connected to the lens frame, the optical element is provided in the lens, the camera is provided in the wearing member, and the infrared light source is provided in the lens frame or the wearing member.

13. A near-eye display device comprising an eye tracking apparatus according to any one of claims 8-12.