CN120526469A

CN120526469A - Eye movement tracking dynamic target identification method and device based on three-dimensional fixation point

Info

Publication number: CN120526469A
Application number: CN202510500113.6A
Authority: CN
Inventors: 金华; 仇跃荣; 黄琪辉; 宋雪桦; 王昌达
Original assignee: Jiangsu University
Current assignee: Jiangsu University
Priority date: 2025-04-21
Filing date: 2025-04-21
Publication date: 2025-08-22

Abstract

The invention belongs to the field of computer vision and deep learning, and particularly relates to an eye movement tracking dynamic target identification method and device based on a three-dimensional gaze point. Firstly, screening out a three-dimensional gazing point in a gazing state according to three-dimensional gazing point data, mapping the three-dimensional gazing point into a two-dimensional gazing point in a foreground camera coordinate system, identifying a gazing target in a foreground picture of a user according to the mapped two-dimensional gazing point, and informing the user of the identification result through a result broadcasting device. And then tracking the target to ensure that the user's sight does not repeat the recognition operation when looking at the target, and starting the next recognition if the user's sight deviates from the target for a long time, thereby ensuring that the user's looking at the target can be correctly recognized. The invention effectively reduces the loss phenomenon caused by deformation or high-speed movement of the target, does not need to repeatedly carry out repeated uploading identification on the same target, reduces the communication quantity between the target and the server, and reduces the pressure of the server.

Description

Eye movement tracking dynamic target identification method and device based on three-dimensional fixation point

Technical Field

The invention belongs to the field of computer vision and deep learning, and particularly relates to an eye movement tracking dynamic target identification method and device based on a three-dimensional gaze point.

Background method

Most of the current three-dimensional fixation point-based recognition algorithms need to recognize each frame of image when recognizing objects, which greatly increases the amount of computation, and often encounters a problem of losing the object when facing a fast moving object. This not only affects the accuracy of the identification, but also severely constrains the real-time performance of the system. In addition, since the fixation points of the eyes of the user and the target to be identified have time lag, the target identification which is simply dependent on each three-dimensional fixation point can cause a large number of false positives, and the reliability of the system is reduced.

The single-target tracker is integrated in eye movement tracking, so that the attention model of the user in a real scene can be better simulated, and the gazing target of the user can be accurately reflected. The application of the parameter to dynamic target recognition accords with the actual use experience of a user, the accuracy and the instantaneity of target recognition are obviously improved, and the method is particularly favorable for recognizing and tracking moving targets more stably in a dynamic environment.

Disclosure of Invention

Aiming at the problems, the invention provides an eye tracking dynamic identification method and device based on a three-dimensional gaze point, which are used for solving the problem that a user expects to look at a target inconsistent with an identification target and the problem of server blocking caused by repeated uploading identification.

In order to achieve the purpose, the specific method scheme of the invention is as follows, namely an eye tracking dynamic identification method based on a three-dimensional fixation point comprises the following steps:

s1, acquiring three-dimensional gaze point data of a user by using eye movement equipment, constructing a gaze point-time sequence, and screening out high-density three-dimensional gaze points of the user;

S2, establishing a space coordinate system based on eye movement equipment, wherein the center point of a binocular camera of the eye movement equipment of a user is taken as an original point, the horizontal direction of the user is taken as an X axis, the right side of the user is taken as the positive direction of the X axis, the vertical direction is taken as a Y axis, the upper side of the user is taken as the positive direction of the Y axis, the front and rear directions are taken as Z axes, and the positive direction of the Z axes is taken as the positive direction of the front;

s3, the position of the center point of the eyes of the user is E, the three-dimensional fixation point data is (x _{Gaze fixation},y_{Gaze fixation},z_{Gaze fixation}), the position of the center point of the eyes of the user and the three-dimensional fixation point data are substituted into a space coordinate system based on eye movement equipment, a sight line equation is established to obtain a fixation sight line function L (x _{Line of sight},y_{Line of sight},z_{Line of sight}) of the user, wherein the starting point of L (x _{Line of sight},y_{Line of sight},z_{Line of sight}) is the position E of the center point of the eyes of the user, and the end point is the three-dimensional fixation point (x _{Gaze fixation},y_{Gaze fixation},z_{Gaze fixation});

S4, mapping a gaze line-of-sight function L (x _{Line of sight},y_{Line of sight},z_{Line of sight}) into a foreground camera coordinate system to obtain a gaze point two-dimensional coordinate (x ', y') of a user;

S5, taking a fixation point two-dimensional coordinate (x ', y') as a center, selecting a rectangular area with a length and width set W.times.H pixels, and establishing an identification area initial coordinate (x ', y', W/2, H/2), wherein x ', y' represent the fixation point two-dimensional coordinate, W/2, H/2 represent the distance from the frame of the rectangular area to the center, and setting an image of the rectangular area with the W.times.H pixels as a template picture to be used for subsequent tracking;

S6, loading a weight file trained under a Yolov framework in advance by using a Pytorch framework, initializing a template picture identification model, and extracting deep features of the template picture by using the template picture model, wherein the deep features comprise low-layer features, middle-layer features, high-layer features and global features, and the global features aggregate the three previous features to generate context information with rich semantics for subsequent target classification and definition;

S7, inputting the extracted deep features of the template picture into a head network of a template picture recognition model to carry out classification tasks, and calculating probability distribution of each category by the head network to obtain a result with highest confidence as a recognition result of the template picture;

S8, sending the recognition result of the template picture to eye movement equipment, recording the received result by the eye movement equipment, comparing the received result with the last result, informing the user of the recognition result if the result is changed, and performing no operation if the result is not changed;

S9, initializing a single-target tracker based on a Siamese frame, extracting Jing Shipin frames before each frame from a 30FPS foreground video stream transmitted by eye movement equipment, and sequencing from small numbers to large numbers according to a time sequence. Inputting initial coordinates (x ', y', W/2, H/2) of the identification area and a template image into a template branch of a single-target tracker, inputting a foreground video frame with the minimum number into a search branch of the single-target tracker, and deleting the foreground video frame with the minimum number;

S10, carrying out feature extraction on a template branch and a search branch of a single-target tracker by adopting HRnet networks, and carrying out deep cross-correlation on an obtained template branch feature map and a search branch feature map to obtain a similarity response map;

S11, generating a plurality of anchor points (anchors) in a region with higher confidence in the similarity response graph, and respectively carrying out a regression task and a classification task on each anchor point to generate a tracking result;

S12, marking a tracking result in a video frame of a foreground image, completing a tracking process of a target image in a frame of foreground image picture, and repeating the steps S9 to S12 in a foreground video stream to realize a tracking effect of the target in the foreground video;

S13, while tracking the target in the foreground video is achieved, whether the two-dimensional coordinate (x ', y') of the gaze point of the user deviates from the tracking candidate frame for a long time is monitored, if the two-dimensional coordinate (x ', y') of the gaze point exceeds a set time threshold value and does not enter the candidate frame area, the user is judged to have lost attention to the target, the step S5 is returned to reselect the initial coordinate of the identification area and the template picture, and if the two-dimensional coordinate (x ', y') of the gaze point of the user does not deviate within the time threshold value, the last result is maintained.

Further, the step S1 includes the following:

S1.1, organizing three-dimensional fixation point data into a time sequence according to a time stamp;

s1.2, screening out high-density fixation points in a period by using a density clustering algorithm, and establishing a group of fixation point-time sequences (x, y, z, t) by combining time information, wherein x, y, z represents the high-density three-dimensional fixation points of the screened user, and t represents a time stamp.

Further, the step S1.2 includes:

s1.2.1 defines a space-time distance calculation formula,

Alpha represents a time weight coefficient;

S1.2.2 processing a time sequence by using a DBSCAN algorithm, determining epsilon and MinPts values according to the distribution condition of a three-dimensional point of regard along with time, and comparing the number of points meeting the condition in the neighborhood of each data point with MinPts, wherein epsilon is a neighborhood radius and is used for defining a neighborhood range around one point, minPts is a minimum neighbor point number, and if the number of points around one point is greater than or equal to MinPts, the point is regarded as a core point;

s1.2.3 if one point is confirmed to be a core point and other points exist in the neighborhood of the point, the points belong to the same cluster, clusters which exceed a set quantity threshold are selected, and the points in the clusters represent the screened high-density three-dimensional fixation points of the user.

Further, the step S4 includes the following:

s4.1, the foreground camera coordinate system takes the geometric center of an external camera lens arranged on the eye movement instrument as an origin, and the directions of X, Y and Z axes are consistent with the eye movement equipment space coordinate system;

S4.2, converting the user gazing sight line function L (x _{Line of sight},y_{Line of sight},z_{Line of sight}) and the internal reference of the foreground camera, converting the gazing sight line function under the coordinate system of the eye movement device into the foreground sight line function K (x _{Foreground of},y_{Foreground of},z_{Foreground of}) under the coordinate system of the foreground camera by utilizing a view transformation matrix, obtaining a conversion formula of K (x, y, z) =R+L (x, y, z) +T,

Wherein R is a 3×3 rotation matrix representing rotation of the eye movement device coordinate system relative to the foreground camera coordinate system, and T is a 3×1 translation vector representing the position of the eye movement device origin in the foreground camera coordinate system;

S4.3 under the foreground camera coordinate system, the X-axis and Y-axis coordinates of the foreground Jing Shixian function K (X _{Foreground of},y_{Foreground of},z_{Foreground of}) when Z _{Foreground of} equals g are the gaze point two-dimensional coordinates (X ', Y ') of the user ' S three-dimensional gaze point on the foreground image, assuming that the foreground image curtain is parallel to the X-axis Y-axis under the foreground camera coordinate system, the Z-axis coordinates are g, according to the front Jing Shixian function K (X _{Foreground of},y_{Foreground of},z_{Foreground of}).

Further, the step S7 includes the following:

S7.1, the extracted deep features are transmitted to a head network of a template picture recognition model, and PAnet is used for further polymerizing and optimizing the multi-layer features so as to ensure that the model has global and local features;

S7.2, outputting confidence scores of each category according to the classification loss function yolov of the multi-layer characteristics after aggregation;

and S7.3, selecting the category with the highest confidence as a final template picture recognition result according to the confidence scores of all the categories.

Further, the step S10 includes the following:

S10.1, the template branches and the search branches input by the system adopt a characteristic extraction network structure with shared weight, so that the template branches and the search branches are matched in the same characteristic space, and the backbone network of the high-resolution characteristic extraction network HRnet with the same structure is utilized to respectively extract the characteristics of the template branches and the search branch images so as to retain more detail information;

S10.2, after feature extraction is completed, performing deep cross-correlation operation on the obtained template branch feature map and the search branch feature map, wherein the deep cross-correlation operation is to perform convolution operation on the template branch feature map and the search branch feature map in a channel dimension channel by channel to obtain a two-dimensional similarity response map, and the value of each position represents the similarity between the template and the corresponding position of the search area.

Further, the step S11 includes the following:

S11.1, based on the obtained similarity response graph, setting a threshold according to the percentage of the maximum value, and marking the area which is 80% higher than the maximum value as a high-confidence area; generating a plurality of anchor points in a high confidence coefficient region of the similarity response graph, wherein the anchor points are centered on the high confidence coefficient region, each anchor point is provided with a plurality of different scales and aspect ratios respectively so as to adapt to the shape and the size of a target, and the anchor points serve as a reference frame for target detection and provide a basis for subsequent regression and classification tasks;

S11.2, respectively carrying out a regression task and a classification task on each anchor point, wherein the regression task calculates one with highest regression score in a plurality of different scales and aspect ratios according to a regression loss function to be used as the size and aspect ratio of an anchor frame to enable the anchor frame to be closer to a target, and the classification branch generates a confidence score for each anchor point according to the classification loss function to represent the probability of the target contained in the anchor point, finds the anchor point with the highest score according to the score of the classification branch and takes a candidate frame of the anchor point as a tracking result.

The invention further provides an eye movement tracking dynamic identification device based on the three-dimensional gazing point, which comprises an eye state detection device, a two-dimensional gazing point mapping device, a target identification device, a single target tracking device, a gazing deviation monitoring device and an identification result broadcasting device, wherein the eye state detection device is used for screening the three-dimensional gazing point in a gazing state according to the three-dimensional gazing point provided by eye movement equipment and according to the gazing time and the gazing point distribution, the two-dimensional gazing point mapping device is responsible for converting the screened three-dimensional gazing point into corresponding two-dimensional coordinates and mapping the corresponding two-dimensional coordinates onto a foreground image corresponding to the two-dimensional gazing point, the target identification device is responsible for identifying an image of a user gazing point area, identifying a target through a local identification library or an online identification interface, extracting the target as a template picture, the single target tracking device is used for tracking the target, the single target tracking device is used for combining information of the tracked target with the user gazing point information, judging whether the area deviates from the target in a long time, and broadcasting the voice gazing point is used for broadcasting the identified result to a user, and the voice broadcasting is provided for a conventional operation.

Compared with the prior art, the method has the advantages that the obtained result is more in line with the target expected to be watched by the user, the loss phenomenon caused by deformation or high-speed movement of the target can be effectively reduced, and even if the target is temporarily lost, the target can be still obtained again by combining the user's watching point. Meanwhile, the method does not need to repeatedly carry out repeated uploading identification on the same target, reduces the communication quantity between the server and the method, and reduces the pressure of the server.

Drawings

Fig. 1 is a flow chart of a method for tracking a dynamic target based on eye movement of a three-dimensional gaze point.

Fig. 2 is a schematic diagram of an eye tracking dynamic target device structure based on a three-dimensional gaze point.

FIG. 3 is a flow chart of a single target tracking device.

Detailed Description

The invention will be further described with reference to the drawings and the specific examples, it being pointed out that the method variant and the design principle of the invention will be described in more detail below with only one optimized method variant, but the scope of the invention is not limited thereto.

The examples are preferred embodiments of the present invention, but the present invention is not limited to the above-described embodiments, and any obvious modifications, substitutions or variations that can be made by a person skilled in the art without departing from the spirit of the present invention are within the scope of the present invention.

The invention relates to an eye movement tracking dynamic identification method based on a three-dimensional fixation point, which is shown in fig. 1 and comprises the following steps:

as a preferred embodiment of the present invention, step S1 includes the following:

S1.2, screening out high-density fixation points in a period by using a density clustering algorithm, and establishing a group of fixation point-time sequences (x, y, z, t) by combining time information, wherein x, y, z represents the screened out high-density three-dimensional fixation points of the user, and t represents a time stamp.

As a preferred embodiment of the present invention, step S1.2 comprises:

s1.2.1 defines a space-time distance calculation formula,

Alpha represents a time weight coefficient;

S2) establishing a space coordinate system based on eye movement equipment, wherein the center point of a binocular camera of the eye movement equipment of a user is taken as an original point, the horizontal direction of the user is taken as an X axis, the right side is taken as the positive direction of the X axis, the vertical direction is taken as a Y axis, the upper side is taken as the positive direction of the Y axis, the front-back direction is taken as a Z axis, and the positive direction of the Z axis is taken as the positive direction;

s3) the position of the center point of the eyes of the user is E, the three-dimensional fixation point data is (x _{Gaze fixation},y_{Gaze fixation},z_{Gaze fixation}), the position of the center point of the eyes of the user and the three-dimensional fixation point data are substituted into a space coordinate system based on eye movement equipment, a sight line equation is established to obtain a fixation sight line function L (x _{Line of sight},y_{Line of sight},z_{Line of sight}) of the user, wherein the starting point of L (x _{Line of sight},y_{Line of sight},z_{Line of sight}) is the position E of the center point of the eyes of the user, and the end point is the three-dimensional fixation point (x _{Gaze fixation},y_{Gaze fixation},z_{Gaze fixation});

S4, mapping the gaze point sight line function L (x _{Line of sight},y_{Line of sight},z_{Line of sight}) into a foreground camera coordinate system to obtain the gaze point two-dimensional coordinates (x ', y') of the user.

As a preferred embodiment of the present invention, step S4 includes the following:

S6, loading a weight file trained under a Yolov framework in advance by using a Pytorch framework, initializing a template picture identification model, and extracting deep features of the template picture by using the template picture model, wherein the deep features comprise low-layer features, middle-layer features, high-layer features and global features, and the global features aggregate the three previous features to generate context information with rich semantics for subsequent target classification and definition.

As a preferred embodiment of the invention, the invention uses Pytorch frames to load weight files that have been trained in advance under Yolov8 frames, initializing a template picture model. The weight file mainly contains the following four contents:

s6.1 model architecture information comprises specific layers of a network, and parameters of each layer comprise channel number, stride and convolution kernel size.

S6.2, loss function related parameters, including the regression loss of the boundary box, are used for predicting the coordinates and the size of the boundary box. And judging whether an object exists at a certain position or not according to the target confidence loss. And the classification loss is responsible for detecting classification tasks of the category.

S6.3, detecting head parameters, which are used for boundary box prediction and comprise weights of a classification head, a confidence coefficient head and a regression head.

And S6.4, training state information, including training super-parameters such as learning rate, batch Size and WEIGHT DECAY.

As a preferred embodiment of the invention, the template picture is input into the template picture model for preprocessing, the image is scaled up equally, and it is adjusted (640) to black fill on the shorter side to complement the size. And carrying out normalization operation on the amplified pictures, and adjusting the channel sequence to be (3,640,640) through dimension conversion so as to adapt to the input requirement of the model. Inputting the preprocessed image into a template picture recognition model, and extracting deep features through a backbone network of the template picture recognition model, wherein the deep features comprise low-layer features, middle-layer features, high-layer features and global features. The low-layer feature resolution is higher, and rich edge and texture information is reserved. The middle layer features are used to extract the contour, shape and local structural information of the object. The high-level features are used to identify the class of the object, background information. The global features aggregate the three previous features to generate semantically rich context information for subsequent target classification and definition.

S7, inputting the extracted deep features of the template picture into a head network of a template picture recognition model to carry out classification tasks, and calculating probability distribution of each category by the head network to obtain a result with highest confidence as a recognition result of the template picture.

As a preferred embodiment of the present invention, step S7 includes the following:

and S10, extracting features of a template branch and a search branch of the single-target tracker by adopting HRnet networks, and performing deep cross-correlation on the obtained template branch feature map and the obtained search branch feature map to obtain a similarity response map.

As a preferred embodiment of the present invention, step S10 includes the following:

As a preferred embodiment of the present invention, step S11 includes the following:

S12, marking a tracking result in a video frame of a foreground image, completing a tracking process of a target image in a frame of foreground image picture, and repeating S9 to S12 in a foreground video stream to realize a tracking effect of the target in the foreground video;

As a preferred embodiment of the present invention, as shown in fig. 3, the implementation of the single-object tracking device of the present invention includes the following:

1) The improved Siamese model is loaded, and the initial coordinates (x ', y', W/2, H/2) of the identification area and the template image are input into the template branch of the single-target tracker. And inputting the foreground video frame with the minimum number into a searching branch of the single-target tracker, and deleting the foreground video frame with the minimum number.

2) The system integration HRnet network performs feature extraction on the template image and the search image respectively to retain more high-resolution detailed information.

3) And performing depth cross-correlation operation on the obtained template branch feature map and the search branch feature map to generate a two-dimensional similarity response map, wherein the value of each position represents the similarity between the template and the corresponding position of the search area.

4) And screening out a high-confidence-degree region according to a set threshold value based on the obtained similarity response graph, and generating a plurality of anchor points in the high-confidence-degree region for subsequent target positioning.

5) And respectively carrying out a regression task and a classification task for each anchor point, wherein the regression task determines the size and the aspect ratio of the anchor frame. The classification task finds the anchor point with the highest score and takes its candidate frame as the tracking result.

6) And outputting a final result, continuously inputting the foreground video frame with the minimum number into a searching branch of the single-target tracker, and repeating the processes from the first step to the sixth step.

Claims

1. A method for dynamic target recognition based on eye tracking of three-dimensional gaze points, characterized by comprising the following steps:

S1 uses eye-tracking equipment to obtain the user's 3D gaze point data, constructs a gaze point-time series, and filters out the user's high-density 3D gaze points;

S2 establishes a spatial coordinate system based on the eye tracking device, with the center point of the binocular camera of the user's eye tracking device as the origin, the user's horizontal direction as the X-axis, and the right side as the positive direction of the X-axis; the vertical direction as the Y-axis, and the upward direction as the positive direction of the Y-axis; the front-back direction as the Z-axis, and the front direction as the positive direction of the Z-axis;

S3: The position of the center of the user's eyes is E, and the three-dimensional gaze point data is (x _gaze , y _gaze , z _gaze ). Substitute the position of the center of the user's eyes and the three-dimensional gaze point data into the spatial coordinate system based on the eye tracking device, and establish the gaze equation to obtain the user's gaze function L(x _gaze , y _gaze , z _gaze ), where the starting point of L(x _gaze , y _gaze , z _gaze ) is the position of the center of the user's eyes E, and the end point is the three-dimensional gaze point (x _gaze , y _gaze , z _gaze );

S4 maps the gaze point line of sight function L (x _{line of sight} , y _{line of sight} , z _{line of sight} ) to the foreground camera coordinate system to obtain the two-dimensional coordinates of the user's gaze point (x', y');

S5 selects a rectangular area with a set length and width of W*H pixels with the two-dimensional coordinates (x', y') of the gaze point as the center, establishes the initial coordinates of the recognition area (x', y', W/2, H/2), where x', y' represent the two-dimensional coordinates of the gaze point, W/2, H/2 represent the distance from the border of the rectangular area to the center, and sets the image of the W*H pixel rectangular area as a template image for subsequent tracking;

S6 uses the Pytorch framework to load the weight file pre-trained in the Yolov8 framework, initializes the template image recognition model, and uses the template image model to extract deep features of the template image. The deep features include low-level features, mid-level features, high-level features, and global features. The global features aggregate the features of the first three layers to generate semantically rich contextual information for subsequent target classification and definition;

S7 inputs the extracted deep features of the template image into the head network of the template image recognition model for classification. The head network calculates the probability distribution of each category and obtains the result with the highest confidence as the recognition result of the template image;

S8 sends the recognition result of the template image to the eye tracking device, which records the result received this time and compares the current result with the previous result. If the result changes, the recognition result is notified to the user; if there is no change, no operation is performed;

S9 initializes a single-target tracker based on the Siamese framework. It extracts each foreground video frame from the 30FPS foreground video stream transmitted by the eye tracking device and numbers them in ascending order of time. It inputs the initial coordinates (x’, y’, W/2, H/2) of the recognition area and the template image into the template branch of the single-target tracker. It inputs the foreground video frame with the smallest number into the search branch of the single-target tracker and deletes the foreground video frame with the smallest number.

S10 uses the HRnet network to extract features from the template branch and search branch of the single target tracker, and performs deep cross-correlation on the obtained template branch feature map and search branch feature map to obtain a similarity response map;

S11 generates multiple anchor points in the area with higher confidence in the similarity response map, performs regression and classification tasks on each anchor point, and generates tracking results;

S12 marks the tracking result in the video frame of the foreground image, completing the tracking process of the target image in one frame of the foreground image, and repeating steps S9 to S12 in the foreground video stream to achieve the tracking effect of the target in the foreground video;

While tracking the target in the foreground video, S13 monitors whether the two-dimensional coordinates (x’, y’) of the user's gaze point deviates from the tracking candidate frame for a long time. If the two-dimensional coordinates (x’, y’) of the gaze point exceed the set time threshold and do not enter the candidate frame area, it is determined that the user has lost attention on the target, and returns to S5 to reselect the initial coordinates of the recognition area and the template image; if there is no deviation within the time threshold, the previous result is maintained.

2. The method for dynamic target recognition based on three-dimensional gaze point eye tracking according to claim 1, wherein step S1 comprises the following contents:

S1.1 Organize the 3D gaze point data into time series according to timestamps;

S1.2 uses a density clustering algorithm to filter out high-density gaze points within a time period, and combines time information to establish a set of gaze point-time series (x, y, z, t), where x, y, z represent the high-density three-dimensional gaze points of the filtered user, and t represents the timestamp.

3. The method for dynamic target recognition based on three-dimensional gaze point eye tracking according to claim 2, wherein step S1.2 comprises:

S1.2.1 Define the space-time distance calculation formula,

α represents the time weight coefficient;

S1.2.2 Use the DBSCAN algorithm to process the time series. Based on the temporal distribution of 3D fixation points, determine the ε and MinPts values. Compare the number of eligible points in the neighborhood of each data point with MinPts. ε is the neighborhood radius, which defines the range of the neighborhood around a point. MinPts is the minimum number of neighboring points. If the number of points around a point is greater than or equal to MinPts, the point is considered a core point.

S1.2.3 If a point is confirmed to be a core point and there are other points in its neighborhood, these points belong to the same cluster; clusters containing more than the set number threshold are selected, and the points in the cluster represent the selected user's high-density three-dimensional gaze points.

4. The method for dynamic target recognition based on three-dimensional gaze point eye tracking according to claim 1, wherein step S4 comprises the following contents:

The S4.1 foreground camera coordinate system has the geometric center of the lens of the external camera mounted on the eye tracker as its origin, and the directions of the X, Y, and Z axes are consistent with the spatial coordinate system of the eye tracking device;

S4.2 converts the user's gaze function L(x _-line , y _-line , z _-line ) with the intrinsic parameters of the foreground camera. Using the view transformation matrix, the gaze function in the eye tracking device coordinate system is converted to the foreground camera coordinate system to obtain the foreground gaze function K( _xforeground , _yforeground , _zforeground ) in the foreground camera coordinate system. The conversion formula is K(x,y,z)=R*L(x,y,z)+T.

Where R is a 3×3 rotation matrix, representing the rotation of the eye tracking device coordinate system relative to the foreground camera coordinate system, and T is a 3×1 translation vector, representing the position of the eye tracking device origin in the foreground camera coordinate system;

S4.3 In the foreground camera coordinate system, according to the foreground line of sight function K( _xforeground , _yforeground , _zforeground ), assuming that the foreground image curtain is parallel to the X-axis and Y-axis in the foreground camera coordinate system, and the Z-axis coordinate is g, the X-axis and Y-axis coordinates of the foreground line of sight function K( _xforeground , _yforeground , _zforeground ) when _zforeground is equal to g are the two-dimensional coordinates (x', y') of the user's three-dimensional gaze point on the foreground image.

5. The method for dynamic target recognition based on three-dimensional gaze point eye tracking according to claim 1, wherein step S7 comprises the following contents:

S7.1 passes the extracted deep features to the head network of the template image recognition model, and uses PAnet to further aggregate and optimize multi-layer features to ensure that the model takes into account both global and local features;

S7.2 outputs the confidence score of each category based on the multi-layer features after aggregation according to the classification loss function of yolov8;

S7.3 selects the category with the highest confidence as the final template image recognition result based on the confidence scores of all categories.

6. The method for dynamic target recognition based on three-dimensional gaze point eye tracking according to claim 1, wherein step S10 comprises the following:

The template branch and search branch of S10.1 use a feature extraction network structure with shared weights to ensure that the template branch and search branch are matched in the same feature space. The backbone network of the high-resolution feature extraction network HRnet with the same structure is used to extract features from the template branch and search branch images respectively to retain more detailed information.

S10.2 After completing the feature extraction, the obtained template branch feature map and the search branch feature map are subjected to a deep cross-correlation operation. The deep cross-correlation operation is to perform a convolution operation on the template branch feature map and the search branch feature map channel by channel in the channel dimension to obtain a two-dimensional similarity response map, in which the value of each position represents the similarity between the template and the corresponding position of the search area.

7. The method for dynamic target recognition based on three-dimensional gaze point eye tracking according to claim 1, wherein step S11 comprises the following contents:

S11.1 sets a threshold based on the percentage of the maximum value based on the obtained similarity response map, marking the area above 80% of the maximum value as a high-confidence area. Multiple anchor points are generated in the high-confidence area of the similarity response map. These anchor points are centered in the high-confidence area and each has a variety of scales and aspect ratios to adapt to the shape and size of the target. These anchor points serve as reference frames for target detection and provide a basis for subsequent regression and classification tasks.

S11.2 performs regression and classification tasks for each anchor point. The regression task calculates the one with the highest regression score among multiple scales and aspect ratios based on the regression loss function as the size and aspect ratio of the anchor box, making it closer to the target; the classification branch generates a confidence score for each anchor point based on the classification loss function, which indicates the probability of the target contained in the anchor point. The anchor point with the highest score is found based on the score of the classification branch, and its candidate box is used as the tracking result.

8. A three-dimensional gaze-based eye tracking dynamic target recognition device, characterized by comprising an eye state detection device, a two-dimensional gaze point mapping device, a target recognition device, a single target tracking device, a gaze deviation monitoring device, and a recognition result broadcasting device, wherein the eye state detection device screens out three-dimensional gaze points in a staring state based on the three-dimensional gaze points provided by the eye movement device, the gaze time, and the distribution of the gaze points; the two-dimensional gaze point mapping device is responsible for converting the screened three-dimensional gaze points into corresponding two-dimensional coordinates and mapping them to a foreground image corresponding to their time; the target recognition device is responsible for identifying the target in the image of the user's gaze point area through a local recognition library or an online recognition interface, and extracting it as a template image; the single target tracking device is responsible for tracking the target and tracking the target in the foreground image using the extracted template image; the gaze deviation monitoring device is responsible for combining the tracking target information with the user's gaze point information to determine whether the gaze area has deviated from the tracking target for a long time; the recognition result broadcasting device is responsible for broadcasting the recognition result to the user through voice and providing a voice broadcast of routine operation prompts.