CN116430992A - Implicit calibration method and device for gaze tracking - Google Patents

Abstract

The invention provides an implicit calibration method and device for sight tracking, comprising the following steps: acquiring N sight tracking results to be calibrated in a calibration window and M current display content frames of the equipment; wherein N is less than M; the method comprises the steps of adjusting pixels of a content frame currently displayed by equipment to a preset resolution to obtain a content frame to be detected; inputting the content frame to be detected into a visual saliency detection module to obtain a visual saliency detection result; inputting the visual saliency detection result to a visual saliency measurement module to obtain a content frame to be calibrated; inputting the content frame to be calibrated to a significant information extraction module to obtain significant information to be input; and inputting the to-be-input significant information and the to-be-calibrated sight tracking result to a sight tracking calibration module to obtain a sight tracking calibration result. The invention utilizes the saliency information to identify video frames and uses only the useful frames to perform calibration, thereby realizing the calibration of sight tracking which is simpler and has better user experience and does not need user cooperation.

Description

Implicit calibration method and device for eye tracking

Technical Field

The present invention relates to the field of information technology, and in particular to an implicit calibration method and device for line of sight tracking.

Background Art

With the rapid development of computer science and technology and industry, human-computer interaction has made great progress. Gaze tracking is a human-computer interaction method. Due to its fast response speed and convenient operation, it is widely used in various fields and plays an increasingly important role in a series of mobile applications. The basic concept behind gaze tracking is to capture the user's eye movement and map it to points on the gaze plane. Since the estimated gaze position deviates from the true position, the calibration process is designed to compensate for the deviation between the estimated gaze position and the true position. Therefore, gaze calibration is an important part of gaze tracking, which converts the human eye coordinates into screen coordinates.

Most existing gaze tracking calibrations are explicit. In a typical calibration process, users are asked to fix their gaze on certain stimuli on the screen, and their eye movements are captured by a camera. In this process, the stimulus serves as the basic truth of the gaze position, and a transformation vector is captured based on the offset between the estimated gaze position and the true position. Assuming that the relative position between the eye and the screen remains unchanged over a short period of time, the transformation vector can be directly applied to the estimated gaze position for gaze correction. However, such methods can impair the user experience, especially in mobile scenarios, resulting in frequent triggering of the recalibration process due to updating the transformation vector.

In summary, the existing calibration methods for gaze tracking require user cooperation in the calibration, which is a complicated process and has a poor user experience.

Summary of the invention

The present invention provides an implicit calibration method and device for eye tracking, so as to solve the defects of user cooperation calibration, complicated process and poor user experience in the prior art, and realize eye tracking calibration which does not require user cooperation, is simpler and has better user experience.

The present invention provides an implicit calibration method for gaze tracking, comprising:

Obtain N to-be-calibrated gaze tracking results and M device current display content frames within the calibration window; where N is less than M;

Adjusting the pixels of the content frame currently displayed by the device to a preset resolution to obtain a content frame to be detected;

Inputting the content frame to be detected into a visual saliency detection module to obtain a visual saliency detection result; the visual saliency detection module is used to detect the saliency of each pixel in the content frame to be detected;

Inputting the visual saliency detection result into a visual saliency measurement module to obtain a content frame to be calibrated; the visual saliency measurement module is used to measure the saliency effectiveness of each pixel in the visual saliency detection result;

Inputting the content frame to be calibrated into a salient information extraction module to obtain salient information to be input; the salient information extraction module is used to extract salient information from the content frame to be calibrated;

The to-be-input salient information and the to-be-calibrated gaze tracking result are input into a gaze tracking calibration module to obtain a gaze tracking calibration result; the gaze tracking calibration module is used to calibrate the to-be-calibrated gaze tracking result.

According to an implicit calibration method for gaze tracking provided by the present invention, the salient information to be input and the gaze tracking result to be calibrated are input into a gaze tracking calibration module to obtain a gaze tracking calibration result, which also includes:

Performing a first denoising and a second denoising on the sight tracking result to be calibrated;

The first denoising specifically includes:

Performing a first denoising operation on the sight tracking result to be calibrated using a first preset formula;

The first preset formula includes:

Wherein, z represents the first denoising score value, x represents the original value, μ represents the mean value of the integer value, and σ represents the standard deviation;

The second denoising specifically includes:

Obtaining a gaze position according to the sight tracking result to be calibrated;

Calculating an average value of the gaze positions corresponding to the salient information to be input to obtain an average gaze position;

Clustering the salient information to be input and the average gaze position to obtain a clustering result;

Determining the most frequently salient region and the roughly fixated region according to the clustering result;

An offset between the most frequently salient region and the centroid of the roughly fixated region is calculated, and a second denoising is performed according to the offset.

According to an implicit calibration method for gaze tracking provided by the present invention, the visual saliency detection module is used to detect the saliency of each pixel in the content frame to be detected, specifically comprising:

Inputting the content frame to be detected into a visual saliency detection module to generate a visual saliency heat map;

The visual saliency heat map is normalized to a preset range to obtain a visual saliency detection result.

According to an implicit calibration method for gaze tracking provided by the present invention, the visual saliency measurement module is used to measure the saliency effectiveness of each pixel in the visual saliency detection result, specifically comprising:

Binarizing the visual saliency detection result according to a first preset threshold value to obtain salient pixels;

The number of salient regions/objects where salient pixels are located in the frame is calculated by connected component analysis;

Calculating, according to a second preset formula, the saliency concentration of the content frame currently displayed by the device corresponding to the visual saliency detection result;

Filtering out visual saliency detection results whose saliency concentration is lower than a second preset threshold, to obtain a content frame to be calibrated;

The second preset formula includes:

Where S represents the saliency concentration; n represents the number of salient regions/objects; _AS represents the salient region/object pixels, and _AT represents the region pixels of the entire frame.

According to an implicit calibration method for eye tracking provided by the present invention, the salient information extraction module is used to extract salient information from the content frame to be calibrated, specifically comprising:

Extracting the coordinates and serial numbers of the salient regions/objects where the pixels having the highest saliency value in each connected component domain of the content frame to be calibrated are located;

According to a third preset formula, the saliency concentration on the content frame to be calibrated, the coordinates and numbers of all the salient regions/objects are compressed into a feature vector V _i to obtain salient information to be input;

The third preset formula includes:

表示显著区域/对象的横坐标，

表示显著区域/对象的纵坐标。Wherein, V _i represents the feature vector, n _i represents the number of the salient region/object, SCS _i represents the saliency concentration on the content frame to be calibrated,

The horizontal coordinate representing the salient region/object,

The vertical coordinate representing the salient region/object.

According to an implicit calibration method for gaze tracking provided by the present invention, the gaze tracking calibration module is used to calibrate the gaze tracking result to be calibrated, specifically comprising:

The salient information to be input and the gaze tracking result to be calibrated are input into a gaze tracking calibration module, and the transformation vector included in the offset is compensated to the gaze tracking result to be calibrated for implicit calibration to obtain a gaze tracking calibration result.

According to an implicit calibration method for gaze tracking provided by the present invention, the salient information to be input and the gaze tracking result to be calibrated are input into a gaze tracking calibration module, and the transformation vector included in the offset is compensated to the gaze tracking result to be calibrated for implicit calibration to obtain the gaze tracking calibration result, which also includes:

Get the relative position of the user and the device;

If the change in the relative position is less than or equal to a third preset threshold, the transformation vector remains unchanged;

If the change in the relative position is greater than a third preset threshold or the scene is switched, the transformation vector is updated.

The present invention also provides an implicit calibration device for eye tracking, comprising:

An acquisition unit, used to acquire N to-be-calibrated sight tracking results and M device current display content frames in the calibration window; wherein N is less than M;

An adjustment unit, configured to adjust the pixels of the content frame currently displayed by the device to a preset resolution to obtain a content frame to be detected;

A visual saliency detection unit, configured to input the content frame to be detected into a visual saliency detection module to obtain a visual saliency detection result; the visual saliency detection module is configured to detect the saliency of each pixel in the content frame to be detected;

A visual saliency measurement unit, configured to input the visual saliency detection result into a visual saliency measurement module to obtain a content frame to be calibrated; the visual saliency measurement module is configured to measure the saliency effectiveness of each pixel in the visual saliency detection result;

A salient information extraction unit, configured to input the content frame to be calibrated into a salient information extraction module to obtain salient information to be input; the salient information extraction module is configured to extract salient information from the content frame to be calibrated;

The gaze tracking calibration unit is used to input the salient information to be input and the gaze tracking result to be calibrated into the gaze tracking calibration module to obtain the gaze tracking calibration result; the gaze tracking calibration module is used to calibrate the gaze tracking result to be calibrated.

The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, an implicit calibration method for eye tracking as described above is implemented.

The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements any of the above-described implicit calibration methods for line of sight tracking.

The present invention provides an implicit calibration method and device for eye tracking, which obtains N eye tracking results to be calibrated and M content frames currently displayed by the device within a calibration window; wherein N is less than M; adjusts the pixels of the content frame currently displayed by the device to a preset resolution to obtain a content frame to be detected; inputs the content frame to be detected into a visual saliency detection module to obtain a visual saliency detection result; the visual saliency detection module is used to detect the saliency of each pixel in the content frame to be detected; inputs the visual saliency detection result into a visual saliency measurement module to obtain the content frame to be calibrated; the visual saliency measurement module is used to measure the saliency effectiveness of each pixel in the visual saliency detection result; inputs the content frame to be calibrated into a saliency information extraction module to obtain saliency information to be input; the saliency information extraction module is used to extract saliency information from the content frame to be calibrated; inputs the saliency information to be input and the eye tracking result to be calibrated into an eye tracking calibration module to obtain an eye tracking calibration result; and the eye tracking calibration module is used to calibrate the eye tracking result to be calibrated. The present invention utilizes saliency information to identify video frames, and only uses these "useful" frames to perform calibration, thereby achieving a simpler eye tracking calibration that does not require user cooperation and provides a better user experience.

BRIEF DESCRIPTION OF THE DRAWINGS

In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

FIG1 is a schematic diagram of an optical reflection model of a user's eyes and a screen according to an embodiment of an implicit calibration method for eye tracking provided by the present invention;

FIG2 is a schematic diagram of a flow chart of an implicit calibration method for gaze tracking provided by the present invention;

FIG3 is a second flow chart of the implicit calibration method for gaze tracking provided by the present invention;

FIG4 is a picture of an embodiment of an implicit calibration method for gaze tracking provided by the present invention and a corresponding gaze point and visual saliency heat map;

FIG5 is a diagram of user attention migration in an embodiment of an implicit calibration method for eye tracking provided by the present invention;

FIG6 is a schematic diagram of the structure of an implicit calibration device for eye tracking provided by the present invention;

FIG. 7 is a schematic diagram of the structure of an electronic device provided by the present invention.

Reference numerals:

610: acquisition unit; 620: adjustment unit; 630: visual saliency detection unit; 640: visual saliency measurement unit; 650: salient information extraction unit; 660: gaze tracking calibration unit;

710: processor; 720: communication interface; 730: memory; 740: communication bus.

DETAILED DESCRIPTION

In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

The basic concept behind gaze tracking is to capture the user’s eye movements and map them to points on the gaze plane (i.e., the screen), as shown in Figure 1. Two important pieces of information are required in this process:

The first is a 3D model of the eye, based on which the visual axis, i.e., the gaze direction, can be estimated. Then, the gaze point is determined by the intersection of the visual axis and the gaze plane. Among them, the 3D model of the eye can be captured by an RGB-D camera, which is widely used in today's smartphones, such as iPhoneX, Huawei Mate20, OPPOFindX, etc. A presumed projector projects a beam of structured infrared light onto the user's face, especially the eye area, and the RGB-D camera captures the reflection of the eye. The reflected structured light contains depth information, based on which a 3D motion model of the eye can be constructed. The advantage of using infrared light is that it is not easily perceived by the human eye and is not affected by ambient light. In addition, it can protect the user's privacy.

The second is the relative position between the user's eyes and the gaze plane. Without this information, the estimated gaze position will deviate from the true position. Therefore, a gaze calibration process is needed to compensate for this offset. In a typical calibration process, users are asked to fix their gaze on some stimuli on the screen and their eye movements are captured by a camera. In this process, the stimulus serves as the ground truth of the gaze position and a transformation vector is captured based on the offset between the estimated gaze position and the true position. Assuming that the relative position between the eyes and the screen remains unchanged over short periods of time, the transformation vector can be directly applied to the estimated gaze position for gaze correction.

However, as pointed out in the background, such an explicit calibration process will impair the user experience, especially in mobile scenarios, where the recalibration process needs to be triggered frequently to update the transformation vectors.

Based on this, the present invention proposes an implicit calibration method for gaze tracking. The design of the present invention originates from the understanding of the spatiotemporal dependency between visual saliency and user gaze.

The implicit calibration method for gaze tracking of the present invention is described below in conjunction with FIGS. 1 to 5 . FIGS. 2 and 3 are flowchart diagrams of the implicit calibration method for gaze tracking provided by the present invention. As shown in FIG. 2 , the present invention provides an implicit calibration method for gaze tracking, including:

Step 110: Obtain N to-be-calibrated gaze tracking results and M device current display content frames within the calibration window; wherein N is less than M.

As shown in Figure 3, the present invention uses visual saliency for calibration. The input for calibration is a frame from a video or AR content currently played by the device and the corresponding sight tracking result to be calibrated. In some embodiments, the present invention uses a calibration window to split the device's current display content frame and the sight tracking result to be calibrated for calibration. Each calibration window includes N frames {F ₁ , F ₂ , ... , F _N } and M sight tracking results to be calibrated {E ₁ , E ₂ , ... , _EM }. In order to eliminate eye positioning errors, multiple sight tracking results to be calibrated are used in one frame. That is, the sampling rate of the sight tracking result to be calibrated is higher than the frame rate (ie, M>N).

It should be noted that the gaze tracking result to be calibrated can be a rough estimate of the gaze position obtained by roughly projecting the user's eye movement onto the screen coordinates.

Step 120: Adjust the pixels of the content frame currently displayed by the device to a preset resolution to obtain a content frame to be detected.

The visual saliency heat map can be used as a probability distribution of gaze positions, which provides an opportunity for implicit gaze correction. The present invention uses two saliency levels to represent the user's attention and also dynamically selects a suitable detection algorithm according to different time sequences.

Prior to this, the frame size needs to be adjusted to a preset resolution to obtain a content frame to be detected. In some embodiments, the preset resolution may be 68×68.

The reasons for performing this step are: first, processing frames with high multiplexing rates, such as 4K (3840×2160), incurs high CPU, GPU, and energy overheads, which are unaffordable for mobile devices with limited resources. Second, since videos have different resolutions, it is impossible to predict the resolution of each video in advance. Therefore, resizing all frames to a fixed resolution is an effective and efficient way to solve this problem. It should be emphasized that in this step, reducing the resolution of the frame does not affect the salient detection accuracy. This is because the features of the frame used for salient detection (i.e., color, intensity, orientation, shape of the object, etc.) do not change at lower resolutions.

Step 130: inputting the content frame to be detected into a visual saliency detection module to obtain a visual saliency detection result; the visual saliency detection module is used to detect the saliency of each pixel in the content frame to be detected.

After the content frame to be detected is input into the visual saliency detection module, it will be fed into the visual saliency detection component to generate a visual saliency heat map. Then, each heat map is normalized to a preset range to maintain consistency and obtain the visual saliency detection result.

The basic idea behind saliency-based calibration is that users are usually attracted by several salient regions/objects on the screen when watching a video. Such salient regions/objects are collectively referred to as saliency, which stands out from its neighborhood and can immediately attract the user's attention. Therefore, the locations of these salient regions/objects can be regarded as the basic facts of the user's gaze location, which helps to estimate the user's gaze. Nowadays, with the development of computer vision, many effective methods have been proposed to detect salient regions/objects in videos or frames. These methods usually output a visual saliency heat map, which shows the saliency of each pixel in a frame, which can be regarded as the probability distribution of gaze.

In some embodiments, the present invention uses Apple's algorithm to detect saliency and extract salient areas that distinguish color, intensity, and direction. This saliency is called bottom-up saliency. In a specific embodiment, a video material is selected from the EyeTrackUAV dataset and watched by two volunteers. At the same time, the Tobii eye tracker records their gaze positions, and the gaze positions corresponding to the two frames are extracted, as shown in Figure 4, which shows two video frames and corresponding saliency heat maps. It can be seen that the saliency heat map basically captures the salient areas/objects on the frame and the user's line of sight.

In some other embodiments, the Apple algorithm is used for bottom-up saliency detection. For top-down saliency, U2-Net is selected to detect salient objects.

In terms of time, bottom-up attention and top-down attention are transmitted at the level of 100 milliseconds, more specifically 150 milliseconds. In the saliency detection process designed later, in order to match the change of attention, the bottom-up saliency is used in the initial 150 milliseconds (approximately the length of 5 frames in a 30FPS video), and then turns to the top-down saliency to better match the attention mechanism. For the recognition of scene segmentation, it mainly relies on the detection of key frames. Because in the video encoding process, once a scene cut occurs, it will be encoded as a key frame. Therefore, the key frame covers all scene cuts. By comparing the key frame with the previous frame, it can be detected whether there is a scene cut. In other embodiments, pHash is used to hash the frame and calculate the detection distance. In this way, appropriate saliency can be selected in time to represent the user's visual attention.

Step 140: Input the visual saliency detection result into a visual saliency measurement module to obtain a content frame to be calibrated; the visual saliency measurement module is used to measure the saliency effectiveness of each pixel in the visual saliency detection result.

The saliency heatmap basically shows the probability distribution of gaze. However, this may fail in some scenarios, that is, the user's gaze cannot be obtained from the saliency. For example, when the frame contains multiple salient regions/objects, calibration using saliency works poorly because it is impossible to tell which region the user is looking at. On the contrary, the frame may not even contain saliency, such as a completely black frame. In addition, if the saliency is relatively large (such as a close-up frame), it is impossible to determine the sub-region where the user is looking. In summary, it is not always possible to infer the user's gaze using the saliency of the spatial dimension, which is generally referred to as the spatial validity of saliency.

In addition, ignoring the temporal validity of saliency is another reason why existing saliency-based calibration methods are affected. Visual saliency is temporally related to the user's attention. There are two pathways for human visual attention, bottom-up and top-down. In bottom-up visual attention, the information presented to the brain is the original physical features of external stimuli transmitted through the visual pathway, including color, intensity, direction, etc. In short, bottom-up visual attention is driven by external environmental information. As for top-down attention, it refers to the brain's higher-level articulatory cortex, including the prefrontal cortex (PFC) and the posterior parietal cortex (PPC), which execute information in the visual pathway based on the goals of the current task and past knowledge. This is attention driven by internal information in the brain.

In particular, when watching a video or frame, once a new scene appears, the user will first be driven by bottom-up attention, focusing on different areas in the frame, and then will be dominated by top-down attention, focusing on semantic objects based on past knowledge. As shown in Figure 4. In some implementations of the present invention, a user survey was conducted using this picture to verify the above neural theory. Specifically, volunteers were asked to watch one or another picture on the screen, and then replace the picture with Figure 5 to simulate the scene in the video. The user was asked to report his/her first glance, that is, the area observed subconsciously; and the second glance, that is, the area they later observed after the subconscious action. A total of 6 volunteers participated. According to their feedback, all volunteers first looked at the brightest area (the moon) and then shifted their attention to the correct area (the superhero), which is consistent with the theory.

In order to solve the above problems in spatial dimension, the present invention designs a measurement method to quantify the concentration of salient heatmaps. Only salient heatmaps with high concentration can serve as good calibration opportunities.

The concentration of a salient heatmap is determined by two features. One is the number of salient regions/objects on the heatmap, and the other is the area of the salient regions/objects. Based on this, the present invention proposes a saliency measure, called saliency concentration (SCS), whose calculation formula is expressed as the second preset formula, as follows:

Where S represents the saliency concentration; n represents the number of salient regions/objects; _AS represents the salient region/object pixels, and _AT represents the region pixels of the entire frame. The value of SCS varies between 0 and 1. The smaller the value of n and the ratio of _AS to _AT , the closer the SCS value is to 1, and vice versa.

To calculate the saliency concentration, it is necessary to extract features n and _AS from each frame. To this end, the heat map of each frame is first binarized to filter out background pixels whose saliency values are lower than a first preset threshold to obtain salient pixels. In some embodiments, the first preset threshold for binarization may be 170.

显著区域/对象的数量n可以通过在二值化热图上形成连通分量分析来计算。The pixels remaining after filtering are recorded as salient pixels, reflecting the salient region/object. The region of the salient pixel set is the salient region/object. Obviously, the ratio of the remaining pixels gives the ratio

The number n of salient regions/objects can be calculated by performing a connected component analysis on the binarized heatmap.

通过第二预设公式计算得到显著性热图对应的显著性集中度。所述显著性集中度也是视觉显著检测结果对应的设备当前显示内容帧的显著性集中度，同时也是待校准内容帧的显著性集中度。After calculating the number of salient regions/objects n, the ratio

The saliency concentration corresponding to the saliency heat map is calculated by the second preset formula. The saliency concentration is also the saliency concentration of the content frame currently displayed by the device corresponding to the visual saliency detection result, and is also the saliency concentration of the content frame to be calibrated.

The saliency concentration can be used to select saliency in space, and this metric is used to select frames that can be used for calibration. Before performing saliency detection, a selection has been made to determine the saliency type of each frame. After this time selection and saliency detection, there is still a problem, that is, not all frames can provide good opportunities for implicit calibration, so it is necessary to filter out values with lower SCS values. Specifically, the frames with SCS values lower than the second preset threshold are filtered out to obtain the content frames to be calibrated, that is, the content frames to be calibrated that are filtered out can better perform implicit calibration. In some embodiments, the second preset threshold can be 0.6.

Step 150: inputting the content frame to be calibrated into a salient information extraction module to obtain salient information to be input; the salient information extraction module is used to extract salient information from the content frame to be calibrated.

Afterwards, extract salient information from the remaining frames (i.e., the content frames to be calibrated) that can be better implicitly calibrated. Specifically, for each frame F _i , find the pixel with the highest salient value in each connected component domain therein, and its coordinates represent the position of the corresponding salient region/object. Then, compress the coordinates of all n _i salient regions/objects on F _i together with the salient region/object number n _i and the SCS value SCS _i into a feature vector V _i , as shown below, i.e., the third preset formula:

表示显著区域/对象的横坐标，

表示显著区域/对象的纵坐标。Wherein, V _i represents the feature vector, n _i represents the number of the salient region/object, SCS _i represents the saliency concentration on the content frame to be calibrated,

The horizontal coordinate representing the salient region/object,

The vertical coordinate representing the salient region/object.

The feature vector V _i of each frame is recorded as the salient information to be input, which is fed to the calibration component in the gaze tracking calibration module for implicit gaze error compensation.

Step 160: inputting the salient information to be input and the gaze tracking result to be calibrated into a gaze tracking calibration module to obtain a gaze tracking calibration result; the gaze tracking calibration module is used to calibrate the gaze tracking result to be calibrated.

The visual saliency vector ({V ₁ , V ₂ , ..., V _N }) composed of all feature vectors in a frame can be used to correct the error in the gaze position {G ₁ , G ₂ , ..., G _M } in the gaze tracking result to be calibrated.

But before that, the calibrated gaze tracking results need to be preprocessed to filter out two noise sources in the results.

The first noise source is erased by blink events. Specifically, human eyes usually blink 15-20 times in one minute. When a user blinks, the gaze position will change rapidly. A similar phenomenon can also be observed in saccade events. However, the gaze patterns are different in these two events. For blink events, the gaze position changes quickly, but quickly returns to the original position. Therefore, in this case, z-Score can be used to eliminate outliers. Here, the calculation formula of z-Score can be expressed as the first preset formula:

Wherein, z represents the first denoising score value, x represents the original value, μ represents the mean of the integer values, and σ represents the standard deviation.

In actual operation, the z-score of each rough gaze position in the calibration window is calculated, and if the absolute value of its z-score is greater than the z-score threshold, it is identified as an outlier. In some embodiments, the z-score threshold can be set to 3.

由于在一个校准窗口中使用N个帧，因此将获得N个平均注视位置

之后，从待输入显著信息获取显著向量，使用N个平均注视位置

和N个显著向量{v}来进行注视校准。在一些实施例中，可以将显著向量{v}的数量N设置为10。虽然可以使用单个显著向量来预测真实的注视位置，但是由于单个帧可能包含多个显著区域/对象，因此精度仍然不足以确定注视点的准确位置。此外，用户行为的不确定性也会导致单帧的波动。例如，用户的注意力有时会被背景中不明显的区域吸引。因此，利用N个帧的序列来消除这种误差。具体地说，分别对{v}和

进行聚类。然后，对于{v}和

的聚类结果，选择样本最多的聚类来分别表示最频繁的显著区域和对应的粗略注视区域。通过计算两个区域的质心之间的偏移量，得到一个称为校准变换向量V_c的矢量。利用该向量可以补偿粗跟踪结果中的误差。In addition to blink events, errors in eye positioning can also introduce noise into the gaze tracking results, which is called second noise. Specifically, it causes slight jitter in the rough gaze position. In order to filter out this jitter, multiple eye positions are sampled for each frame. The gaze tracking result to be calibrated includes the gaze position. The gaze position is obtained according to the gaze tracking result to be calibrated, and the average value of N _i rough gaze positions {G ₁ , G ₂ , ..., G _M } corresponding to a feature vector V _i is calculated. The average gaze position is expressed as

Since N frames are used in one calibration window, N average gaze positions will be obtained

After that, the salient vector is obtained from the input salient information, using N average gaze positions

And N salient vectors {v} are used to perform gaze calibration. In some embodiments, the number N of salient vectors {v} can be set to 10. Although a single salient vector can be used to predict the true gaze position, the accuracy is still not enough to determine the exact position of the gaze point because a single frame may contain multiple salient regions/objects. In addition, the uncertainty of user behavior can also cause fluctuations in a single frame. For example, the user's attention is sometimes attracted by an obscure area in the background. Therefore, a sequence of N frames is used to eliminate this error. Specifically, {v} and

Then, for {v} and

The clustering results of the training set are selected to represent the most frequent salient region and the corresponding coarse fixation region. By calculating the offset between the centroids of the two regions, a vector called the calibration transformation vector V _c is obtained. This vector can be used to compensate for the error in the coarse tracking result.

In actual operation, an RGB-D camera can be used to track the user's eye movements and roughly project them onto screen coordinates, thereby obtaining a rough estimate of the gaze position, i.e., the gaze tracking result to be calibrated. Afterwards, the rough estimate is used as the input for calibration. Calibration is an opportunistic process that can be called by monitoring head movement and scene cutting. During the calibration process, saliency information is extracted for the frames in the calibration window. Based on the recognition of the spatiotemporal dimension of saliency, bottom-up saliency and top-down saliency are used to match the changes in user attention in the temporal dimension. Then, low-quality saliency maps are filtered by measuring the spatial features of saliency. Appropriate frames are selected for saliency correction and compared with the rough gaze estimate obtained during tracking to generate a transformation vector. The transformation vector is then used to compensate the rough estimate to obtain the gaze tracking calibration result obtained during calibration.

That is, in the second denoising process, by calculating the offset between the centroids of the two regions, a vector called the calibration transformation vector V _c is obtained. This vector can be used to compensate for the error in the rough tracking result. The transformation vector included in the offset is compensated to the sight tracking result to be calibrated for implicit calibration to obtain the sight tracking calibration result.

After completing the above steps, you can basically perform complete gaze tracking.

In the actual tracking operation process, the front RGBD camera is first used to capture the user's 3D eye information {E1, E2, ..., EM}, and the 3D eye information is processed to obtain the gaze position. The intersection of the gaze position and the gaze plane (screen) determines the gaze point {G1, G2, ..., GM}. However, one problem faced here is that the position of the gaze plane relative to the user's eyes is unknown. In the aforementioned steps of the present invention, the inertial information of the mobile phone from the inertial measurement unit (IMU) and the depth information from the camera perspective are combined to estimate the relative position of the screen.

However, the estimation of the above relative position will have some problems when the user holds the phone in landscape posture. In this case, the camera will be rotated 90 degrees to the left or right of the user's face. Therefore, the user's face will be slightly rotated when captured by the camera. The estimated gaze direction is also distorted, especially for the eyes facing the opposite direction of the camera.

Without loss of generality, the present invention first considers the case where the camera is rotated to the left of the user. In the screen coordinate system, the origin is assumed to be at the lower left corner of the screen. Two key observations can be made: i) the farther the line of sight is from the origin along the x-axis, the greater its displacement is when the eyeball rotates to a certain degree; ii) the closer the line of sight is to the origin along the y-axis, the smaller its displacement is when the eyeball rotates to a certain degree. The first observation is caused by the rotation of the user's face captured by the camera. The second observation is caused by the elliptical structure of the eye. In the elliptical structure, the rotation of the eyeball from left to right is more obvious than the rotation from top to bottom, even if the degree of eyeball rotation is the same. In addition, when the user looks up, the eyes are more open than when looking down. Therefore, when the user looks down, it is more difficult for the camera to capture his/her eyes.

Therefore, in order to compensate for the distortion on the x-axis, the rotation of the user's face captured by the front camera of the mobile device is used to compensate the x value of the gaze position (i.e., the sight tracking result to be calibrated). For the distortion on the y-axis, when y is less than the y-axis distortion threshold, a constant value is used to compensate the y value of the viewpoint position (i.e., the sight tracking result to be calibrated). The y-axis distortion threshold can be 300. With this compensation, the present invention can perform gaze tracking in both portrait and landscape postures. Finally, the feature vector contained in the salient information to be input is compensated to the sight tracking result to be calibrated to obtain the calibrated sight tracking calibration result.

In some embodiments, the present invention further includes triggering conditions for the calibration process. The present invention proposes an implicit calibration solution based on a user attention spatiotemporal model, which fundamentally solves the problem of explicit calibration reducing the quality of user experience.

The calibration mechanism provided by the present invention is opportunistic, and the reference is selected in two aspects:

First, calibration is based on a stable relative position between the user and the device. Therefore, when a change in relative position is detected, recalibration is required. Otherwise, the calibration tracking can be performed directly using the pre-calculated transformation vector. In some embodiments, such detection is achieved by tracking the user's face movement using a front RGB-D camera. Specifically, when the relative position between the user and the device changes, the facial posture of the user captured by the camera will inevitably change. Therefore, during the line of sight tracking process, 3D information of the user's face is continuously captured. Once the distance between two consecutive facial postures is greater than a third preset threshold, in some embodiments, the third preset threshold can be 0.005, a change in relative position is detected. A new calibration process is then triggered to update the transformation vector. This recalibration can also be performed during the existing calibration process to maintain the quality of the calibration.

In addition, calibration is performed when a scene switch occurs. As mentioned in step 130, bottom-up attention will dominate the user's gaze immediately after the scene switch. This subconscious behavior has a strong degree of confidence, linking the user's gaze to the bottom-up saliency. Therefore, calibration of the detected scene switch needs to be triggered to maintain the quality of the cut result. For this calibration, in order to maintain consistency with the attention duration, the length of the calibration window can be defined as 5 frames.

This paper proposes an implicit calibration method for gaze tracking. It uses insights into the temporal and spatial dimensions of the user's attention mechanism, tracks the user's head and eye movements, combines analysis of the screen content, and uses the temporal and spatial relationship between the user's attention and the screen content. It uses dynamic low-quality images to implicitly and passively calibrate the user's eye movements, uses "useful" saliency information to identify video frames, and only uses these "useful" frames to perform opportunistic calibration. It solves the problem that the calibration process in extended reality gaze tracking seriously reduces the quality of user experience, and can achieve highly reliable and accurate gaze tracking even in mobile scenes, and finally achieves continuous gaze tracking suitable for extended reality.

Based on the above embodiment, in the method, the to-be-input salient information and the to-be-calibrated gaze tracking result are input into a gaze tracking calibration module to obtain a gaze tracking calibration result, and the method also includes:

Performing a first denoising and a second denoising on the sight tracking result to be calibrated;

The first denoising specifically includes:

Performing a first denoising operation on the sight tracking result to be calibrated using a first preset formula;

The first preset formula includes:

Wherein, z represents the first denoising score value, x represents the original value, μ represents the mean value of the integer value, and σ represents the standard deviation;

The second denoising specifically includes:

Obtaining a gaze position according to the sight tracking result to be calibrated;

Calculating an average value of the gaze positions corresponding to the salient information to be input to obtain an average gaze position;

Clustering the salient information to be input and the average gaze position to obtain a clustering result;

Determining the most frequently salient region and the roughly fixated region according to the clustering result;

An offset between the most frequently salient region and the centroid of the roughly fixated region is calculated, and a second denoising is performed according to the offset.

Specifically, before the salient information to be input and the gaze tracking result to be calibrated are input into the gaze tracking calibration module, the gaze tracking result to be calibrated needs to be preprocessed first to filter out two noise sources in the result.

The first noise source is erased by blink events. Specifically, human eyes usually blink 15-20 times in one minute. When a user blinks, the gaze position will change rapidly. A similar phenomenon can also be observed in saccade events. However, the gaze patterns are different in these two events. For blink events, the gaze position changes quickly, but quickly returns to the original position. Therefore, in this case, z-Score can be used to eliminate outliers. Here, the calculation formula of z-Score can be expressed as the first preset formula:

Wherein, z represents the first denoising score value, x represents the original value, μ represents the mean of the integer values, and σ represents the standard deviation.

In actual operation, the z-score of each rough gaze position in the calibration window is calculated, and if the absolute value of its z-score is greater than the z-score threshold, it is identified as an outlier. In some embodiments, the z-score threshold can be set to 3.

由于在一个校准窗口中使用N个帧，因此将获得N个平均注视位置

之后，使用N个平均注视位置

和N个显著向量{v}来进行注视校准。在一些实施例中，可以将显著向量{v}的数量N设置为10。虽然可以使用单个显著向量来预测真实的注视位置，但是由于单个帧可能包含多个显著区域/对象，因此精度仍然不足以确定注视点的准确位置。此外，用户行为的不确定性也会导致单帧的波动。例如，用户的注意力有时会被背景中不明显的区域吸引。因此，利用N个帧的序列来消除这种误差。具体地说，分别对{v}和

进行聚类。然后，对于{v}和

的聚类结果，选择样本最多的聚类来分别表示最频繁的显著区域和对应的粗略注视区域。通过计算两个区域的质心之间的偏移量，得到一个称为校准变换向量V_c的矢量。利用该向量可以补偿粗跟踪结果中的误差。In addition to blink events, errors in eye positioning can also introduce noise into the gaze tracking results, which is called second noise. Specifically, it causes slight jitter in the coarse gaze position. To filter out this jitter, multiple eye positions are sampled for each frame. The average of the N _i coarse gaze positions {G ₁ , G ₂ , ..., G _M } corresponding to a feature vector V _i is calculated. The average gaze position is expressed as

Since N frames are used in one calibration window, N average gaze positions will be obtained

Then, use the N average gaze positions

And N salient vectors {v} are used to perform gaze calibration. In some embodiments, the number N of salient vectors {v} can be set to 10. Although a single salient vector can be used to predict the true gaze position, the accuracy is still not enough to determine the exact position of the gaze point because a single frame may contain multiple salient regions/objects. In addition, the uncertainty of user behavior can also cause fluctuations in a single frame. For example, the user's attention is sometimes attracted by an obscure area in the background. Therefore, a sequence of N frames is used to eliminate this error. Specifically, {v} and

Then, for {v} and

The clustering results of the training set are selected to represent the most frequent salient region and the corresponding coarse fixation region. By calculating the offset between the centroids of the two regions, a vector called the calibration transformation vector V _c is obtained. This vector can be used to compensate for the error in the coarse tracking result.

Based on the above embodiment, in the method, the visual saliency detection module is used to detect the saliency of each pixel in the content frame to be detected, specifically including:

Inputting the content frame to be detected into a visual saliency detection module to generate a visual saliency heat map;

The visual saliency heat map is normalized to a preset range to obtain a visual saliency detection result.

Specifically, after the content frame to be detected is input into the visual saliency detection module, it will be fed into the visual saliency detection component to generate a visual saliency heat map. Then, each heat map is normalized to a preset range to maintain consistency and obtain a visual saliency detection result.

Based on the above embodiment, in the method, the visual saliency measurement module is used to measure the saliency effectiveness of each pixel in the visual saliency detection result, specifically including:

Binarizing the visual saliency detection result according to a first preset threshold value to obtain salient pixels;

The number of salient regions/objects where salient pixels are located in the frame is calculated by connected component analysis;

Calculating, according to a second preset formula, the saliency concentration of the content frame currently displayed by the device corresponding to the visual saliency detection result;

Filtering out visual saliency detection results whose saliency concentration is lower than a second preset threshold, to obtain a content frame to be calibrated;

The second preset formula includes:

Where S represents the saliency concentration; n represents the number of salient regions/objects; _AS represents the salient region/object pixels, and _AT represents the region pixels of the entire frame.

Specifically, the concentration of the salient heat map is determined by two features. One is the number of salient regions/objects on the heat map, and the other is the area of the salient regions/objects. Based on this, the present invention proposes a saliency measure, called saliency concentration (SCS), whose calculation formula is expressed as the second preset formula, as follows:

Where S represents the saliency concentration; n represents the number of salient regions/objects; _AS represents the salient region/object pixels, and _AT represents the region pixels of the entire frame. The value of SCS varies between 0 and 1. The smaller the value of n and the ratio of _AS to _AT , the closer the SCS value is to 1, and vice versa.

To calculate the saliency concentration, it is necessary to extract features n and _AS from each frame. To this end, the heat map of each frame is first binarized to filter out background pixels whose saliency values are lower than a first preset threshold to obtain salient pixels. In some embodiments, the first preset threshold for binarization may be 170.

显著区域/对象的数量n可以通过在二值化热图上形成连通分量分析来计算。The pixels remaining after filtering are recorded as salient pixels, reflecting the salient region/object. The region of the salient pixel set is the salient region/object. Obviously, the ratio of the remaining pixels gives the ratio

The number n of salient regions/objects can be calculated by performing a connected component analysis on the binarized heatmap.

通过第二预设公式计算得到显著性热图对应的显著性集中度。所述显著性集中度也是视觉显著检测结果对应的设备当前显示内容帧的显著性集中度，同时也是待校准内容帧的显著性集中度。After calculating the number of salient regions/objects n, the combined ratio

The saliency concentration corresponding to the saliency heat map is calculated by the second preset formula. The saliency concentration is also the saliency concentration of the content frame currently displayed by the device corresponding to the visual saliency detection result, and is also the saliency concentration of the content frame to be calibrated.

The saliency concentration can be used to select saliency in space, and this metric is used to select frames that can be used for calibration. Before performing saliency detection, a selection has been made to determine the saliency type of each frame. After this time selection and saliency detection, there is still a problem, that is, not all frames can provide good opportunities for implicit calibration, so it is necessary to filter out values with lower SCS values. Specifically, the frames with SCS values lower than the second preset threshold are filtered out to obtain the content frames to be calibrated, that is, the content frames to be calibrated that are filtered out can better perform implicit calibration. In some embodiments, the second preset threshold can be 0.6.

Based on the above embodiment, in the method, the salient information extraction module is used to extract salient information from the content frame to be calibrated, specifically including:

Extracting the coordinates and serial numbers of the salient regions/objects where the pixels having the highest saliency value in each connected component domain of the content frame to be calibrated are located;

According to a third preset formula, the saliency concentration on the content frame to be calibrated, the coordinates and numbers of all the salient regions/objects are compressed into a feature vector V _i to obtain salient information to be input;

The third preset formula includes:

表示显著区域/对象的横坐标，

表示显著区域/对象的纵坐标。Wherein, V _i represents the feature vector, n _i represents the number of the salient region/object, SCS _i represents the saliency concentration on the content frame to be calibrated,

The horizontal coordinate representing the salient region/object,

The vertical coordinate representing the salient region/object.

Specifically, then, extract salient information from the remaining frames that can be better implicitly calibrated (i.e., the content frames to be calibrated). Specifically, for each frame F _i , find the pixel with the highest salient value in each connected component domain therein, and its coordinates represent the position of the corresponding salient region/object. Then, compress the coordinates of all n _i salient regions/objects on F _i together with the salient region/object number n _i and the SCS value SCS _i into a feature vector V _i , as shown below, i.e., the third preset formula:

表示显著区域/对象的横坐标，

表示显著区域/对象的纵坐标。Wherein, V _i represents the feature vector, n _i represents the number of the salient region/object, SCS _i represents the saliency concentration on the content frame to be calibrated,

The horizontal coordinate representing the salient region/object,

The vertical coordinate representing the salient region/object.

The feature vector V _i of each frame is recorded as the salient information to be input, which is fed to the calibration component in the gaze tracking calibration module for implicit gaze error compensation.

Based on the above embodiment, in the method, the sight tracking calibration module is used to calibrate the sight tracking result to be calibrated, specifically including:

The salient information to be input and the gaze tracking result to be calibrated are input into a gaze tracking calibration module, and the transformation vector included in the offset is compensated to the gaze tracking result to be calibrated for implicit calibration to obtain a gaze tracking calibration result.

Specifically, the visual saliency vector ({V ₁ , V ₂ , ..., V _N }) composed of all feature vectors in a frame can be used to correct the error in the gaze position {G ₁ , G ₂ , ..., G _M } in the gaze tracking result to be calibrated.

In actual operation, an RGB-D camera can be used to track the user's eye movements and roughly project them onto screen coordinates, thereby obtaining a rough estimate of the gaze position, i.e., the gaze tracking result to be calibrated. Afterwards, the rough estimate is used as the input for calibration. Calibration is an opportunistic process that can be called by monitoring head movement and scene cutting. During the calibration process, saliency information is extracted for the frames in the calibration window. Based on the recognition of the spatiotemporal dimension of saliency, bottom-up saliency and top-down saliency are used to match the changes in user attention in the temporal dimension. Then, low-quality saliency maps are filtered by measuring the spatial features of saliency. Appropriate frames are selected for saliency correction and compared with the rough gaze estimate obtained during tracking to generate a transformation vector. The transformation vector is then used to compensate the rough estimate to obtain the gaze tracking calibration result obtained during calibration.

That is, in the second denoising process, by calculating the offset between the centroids of the two regions, a vector called the calibration transformation vector V _c is obtained. This vector can be used to compensate for the error in the rough tracking result. The transformation vector included in the offset is compensated to the sight tracking result to be calibrated for implicit calibration to obtain the sight tracking calibration result.

Based on the above embodiment, in the method, the to-be-input salient information and the to-be-calibrated gaze tracking result are input into a gaze tracking calibration module, and the transformation vector included in the offset is compensated to the to-be-calibrated gaze tracking result for implicit calibration to obtain a gaze tracking calibration result, which also includes:

Get the relative position of the user and the device;

If the change in the relative position is less than or equal to a third preset threshold, the transformation vector remains unchanged;

If the change in the relative position is greater than a third preset threshold or the scene is switched, the transformation vector is updated.

Specifically, when the user holds the phone in landscape position, the estimation of the above relative position will have some problems. In this case, the camera will be rotated 90 degrees to the left or right of the user's face. Therefore, the user's face will be slightly rotated when captured by the camera. The estimated gaze direction is also distorted, especially for the eyes in the opposite direction of the camera.

Without loss of generality, the present invention first considers the case where the camera is rotated to the left of the user. In the screen coordinate system, the origin is assumed to be at the lower left corner of the screen. Two key observations can be made: i) the farther the line of sight is from the origin along the x-axis, the greater its displacement is when the eyeball rotates to a certain degree; ii) the closer the line of sight is to the origin along the y-axis, the smaller its displacement is when the eyeball rotates to a certain degree. The first observation is caused by the rotation of the user's face captured by the camera. The second observation is caused by the elliptical structure of the eye. In the elliptical structure, the rotation of the eyeball from left to right is more obvious than the rotation from top to bottom, even if the degree of eyeball rotation is the same. In addition, when the user looks up, the eyes are more open than when looking down. Therefore, when the user looks down, it is more difficult for the camera to capture his/her eyes.

Therefore, in order to compensate for the distortion on the x-axis, the rotation of the user's face captured by the front camera of the mobile device is used to compensate the x value of the gaze position (i.e., the sight tracking result to be calibrated). For the distortion on the y-axis, when y is less than the y-axis distortion threshold, a constant value is used to compensate the y value of the viewpoint position (i.e., the sight tracking result to be calibrated). The y-axis distortion threshold can be 300. With this compensation, the present invention can perform gaze tracking in both portrait and landscape postures. Finally, the feature vector contained in the salient information to be input is compensated to the sight tracking result to be calibrated to obtain the calibrated sight tracking calibration result.

In some embodiments, the present invention further includes triggering conditions for the calibration process. The present invention proposes an implicit calibration solution based on a user attention spatiotemporal model, which fundamentally solves the problem of explicit calibration reducing the quality of user experience.

The calibration mechanism provided by the present invention is opportunistic, and the reference is selected in two aspects:

First, calibration is based on a stable relative position between the user and the device. Therefore, when a change in relative position is detected, recalibration is required. Otherwise, the calibration tracking can be performed directly using the pre-calculated transformation vector. In some embodiments, such detection is achieved by tracking the user's face movement using a front RGB-D camera. Specifically, when the relative position between the user and the device changes, the facial posture of the user captured by the camera will inevitably change. Therefore, during the line of sight tracking process, 3D information of the user's face is continuously captured. Once the distance between two consecutive facial postures is greater than a third preset threshold, in some embodiments, the third preset threshold can be 0.005, a change in relative position is detected. A new calibration process is then triggered to update the transformation vector. This recalibration can also be performed during the existing calibration process to maintain the quality of the calibration.

In addition, calibration is performed when a scene switch occurs. As mentioned in step 130, bottom-up attention will dominate the user's gaze immediately after the scene switch. This subconscious behavior has a strong degree of confidence, linking the user's gaze to the bottom-up saliency. Therefore, calibration of the detected scene switch needs to be triggered to maintain the quality of the cut result. For this calibration, in order to maintain consistency with the attention duration, the length of the calibration window can be defined as 5 frames.

In a specific embodiment, an iPhone Xs Max is selected as the current device, which integrates a 2.49GHz Apple A12 Bionic, 4GB RAM, a 6.5-inch screen, a TrueDepth camera, and runs an iOS 13.6 operating system. The TrueDepth camera provides an RGB-D camera. The implementation of the implicit calibration method for gaze tracking provided by the present invention can be applied to any iOS device equipped with a TrueDepth camera, such as an iPhone 11, an iPad Pro, etc. In addition, the present invention can be implemented on any Android device equipped with an RGB-D camera, such as Huawei Mate 20, OPPO Find X, Honor Magic 2, etc. The algorithm of the implicit calibration method for gaze tracking provided by the present invention is written in SWIFT and Objective-C++. In order to ensure the repeatability of the evaluation frame between different users, video is used as a visual input in the implementation. This implementation can be easily converted to an AR scene with a simple setting. In order to obtain RGB-D camera data, the ARKit framework is used for OpenCV of iOS to provide multiple frame processing functions.

The present invention provides an implicit calibration method for eye tracking, which obtains N eye tracking results to be calibrated and M content frames currently displayed by the device within a calibration window; wherein N is less than M; the pixels of the content frame currently displayed by the device are adjusted to a preset resolution to obtain a content frame to be detected; the content frame to be detected is input into a visual saliency detection module to obtain a visual saliency detection result; the visual saliency detection module is used to detect the saliency of each pixel in the content frame to be detected; the visual saliency detection result is input into a visual saliency measurement module to obtain the content frame to be calibrated; the visual saliency measurement module is used to measure the saliency effectiveness of each pixel in the visual saliency detection result; the content frame to be calibrated is input into a saliency information extraction module to obtain saliency information to be input; the saliency information extraction module is used to extract saliency information from the content frame to be calibrated; the saliency information to be input and the eye tracking result to be calibrated are input into an eye tracking calibration module to obtain an eye tracking calibration result; the eye tracking calibration module is used to calibrate the eye tracking result to be calibrated. The present invention utilizes saliency information to identify video frames, and only uses these "useful" frames to perform calibration, thereby achieving a simpler eye tracking calibration that does not require user cooperation and provides a better user experience.

The implicit calibration device for gaze tracking provided by the present invention is described below. The implicit calibration device for gaze tracking described below and the implicit calibration method for gaze tracking described above can refer to each other.

FIG6 is a schematic diagram of the structure of an implicit calibration device for eye tracking provided by an embodiment of the present invention. As shown in FIG6 , an implicit calibration device for eye tracking provided by an embodiment of the present invention includes: an acquisition unit 610; an adjustment unit 620; a visual saliency detection unit 630; a visual saliency measurement unit 640; a saliency information extraction unit 650; and an eye tracking calibration unit 660.

in,

An acquisition unit 610 is used to acquire N to-be-calibrated sight tracking results and M device current display content frames in the calibration window; wherein N is less than M;

An adjusting unit 620, configured to adjust the pixels of the content frame currently displayed by the device to a preset resolution to obtain a content frame to be detected;

A visual saliency detection unit 630, configured to input the content frame to be detected into a visual saliency detection module to obtain a visual saliency detection result; the visual saliency detection module is configured to detect the saliency of each pixel in the content frame to be detected;

A visual saliency measurement unit 640 is used to input the visual saliency detection result into a visual saliency measurement module to obtain a content frame to be calibrated; the visual saliency measurement module is used to measure the saliency effectiveness of each pixel in the visual saliency detection result;

The salient information extraction unit 650 is used to input the content frame to be calibrated into a salient information extraction module to obtain salient information to be input; the salient information extraction module is used to extract salient information from the content frame to be calibrated;

The gaze tracking calibration unit 660 is used to input the salient information to be input and the gaze tracking result to be calibrated into the gaze tracking calibration module to obtain the gaze tracking calibration result; the gaze tracking calibration module is used to calibrate the gaze tracking result to be calibrated.

Based on the above embodiment, in the device, the to-be-input salient information and the to-be-calibrated gaze tracking result are input into a gaze tracking calibration module to obtain a gaze tracking calibration result, which also includes:

Performing a first denoising and a second denoising on the sight tracking result to be calibrated;

The first denoising specifically includes:

Performing a first denoising operation on the sight tracking result to be calibrated using a first preset formula;

The first preset formula includes:

Wherein, z represents the first denoising score value, x represents the original value, μ represents the mean value of the integer value, and σ represents the standard deviation;

The second denoising specifically includes:

Obtaining a gaze position according to the sight tracking result to be calibrated;

Calculating an average value of the gaze positions corresponding to the salient information to be input to obtain an average gaze position;

Clustering the salient information to be input and the average gaze position to obtain a clustering result;

Determining the most frequently salient region and the roughly fixated region according to the clustering result;

An offset between the most frequently salient region and the centroid of the roughly fixated region is calculated, and a second denoising is performed according to the offset.

Based on the above embodiment, in the device, the visual saliency detection module is used to detect the saliency of each pixel in the content frame to be detected, specifically including:

Inputting the content frame to be detected into a visual saliency detection module to generate a visual saliency heat map;

The visual saliency heat map is normalized to a preset range to obtain a visual saliency detection result.

Based on the above embodiment, in the device, the visual saliency measurement module is used to measure the saliency effectiveness of each pixel in the visual saliency detection result, specifically including:

Binarizing the visual saliency detection result according to a first preset threshold value to obtain salient pixels;

The number of salient regions/objects where salient pixels are located in the frame is calculated by connected component analysis;

Calculating, according to a second preset formula, the saliency concentration of the content frame currently displayed by the device corresponding to the visual saliency detection result;

Filtering out visual saliency detection results whose saliency concentration is lower than a second preset threshold, to obtain a content frame to be calibrated;

The second preset formula includes:

Where S represents the saliency concentration; n represents the number of salient regions/objects; _AS represents the salient region/object pixels, and _AT represents the region pixels of the entire frame.

Based on the above embodiment, in the device, the salient information extraction module is used to extract salient information from the content frame to be calibrated, specifically including:

Extracting the coordinates and serial numbers of the salient regions/objects where the pixels having the highest saliency value in each connected component domain of the content frame to be calibrated are located;

According to a third preset formula, the saliency concentration on the content frame to be calibrated, the coordinates and numbers of all the salient regions/objects are compressed into a feature vector V _i to obtain salient information to be input;

The third preset formula includes:

表示显著区域/对象的横坐标，

表示显著区域/对象的纵坐标。Wherein, V _i represents the feature vector, n _i represents the number of the salient region/object, SCS _i represents the saliency concentration on the content frame to be calibrated,

The horizontal coordinate representing the salient region/object,

The vertical coordinate representing the salient region/object.

Based on the above embodiment, in the device, the sight tracking calibration module is used to calibrate the sight tracking result to be calibrated, specifically including:

The salient information to be input and the gaze tracking result to be calibrated are input into a gaze tracking calibration module, and the transformation vector included in the offset is compensated to the gaze tracking result to be calibrated for implicit calibration to obtain a gaze tracking calibration result.

Based on the above embodiment, in the device, the to-be-input salient information and the to-be-calibrated gaze tracking result are input into a gaze tracking calibration module, and the transformation vector included in the offset is compensated to the to-be-calibrated gaze tracking result for implicit calibration to obtain a gaze tracking calibration result, which also includes:

Get the relative position of the user and the device;

If the change in the relative position is less than or equal to a third preset threshold, the transformation vector remains unchanged;

If the change in the relative position is greater than a third preset threshold or the scene is switched, the transformation vector is updated.

The present invention provides an implicit calibration device for eye tracking, which obtains N eye tracking results to be calibrated and M content frames currently displayed by the device within a calibration window; wherein N is less than M; adjusts the pixels of the content frame currently displayed by the device to a preset resolution to obtain a content frame to be detected; inputs the content frame to be detected into a visual saliency detection module to obtain a visual saliency detection result; the visual saliency detection module is used to detect the saliency of each pixel in the content frame to be detected; inputs the visual saliency detection result into a visual saliency measurement module to obtain the content frame to be calibrated; the visual saliency measurement module is used to measure the saliency effectiveness of each pixel in the visual saliency detection result; inputs the content frame to be calibrated into a saliency information extraction module to obtain saliency information to be input; the saliency information extraction module is used to extract saliency information from the content frame to be calibrated; inputs the saliency information to be input and the eye tracking result to be calibrated into an eye tracking calibration module to obtain an eye tracking calibration result; the eye tracking calibration module is used to calibrate the eye tracking result to be calibrated. The present invention utilizes saliency information to identify video frames, and only uses these "useful" frames to perform calibration, thereby achieving a simpler eye tracking calibration that does not require user cooperation and provides a better user experience.

Figure 7 illustrates a schematic diagram of the physical structure of an electronic device. As shown in Figure 7, the electronic device may include: a processor (processor) 710, a communication interface (Communications Interface) 720, a memory (memory) 730 and a communication bus 740, wherein the processor 710, the communication interface 720, and the memory 730 communicate with each other through the communication bus 740. The processor 710 can call the logic instructions in the memory 730 to execute an implicit calibration method for eye tracking, which includes: obtaining N eye tracking results to be calibrated and M device currently displayed content frames within the calibration window; wherein N is less than M; adjusting the pixels of the device currently displayed content frame to a preset resolution to obtain a content frame to be detected; inputting the content frame to be detected into a visual saliency detection module to obtain a visual saliency detection result; the visual saliency detection module is used to detect the saliency of each pixel in the content frame to be detected; inputting the visual saliency detection result into a visual saliency measurement module to obtain the content frame to be calibrated; the visual saliency measurement module is used to measure the saliency validity of each pixel in the visual saliency detection result; inputting the content frame to be calibrated into a saliency information extraction module to obtain saliency information to be input; the saliency information extraction module is used to extract saliency information from the content frame to be calibrated; inputting the saliency information to be input and the eye tracking result to be calibrated into an eye tracking calibration module to obtain an eye tracking calibration result; the eye tracking calibration module is used to calibrate the eye tracking result to be calibrated.

In addition, the logic instructions in the above-mentioned memory 730 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the implicit calibration method for eye tracking provided by the above methods, the method comprising: obtaining N eye tracking results to be calibrated and M device current display content frames within a calibration window; wherein N is less than M; adjusting the pixels of the device current display content frame to a preset resolution to obtain a content frame to be detected; inputting the content frame to be detected into a visual saliency detection module to obtain a visual saliency detection result; the visual saliency detection module is used to detect each of the content frames to be detected. The saliency of each pixel; the visual saliency detection result is input into the visual saliency measurement module to obtain the content frame to be calibrated; the visual saliency measurement module is used to measure the saliency effectiveness of each pixel in the visual saliency detection result; the content frame to be calibrated is input into the salient information extraction module to obtain the salient information to be input; the salient information extraction module is used to extract salient information from the content frame to be calibrated; the salient information to be input and the sight tracking result to be calibrated are input into the sight tracking calibration module to obtain the sight tracking calibration result; the sight tracking calibration module is used to calibrate the sight tracking result to be calibrated. The device embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without creative labor.

Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM/RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims (10)

1. An implicit calibration method for eye tracking, comprising: Obtain N to-be-calibrated gaze tracking results and M device current display content frames within the calibration window; where N is less than M; Adjusting the pixels of the content frame currently displayed by the device to a preset resolution to obtain a content frame to be detected; Inputting the content frame to be detected into a visual saliency detection module to obtain a visual saliency detection result; the visual saliency detection module is used to detect the saliency of each pixel in the content frame to be detected; Inputting the visual saliency detection result into a visual saliency measurement module to obtain a content frame to be calibrated; the visual saliency measurement module is used to measure the saliency effectiveness of each pixel in the visual saliency detection result; Inputting the content frame to be calibrated into a salient information extraction module to obtain salient information to be input; the salient information extraction module is used to extract salient information from the content frame to be calibrated; The to-be-input salient information and the to-be-calibrated gaze tracking result are input into a gaze tracking calibration module to obtain a gaze tracking calibration result; the gaze tracking calibration module is used to calibrate the to-be-calibrated gaze tracking result. 2. The implicit calibration method for gaze tracking according to claim 1, characterized in that the salient information to be input and the gaze tracking result to be calibrated are input into a gaze tracking calibration module to obtain a gaze tracking calibration result, before which it further comprises: Performing a first denoising and a second denoising on the sight tracking result to be calibrated; The first denoising specifically includes: Performing a first denoising operation on the sight tracking result to be calibrated using a first preset formula; The first preset formula includes:

Wherein, z represents the first denoising score value, x represents the original value, μ represents the mean value of the integer value, and σ represents the standard deviation; The second denoising specifically includes: Obtaining a gaze position according to the sight tracking result to be calibrated; Calculating an average value of the gaze positions corresponding to the salient information to be input to obtain an average gaze position; Clustering the salient information to be input and the average gaze position to obtain a clustering result; Determining the most frequently salient region and the roughly fixated region according to the clustering result; An offset between the most frequently salient region and the centroid of the roughly fixated region is calculated, and a second denoising is performed according to the offset. 3. The implicit calibration method for eye tracking according to claim 1, wherein the visual saliency detection module is used to detect the saliency of each pixel in the content frame to be detected, specifically comprising: Inputting the content frame to be detected into a visual saliency detection module to generate a visual saliency heat map; The visual saliency heat map is normalized to a preset range to obtain a visual saliency detection result. 4. The implicit calibration method for gaze tracking according to claim 3, wherein the visual saliency measurement module is used to measure the saliency effectiveness of each pixel in the visual saliency detection result, specifically comprising: Binarizing the visual saliency detection result according to a first preset threshold value to obtain salient pixels; The number of salient regions/objects where salient pixels are located in the frame is calculated by connected component analysis; Calculating, according to a second preset formula, the saliency concentration of the content frame currently displayed by the device corresponding to the visual saliency detection result; Filtering out visual saliency detection results whose saliency concentration is lower than a second preset threshold, to obtain a content frame to be calibrated; The second preset formula includes:

Where S represents the saliency concentration; n represents the number of salient regions/objects; _AS represents the salient region/object pixels, and _AT represents the region pixels of the entire frame. 5. The implicit calibration method for eye tracking according to claim 4, wherein the salient information extraction module is used to extract salient information from the content frame to be calibrated, specifically comprising: Extracting the coordinates and serial numbers of the salient regions/objects where the pixels having the highest saliency value in each connected component domain of the content frame to be calibrated are located; According to a third preset formula, the saliency concentration on the content frame to be calibrated, the coordinates and numbers of all the salient regions/objects are compressed into a feature vector V _i to obtain salient information to be input; The third preset formula includes:

Wherein, V _i represents the feature vector, n _i represents the number of the salient region/object, SCS _i represents the saliency concentration on the content frame to be calibrated,

The horizontal coordinate representing the salient region/object,

The vertical coordinate representing the salient region/object. 6. The implicit calibration method for gaze tracking according to claim 2, characterized in that the gaze tracking calibration module is used to calibrate the gaze tracking result to be calibrated, specifically comprising: The salient information to be input and the gaze tracking result to be calibrated are input into a gaze tracking calibration module, and the transformation vector included in the offset is compensated to the gaze tracking result to be calibrated for implicit calibration to obtain a gaze tracking calibration result. 7. The implicit calibration method for gaze tracking according to claim 6, characterized in that the salient information to be input and the gaze tracking result to be calibrated are input into a gaze tracking calibration module, and the transformation vector included in the offset is compensated to the gaze tracking result to be calibrated for implicit calibration to obtain the gaze tracking calibration result, and the method further comprises: Get the relative position of the user and the device; If the change in the relative position is less than or equal to a third preset threshold, the transformation vector remains unchanged; If the change in the relative position is greater than a third preset threshold or the scene is switched, the transformation vector is updated. 8. An implicit calibration device for eye tracking, comprising: An acquisition unit, used to acquire N to-be-calibrated sight tracking results and M device current display content frames in the calibration window; wherein N is less than M; An adjustment unit, configured to adjust the pixels of the content frame currently displayed by the device to a preset resolution to obtain a content frame to be detected; A visual saliency detection unit, configured to input the content frame to be detected into a visual saliency detection module to obtain a visual saliency detection result; the visual saliency detection module is configured to detect the saliency of each pixel in the content frame to be detected; A visual saliency measurement unit, configured to input the visual saliency detection result into a visual saliency measurement module to obtain a content frame to be calibrated; the visual saliency measurement module is configured to measure the saliency effectiveness of each pixel in the visual saliency detection result; A salient information extraction unit, configured to input the content frame to be calibrated into a salient information extraction module to obtain salient information to be input; the salient information extraction module is configured to extract salient information from the content frame to be calibrated; The gaze tracking calibration unit is used to input the salient information to be input and the gaze tracking result to be calibrated into the gaze tracking calibration module to obtain the gaze tracking calibration result; the gaze tracking calibration module is used to calibrate the gaze tracking result to be calibrated. 9. An electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, an implicit calibration method for eye tracking as described in any one of claims 1 to 7 is implemented. 10. A non-transitory computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the implicit calibration method for gaze tracking as claimed in any one of claims 1 to 7 is implemented.

Images