CN107773248B - Eye movement instrument and image processing method - Google Patents

Abstract

The invention belongs to the technical field of eye movement instruments and meters, and particularly relates to an eye movement meter and an image processing method. The eye movement instrument provided by the invention comprises: a head-mounted main body; the front cameras are arranged on the head-mounted main body at intervals along the circumferential direction, and at least two of the front cameras can collect image information of a target scene in front of the head-mounted main body at the same time; the control system is arranged on the head-mounted main body and connected with the front-mounted camera, so that the control system can reconstruct a target scene in three dimensions based on image information and/or measure distance of at least one object in the target scene based on the image information, and the head-mounted main body has a wider acquisition range and a wider field of view. The eye movement instrument provided by the embodiment of the invention has the advantages of simple structure, convenience in use, comfort, safety and reliability in wearing.

Description

Eye tracker and image processing method

Technical Field

The invention belongs to the technical field of eye movement instruments and meters, and in particular relates to an eye tracker and an image processing method.

Background Art

An eye tracker is a device that uses machine vision and camera technology to track and record eye movements. It is an important instrument for research in psychology, advertising, industrial engineering, etc., and is also widely used in eye movement control. Eye trackers are used to record the characteristics of eye movements when people process visual information, and are widely used in research in the fields of attention, visual perception, and reading.

Since existing eye trackers only have one front camera, they can neither accurately measure the distance from the eye to the target object nor perform three-dimensional reconstruction of the target scene. In particular, current eye trackers have major limitations in automatically defining the region of interest. Manually calibrating the region of interest is very time-consuming and laborious. This has caused great trouble and obstacles to research in the fields of psychology, visual perception, and reading.

Summary of the invention

Based on the above problems, the present invention provides an eye tracker, which can perform three-dimensional reconstruction of a target scene and/or measure the distance of an object in the target scene and/or automatically define a region of interest, and has a wider acquisition range and field of view. The eye tracker of the embodiment of the present invention has a simple structure, is easy to use, and is comfortable, safe, and reliable to wear.

In a first aspect, the eye tracker proposed in the present invention includes: a head-mounted body; a plurality of front cameras, which are arranged on the head-mounted body at intervals along the circumferential direction, and at least two of the plurality of front cameras can simultaneously collect image information of a target scene in front of the head-mounted body; a control system arranged on the head-mounted body, the control system being connected to the front cameras, so that the control system can perform three-dimensional reconstruction of the target scene based on the image information, and/or measure the distance to at least one object in the target scene based on the image information.

Furthermore, the eye tracker of this embodiment also includes a nose pad arranged at the front end of the head-mounted body. Among the multiple front cameras, some of the front cameras are located on one side of the nose pad, and another part of the front cameras are located on the other side of the nose pad.

Furthermore, the connection between the nose pad and the head-mounted body is a detachable connection.

Based on any of the above eye tracker embodiments, further, the connection between the front camera and the head-mounted body is a detachable connection.

Based on any of the above eye tracker embodiments, further, the front camera includes a visible light camera, an infrared camera or a thermal imaging camera.

Furthermore, the eye tracker of this embodiment also includes an eye camera for collecting image information of the subject's eyes, the eye camera is connected to the head-mounted body through a supporting component, and the eye camera is connected to the control system, so that the control system can know whether the user is focused on the target object in the target scene based on the collected image information of the subject's eyes and the collected image information of the target scene. In the target scene, the position of the target object relative to a pre-set marker is fixed, the marker includes an identification pattern and/or an identification light, the identification pattern includes a one-dimensional barcode and/or a two-dimensional code, and the identification light includes fluorescence and/or an invisible infrared light source.

In a second aspect, the present invention provides an image processing method for three-dimensional reconstruction, the method comprising the following steps:

Step S11: obtaining calibration image data synchronously photographed by at least two camera devices on a calibration three-dimensional scene of a predetermined marker;

Step S12: generating imaging association information of at least two shooting devices for the same shooting scene according to the corrected image data and the pre-stored three-dimensional coordinate data of the predetermined marker;

Step S13: Acquire collected image data of the target three-dimensional scene synchronously photographed by at least two calibrated photographing devices;

Step S14: synthesizing the collected image data into a three-dimensional stereo image according to the imaging association information, where the three-dimensional stereo image is the target three-dimensional scene after three-dimensional reconstruction.

Furthermore, at least two camera devices of the image processing method of this embodiment are pinhole cameras;

Correspondingly, step S12 is:

Eliminating distortion in the corrected image data according to the corrected image data and pre-stored three-dimensional coordinate data of a predetermined marker, respectively, to obtain corrected image data after distortion is eliminated;

At least two shooting devices are calibrated based on the corrected image data after distortion is eliminated and the three-dimensional coordinate data of the pre-stored predetermined marker, and imaging association information of the at least two shooting devices for the same shooting scene is generated. The predetermined marker includes an identification pattern and/or an identification light mark, the identification pattern includes a one-dimensional barcode and/or a two-dimensional code, and the identification light includes fluorescence and/or an invisible infrared light source.

Compared with the prior art, the image processing method for three-dimensional reconstruction proposed in the present invention recognizes predetermined markers in the image and calibrates the camera group according to the coordinate reference system provided by the markers, thereby adaptively acquiring imaging correlation information of the camera group in different shooting stages, and actively compensating for factors such as fine-tuning the position of the camera group, thereby improving the accuracy of three-dimensional reconstruction and reproducing target scene images or videos with more realistic three-dimensional effects.

In a third aspect, the present invention provides an image processing method for analyzing the attention of a target object, characterized in that the method comprises the following steps:

Acquire image data captured by a front camera, wherein the image data includes image information of a predetermined marker and image information of a target object, and a relative position relationship between the predetermined marker and the target object remains unchanged;

Obtain the eye data of the subject captured by the eye camera;

Determine, based on the eye data, a focus area of the subject in the corresponding image data;

Determining a location area of a target object in the image data according to image information of a predetermined marker in the image data;

It is determined whether the position area falls within the focus area. If so, it is determined that the subject is paying attention to the target object; if not, it is determined that the subject is not paying attention to the target object.

The image processing method for analyzing the attention degree of a target object proposed in the present invention obtains the position information of the target object in the image by identifying a preset marker and based on the relatively fixed positional relationship between the preset marker and the target object, thereby tracking the position of the target object in different pictures or continuous videos, thereby determining whether the user pays attention to the image area corresponding to the position of the target object and evaluating the attention degree of the target object.

Compared with the prior art, the image processing method for analyzing the attention degree of a target object proposed in the present invention realizes rapid tracking of the target object with the help of a pre-set marker, and has simple processing and high recognition rate.

In a fourth aspect, the present invention provides an image processing method for ranging, the method comprising the following steps:

Acquire a first ranging image of a ranging target captured by a first front camera;

Acquire a second ranging image of the same ranging target captured by a second front camera;

Taking the line connecting the first front camera and the second front camera as the x-axis in the Cartesian coordinate system, according to the straight-line distance T between the first front camera and the second front camera, the focal length f of the first front camera, the focal length f of the second front camera, the first ranging image and the second ranging image, obtaining the x-axis coordinate xl of the intersection point of the line connecting the ranging target and the first front camera on the imaging plane of the first front camera, and obtaining the x-axis coordinate xr of the intersection point of the line connecting the ranging target and the second front camera on the imaging plane of the second front camera;

According to the following formula, the distance Z between the ranging target and the imaging plane of the first front camera is obtained:

Compared with the existing distance measurement technology, the image processing method for distance measurement proposed by the present invention performs distance measurement based on image information obtained by a camera device, and is simple, convenient, easy to operate, and has high precision.

BRIEF DESCRIPTION OF THE DRAWINGS

In order to more clearly illustrate the technical solutions in the specific embodiments of the present invention, the following is a brief introduction to the drawings required for the description of the specific embodiments. In all drawings, similar elements or parts are generally identified by similar reference numerals. In the drawings, each element or part is not necessarily drawn according to the actual scale.

FIG1 is a perspective view of an eye tracker according to an embodiment of the present invention;

FIG2 is a front view of an eye tracker according to an embodiment of the present invention;

FIG3 is an algorithm flow chart of an image processing method for three-dimensional reconstruction according to an embodiment of the present invention;

FIG4 is a schematic diagram of an application scenario of an image processing method for three-dimensional reconstruction according to an embodiment of the present invention;

5 is an algorithm flow chart of an image processing method for analyzing the attention level of a target object according to an embodiment of the present invention;

6 is a schematic diagram of an application scenario of an image processing method for analyzing the attention level of a target object according to an embodiment of the present invention;

7 is an algorithm flow chart of an image processing method for ranging according to an embodiment of the present invention;

FIG8 is a schematic diagram of calculation of an image processing method for ranging according to an embodiment of the present invention;

FIG. 9 is a schematic diagram of an application scenario of an image processing method for ranging according to an embodiment of the present invention.

DETAILED DESCRIPTION

The following embodiments of the technical solution of the present invention are described in detail in conjunction with the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solution of the present invention, and are therefore only used as examples, and cannot be used to limit the protection scope of the present invention.

Example 1

FIG1 is a stereoscopic view of an eye tracker 10 according to an embodiment of the present invention, and FIG2 is a front view of the eye tracker 10 according to an embodiment of the present invention. The eye tracker 10 includes a head-mounted body 1. The head-mounted body 1 is roughly annular, and can be an integrated structure or a split structure. The eye tracker 10 also includes a plurality of front cameras 21. The plurality of front cameras 21 are arranged on the head-mounted body 1 at intervals along the circumferential direction, and at least two of the plurality of front cameras 21 can simultaneously capture image information of a target scene in front of the head-mounted body 1. The image information includes video, images and/or pictures.

The eye tracker 10 also includes a control system (not shown in FIG. 1 and FIG. 2 ) disposed on the head-mounted body 1. The installation position of the control system is not particularly limited, and it can be installed at the front end, rear end, inside or surface of the head-mounted body 1. For the sake of aesthetics, the control system can be disposed inside the head-mounted body 1. Among them, the control system can include various types of computing processors such as a programmable logic controller PLC, ASIC, ARM core, memory, and electronic components connected to the computing processor, etc., which are well known to those skilled in the art and will not be described in detail here.

The control system is connected to the front camera 21, so that the control system can perform three-dimensional reconstruction of the target scene based on the image information, and/or perform distance measurement on at least one object in the target object based on the image information.

That is to say, the eye tracker 10 of the embodiment of the present invention can simultaneously collect image information of the target scene in front of the head-mounted body 1 through at least two front cameras 21, and reconstruct the target scene in three dimensions and/or measure the distance of at least one object in the target scene based on the image information, thereby promoting scholars' research in the fields of psychology, visual perception, reading, etc. At the same time, since the eye tracker 10 of the embodiment of the present invention has multiple front cameras 21, and they are spaced apart along the circumference of the head-mounted body 1, the eye tracker 10 has a wider collection range and field of view than the existing eye tracker with only one front camera. In addition, the eye tracker 10 of the embodiment of the present invention has a simple structure, is easy to use, and is comfortable, safe, and reliable to wear.

As shown in FIG2 , the eye tracker 10 further includes a nose pad 3 disposed at the front end of the head-mounted body 1. The nose pad 3 can be stuck on the user's nose, which better stabilizes the wearing of the eye tracker 10 and prevents the eye tracker 10 from slipping or shaking. Among the multiple front cameras 21, a part of the front cameras 21 is located on one side of the nose pad 3, while another part of the front cameras 21 is located on the other side of the nose pad 3. If all of the multiple front cameras 21 are located on one side of the nose pad 3, the camera range or acquisition range of the eye tracker 10 is too narrow, but when a part of the front cameras 21 is located on one side of the nose pad 3 and another part of the front cameras 21 is located on the other side of the nose pad 3, this can effectively expand the camera range or acquisition range of the eye tracker 10.

Preferably, the eye tracker 10 may include two front cameras 21 spaced apart along the circumference of the head-mounted body 1, the two front cameras 21 are respectively located on both sides of the nose pad 3, and the two front cameras 21 have a certain inclination angle relative to the center of the head-mounted body, such as 120 degrees.

According to an embodiment of the present invention, the eye tracker 10 further includes an eye camera 22 for collecting image information of the subject's eyes, and is used to collect image information of the user's eyes. The eye tracker 10 can record the eye movement trajectory characteristics of a person when processing visual information through a control system, a front camera 21, and an eye camera 22, so as to be used for research in the fields of attention, visual perception, and reading.

The eye camera 22 can be connected to the head-mounted body 1 through a support component 5. The support component 5 can be selected as a conventional support member such as a curved rod, a straight rod, a universal rod, etc., and the eye camera 22 is adjusted to correspond to the subject's eye, so as to facilitate the eye camera 22 to collect image information of the subject's eye. The eye camera 22 is connected to the control system, so that the control system can know whether the user is focused on the target object in the target scene based on the collected image information of the subject's eye and the collected image information of the target scene. In the target scene, the position of the target object relative to the pre-set marker is fixed, wherein the marker includes a marker pattern and/or a marker light.

In detail, the image information of the marker is used as pre-stored information and stored in the memory of the control system. When the control system detects that the image information of the front camera 21 is consistent with the pre-stored information, the control system can locate the position information of the marker in the current image. Since the position of the target object relative to the pre-set marker is fixed, the position information of the target object in the current image can be further located. By processing the image information of the subject's eyes collected by the eye camera 22, the area where the subject's eyes are focused in the current image can be known. By comparing the area where the subject's eyes are focused and the position information of the target object, it can be determined whether the user is focused on the target object in the target scene.

Preferably, the eye tracker 10 may include an eye camera 22 .

Preferably, the eye tracker 10 may include multiple eye cameras 22 .

Preferably, the identification pattern includes a one-dimensional barcode and/or a two-dimensional code, and the identification light includes a fluorescent light and/or an invisible infrared light source. Among them, the one-dimensional barcode includes EAN code, UPC code, 128 code, cross 25 code, 39 code and Codabar code, etc., and the two-dimensional code includes an array two-dimensional barcode, a matrix two-dimensional barcode, Microsoft Tag code, etc. Among them, the one-dimensional barcode and the two-dimensional barcode can be realized by any product that can display it or them, such as a spray picture, a display screen or an LED array.

In one embodiment, the connection between the front camera 21 and the head-mounted body 1 is a detachable connection. The detachable connection includes conventional detachable connections such as threaded connection, magnet adsorption, bolt connection or snap connection. On the one hand, this detachable manner can facilitate people to replace the front camera 21 with a new one when the front camera 21 is damaged. On the other hand, the front camera 21 can be replaced with a different type of camera according to specific needs, such as conversion between an infrared camera and an ordinary camera (i.e., a visible light camera). Among them, the front camera 21 includes a visible light camera, an infrared camera or a thermal imaging camera. The so-called visible light camera refers to a camera that can convert visible light into image information, such as a computer camera or a mobile phone camera.

In a preferred embodiment, the connection between the nose pad 3 and the head-mounted body 1 is preferably a detachable connection, so that the user can replace or clean the nose pad 3 by himself. The detachable connection includes a threaded connection, a magnet adsorption, a clamp connection, a clamp connection, or a plug-in connection in an interference fit manner. Since the facial features of users vary from person to person, a uniform nose pad 3 is difficult to fit all users, which may cause discomfort to individual or most users. As a preferred solution, there are multiple accessory solutions for the height and width of the nose pad to adapt to users of different genders and races and users wearing glasses. Especially for users who are accustomed to wearing glasses, they must take off their glasses to wear an eye tracker with a nose pad 3. If they must wear glasses, they also need to customize glasses of a specific shape, which brings great inconvenience. However, the nose pad 3 of the eye tracker of the present invention can be detached from the head-mounted body 1, and users can choose a nose pad 3 that suits them according to their facial features or whether they wear glasses. When the user wears glasses, he/she selects a special nose pad 3 of shorter length. The nose pad 3 has a groove design for the glasses. When wearing the glasses, the nose pad is placed on top of the special nose pad. The eye tracker using the special nose pad 3 is more convenient and quick to wear.

In a preferred embodiment, the eye tracker 10 further includes a signal transceiver (not shown). The signal transceiver is disposed on the head-mounted body 1 and is connected to the control system. The location of the signal transceiver is not particularly limited, and it can be installed at the front end, rear end, inside or surface of the head-mounted body 1. The signal transceiver is used to receive control signals from a predetermined external terminal (such as a mobile phone, a computer, and a remote controller, etc.) and/or transmit information to the external terminal. The signal transceiver can be a short-range communication module, such as a common Bluetooth module, a WiFi module (such as WiFi Direct) or an infrared module, etc. Of course, the signal transceiver can also be a long-range communication module, such as a common radio transceiver module or an Internet communication module, etc. The user can use the external terminal to send the control signal to the signal transceiver, and the signal transceiver can also transmit the received image information collected by the front camera 21 and the eye camera 22 to the external terminal, thereby increasing the interactivity of the eye tracker 10 of the embodiment of the present invention, and the image information collected by the eye tracker can also be used for other devices or purposes.

Example 2

As shown in FIG3 , the image processing method for three-dimensional reconstruction according to an embodiment of the present invention comprises the following steps:

Step S11: obtaining calibration image data synchronously photographed by at least two camera devices on a calibration three-dimensional scene of a predetermined marker;

Step S12: generating imaging association information of at least two shooting devices for the same shooting scene according to the corrected image data and the pre-stored three-dimensional coordinate data of the predetermined marker;

Step S13: Acquire collected image data of the target three-dimensional scene synchronously photographed by at least two calibrated photographing devices;

Step S14: synthesizing the collected image data into a three-dimensional stereo image according to the imaging association information, where the three-dimensional stereo image is the target three-dimensional scene after three-dimensional reconstruction.

It should be noted that the types and imaging parameters of the at least two camera devices may be the same or different.

Preferably, the at least two camera devices are pinhole cameras respectively; correspondingly, step S12 is:

Eliminating distortion in the corrected image data according to the corrected image data and pre-stored three-dimensional coordinate data of a predetermined marker, respectively, to obtain corrected image data after distortion is eliminated;

The at least two photographing devices are calibrated according to the corrected image data after distortion is eliminated and the three-dimensional coordinate data of the predetermined marker stored in advance, and imaging association information of the at least two photographing devices for the same photographing scene is generated.

Preferably, step S14 is: calculating the visual difference in the collected image data synchronously shot by two shooting devices according to the imaging association information, and synthesizing the collected image data into a three-dimensional stereo image according to the visual difference, and the three-dimensional stereo image is the target three-dimensional scene after three-dimensional reconstruction.

Preferably, the predetermined marker is marked by an identification pattern and/or identification light.

Preferably, the identification pattern includes a one-dimensional barcode and/or a two-dimensional code, and the identification light includes fluorescence and/or light with an intensity greater than that of sunlight. Among them, the one-dimensional barcode includes EAN code, UPC code, 128 code, cross 25 code, 39 code and Codabar code, etc., and the two-dimensional code includes an array two-dimensional barcode, a matrix two-dimensional barcode, Microsoft Tag code, etc. Among them, the one-dimensional barcode and the two-dimensional barcode can be realized by any product that can display it or them, such as a spray picture, a display screen or an LED array.

It should be understood that the labeling methods of the predetermined markers may be the same or different.

For a pinhole camera, the farther away from the center of the camera's photosensitive element, the worse the pixel quality, and the more distortion and noise there will be. For example, due to the imaging distortion of the pinhole camera at different positions, the image of a straight line in the pinhole camera is not a straight line. In order to improve the image quality, the image taken by the pinhole camera needs to be dedistorted.

A commonly used calibration method for pinhole cameras is to use a standard black and white chessboard to calculate the basic parameters of the camera, including the focal length f of the camera, the intrinsic parameter matrix, the extrinsic parameter matrix (such as the rotation R matrix and the translation T matrix), the distortion coefficient, etc. Specifically, in the black and white chessboard image captured by the pinhole camera, the chessboard will be distorted. For example, a straight line in reality will not be a straight line in the camera image. By detecting the position of the corner points on the black and white chessboard in the captured image, and then based on the fact that the corner points on the black and white chessboard should be straight lines, the basic parameters of the pinhole camera can be inferred.

For an eye tracker with two front cameras (i.e. binocular cameras), binocular camera calibration is required. Specifically, the two front cameras simultaneously capture images of the same calibration object, and the camera calibration is completed through the correspondence between the salient points of the calibration object in the two cameras. After obtaining the rotation matrix of the left front camera relative to the right front camera, the distortion elimination and correction of the binocular camera can be completed.

The image processing method for three-dimensional reconstruction in the embodiment of the present invention can reconstruct a three-dimensional scene synchronously photographed by at least two camera devices with high restoration degree.

FIG. 4 is a schematic diagram of an application scenario in which the image processing method for three-dimensional reconstruction according to an embodiment of the present invention is applied in the eye tracker of Example 1. FIG.

The subject 305 wears an eye tracker 10 with two front cameras, and the two calibrated front cameras are controlled by the subject or by other people to synchronously shoot the target scene 301 (A), and obtain pictures 302 (A1) and 303 (A2) respectively. The control system in the eye tracker 10 implements the image processing method for three-dimensional reconstruction of the embodiment of the present invention, reconstructs the pictures 302 (A1) and 303 (A2) synchronously shot by the two front cameras with high restoration, and reproduces the target scene picture 304 (A′) with a three-dimensional effect.

The processing of video images is similar to that of pictures, and will not be described in detail here.

It should be understood that the eye tracker can also be worn by a dummy or fixed on a mounting frame to obtain a target scene image or video with a three-dimensional effect.

Preferably, an eye tracker provided with multiple front cameras obtains the original view of the scene from the multiple front cameras, and then eliminates distortion by pinhole camera calibration technology. Then, the salient points in the original view of the scene are detected by machine vision, and the multiple front cameras are calibrated according to the positional relationship between the views of the salient points in different cameras.

It should be noted that the salient point may be the aforementioned predetermined marker, an infrared light source, or any object that is not smooth.

The multiple front cameras can be considered as a camera group. Then, the calibrated camera group is used to synthesize a 3D stereogram by calculating the visual differences in the views captured by each front camera, thereby completing the 3D reconstruction.

The image processing method for three-dimensional reconstruction in the embodiment of the present invention recognizes predetermined markers in the image and calibrates the camera group according to the coordinate reference system provided by the marker, thereby adaptively acquiring imaging correlation information of the camera group in different shooting stages, and actively compensating for factors such as fine-tuning the position of the camera group, thereby improving the accuracy of three-dimensional reconstruction and reproducing a target scene image or video with a more realistic three-dimensional effect.

Example 3

As shown in FIG5 , the image processing method for analyzing the attention degree of a target object according to an embodiment of the present invention comprises the following steps:

Step S21: acquiring image data captured by a front camera, wherein the image data includes image information of a predetermined marker and image information of a target object, and a relative positional relationship between the predetermined marker and the target object remains unchanged;

Step S22: obtaining the eye data of the subject photographed by the eye camera;

Step S23: determining the focus area of the subject in the corresponding image data according to the eye data;

Step S24: determining a location area of the target object in the image data according to the image information of the predetermined marker in the image data;

Step S25: determining whether the position area falls within the focus area; if so, determining that the subject is focusing on the target object; if not, determining that the subject is not focusing on the target object.

Preferably, the predetermined marker includes a logo pattern and/or a logo light. The logo pattern includes a one-dimensional barcode and/or a two-dimensional code, and the logo light includes fluorescence and/or light with an intensity greater than the intensity of daylight and/or an invisible infrared light source. In addition, the above-mentioned predetermined marker may be one, multiple, or a component of multiple predetermined markers. Among them, the one-dimensional barcode includes EAN code, UPC code, 128 code, cross 25 code, 39 code and Codabar code, etc., and the two-dimensional code includes an array two-dimensional barcode, a matrix two-dimensional barcode, Microsoft Tag code, etc. Among them, both the one-dimensional barcode and the two-dimensional barcode can be realized by any product that can display it or them, such as a spray picture, a display screen or an LED array.

In any visual research, such as user experience research or advertising effectiveness evaluation, it is necessary to know the degree of user attention to the target object. For example, when a company puts a short advertising video on its image in various media, it is necessary to evaluate the degree of user attention to the company's trademark or name. If the user's attention to the company's trademark or name is lower than expected, the image design of the short advertising video needs to be adjusted to improve the advertising effect and increase advertising revenue.

Another example is that it is necessary to count the audience's attention to the embedded advertisements in film and television dramas in order to evaluate the advertising effect. The above-mentioned corporate trademark or corporate name or embedded advertisement content is the target object that the subjects are expected to pay attention to in the picture.

Currently, the method for determining the target object in a frame of an image is usually to manually define the target area. Given that the position of the target object in different frames of an image changes frequently, manually defining the target object requires the user to manually select or box the area in the image frame by frame to define the target object, which is time-consuming and labor-intensive. In addition, the massive workload means that larger studies must use the method of manually marking videos.

Specifically, determining the focus area of the subject in the corresponding image data according to the eye data may include the following steps:

Perform black and white thresholding, face detection, eye detection, and pupil detection on the images in the eye data to obtain the position where the eyes are looking;

The position where the subject's eyes are looking is marked in the image of the test image currently viewed by the subject taken by the front camera device, so as to know the specific part of the object in the real world that the eyes have seen.

In specific applications, after setting a predetermined marker near a predetermined target object, a video or picture is recorded by a camera device to obtain image data. This image data will be used for user experience research, such as advertising clips and psychology experiment pictures.

FIG6 is a schematic diagram of an application scenario in which the image processing method for determining the attention degree of a target object according to an embodiment of the present invention is applied in the eye tracker of Example 1. FIG.

After the subject 205 wears the eye tracker 10 equipped with at least one front camera and at least one eye camera, he/she watches pre-recorded video, pictures, web pages, text and other display materials. At least one front camera on the eye tracker synchronously shoots the image that the subject is watching to obtain scene image data; at least one eye camera on the eye tracker 10 synchronously shoots the movement of the subject's eyeball to obtain eye data; the control system in the eye tracker 10 implements the image processing method for analyzing the degree of attention of the target object according to the embodiment of the present invention, determines the focus area of the subject in the current viewing image according to the eye data, determines the target position of the target object according to the position of the predetermined marker in the image data; determines whether the target position falls into the focus area, if so, it is determined that the subject pays attention to the target object; if not, it is determined that the subject does not pay attention to the target object.

Specifically, in conjunction with FIG6 , 201 in the diagram is the area where the predetermined target object is located, and 202 in the diagram is the area where the predetermined marker is located. When the subject is viewing picture C, through the image processing method for determining the degree of attention of the target object according to the embodiment of the present invention, it can be determined that the focus area of the subject's eyes falls on the area where the apple is located in picture C, rather than 202 where the predetermined marker is located or 201 where the predetermined target object is located. Therefore, it can be determined that the subject is not paying attention to the target object. When the subject is viewing picture D, through the image processing method for determining the degree of attention of the target object according to the embodiment of the present invention, it can be determined that the focus area of the subject's eyes falls on 201 where the predetermined target object is located in picture D, rather than 202 where the predetermined marker is located or the area where the fruit grapes are located. Therefore, it can be determined that the subject is paying attention to the target object.

It should be noted that multiple interesting target objects can be set on the same image or video material. In this case, the multiple interesting target objects can be grouped into an area of interest group. The shape of the area of interest can be square, circle, ellipse, triangle, or a custom unfixed shape.

Based on whether the subject focuses on the target object, we can further count many variables such as the cumulative number of times the subject focuses on the target object in a video/picture area, or the cumulative time the subject focuses on the target object in a video/picture area, or the time from the presentation of the picture/video to the first focus, and the cumulative focus time in a video.

According to an embodiment of the present invention, an image processing method for analyzing the attention degree of a target object recognizes a preset marker and obtains the position information of the target object in the image based on the fixed relative position relationship between the preset marker and the target object, thereby tracking the position of the target object in different pictures or continuous videos, thereby determining whether the user pays attention to the image area corresponding to the location of the target object and evaluating the attention degree of the target object.

Compared with the prior art, the image processing method for analyzing the attention degree of a target object according to an embodiment of the present invention realizes rapid tracking of the target object with the help of pre-set markers, simple processing and high recognition rate.

The control system in the eye tracker implements the image processing method for three-dimensional reconstruction of the embodiment of the present invention, so that the eye tracker can automatically identify markers and use codes to automatically calculate the interaction between the eye gaze position and the target object of interest, thereby getting rid of the traditional time-consuming and labor-intensive manual marking work.

Example 4

As shown in FIG. 7 , the image processing method for ranging according to an embodiment of the present invention includes the following steps:

Step S31: acquiring a first ranging image of a ranging target taken by a first front camera; and acquiring a second ranging image of the same ranging target taken by a second front camera;

Step S32: Taking the line connecting the first front camera and the second front camera as the x-axis in the Cartesian coordinate system, according to the straight-line distance T between the first front camera and the second front camera, the focal length f of the first front camera, the focal length f of the second front camera, and the first ranging image and the second ranging image, obtaining the x-axis coordinate xl of the intersection point of the line connecting the ranging target and the first front camera and the imaging plane of the first front camera, and obtaining the x-axis coordinate xr of the intersection point of the line connecting the ranging target and the second front camera and the imaging plane of the second front camera;

Step S33: Obtain the distance Z between the ranging target and the imaging plane of the first front camera according to the following formula:

Specifically, as shown in Figure 8, the object to be measured is equivalent to a point in space, denoted as P, and the left and right cameras are equivalent to points in space, denoted as Ol and Or respectively (perpendicular to the paper, the left side is the left camera, and the right side is the right camera). The intersection of the line connecting P and Ol and the imaging plane is denoted as Pl, and the intersection of the line connecting P and Or and the imaging plane is denoted as Pr. The triangle formed by the lines connecting the three points P, Pl and Pr and the triangle formed by the lines connecting the three points P, Ol and Or are similar triangles. Z is the distance from the imaging plane of the camera to be measured to the object to be measured. f is the focal length of the camera, and T is the straight-line distance between the two cameras. According to the principle of similar triangles, Z can be calculated.

FIG. 9 is a schematic diagram of an application scenario in which the image processing method for distance measurement according to an embodiment of the present invention is applied in the eye tracker of Example 1. FIG.

After the user wears the eye tracker with two front cameras, the two front cameras are controlled to synchronously shoot the ranging image of the ranging target. The control system in the eye tracker implements the image processing method for ranging according to the embodiment of the present invention, processes the ranging images of the ranging target synchronously shot by the two front cameras, obtains the distance between the ranging target and the imaging planes of the two front cameras, and obtains the straight-line distance between the ranging target and the user in combination with the focal length of the front camera.

Compared with the existing distance measurement technology, the image processing method for distance measurement according to the embodiment of the present invention performs distance measurement based on image information acquired by a camera device, which is simple, convenient, easy to operate, and has high precision.

Although the present invention has been described with reference to preferred embodiments, various modifications may be made thereto and parts thereof may be replaced by equivalents without departing from the scope of the present invention. In particular, the various technical features mentioned in the various embodiments may be combined in any manner as long as there are no structural conflicts. The present invention is not limited to the specific embodiments disclosed herein, but includes all technical solutions falling within the scope of the claims.

Claims (9)

1. An eye movement apparatus, comprising:

a head-mounted main body;

The front cameras are arranged on the head-mounted main body at intervals along the circumferential direction, and at least two of the front cameras can collect image information of a target scene in front of the head-mounted main body at the same time;

The control system is arranged on the head-mounted main body and connected with the front-mounted camera, so that the control system can reconstruct the target scene in three dimensions based on the image information and/or measure the distance of at least one object in the target scene based on the image information;

The eye camera is connected with the head-mounted main body through a supporting component, the eye camera is connected with a control system, so that the control system can know whether a user focuses on a target object in a target scene or not based on the collected image information of the eyes of the tested person and the collected image information of the target scene, the position of the target object in the target scene relative to a preset marker is fixed, the marker comprises a marker pattern and/or marker light, the marker pattern comprises a one-dimensional bar code and/or a two-dimensional bar code, and the marker light comprises fluorescent and/or invisible infrared light sources.

2. The eye-piece according to claim 1, further comprising a nose pad provided at a front end of the head-mounted body, wherein among the plurality of front cameras, a part of the front cameras is located at one side of the nose pad and another part of the front cameras is located at the other side of the nose pad.

3. The eye-piece according to claim 2, wherein the connection between the nose pad and the head-mounted body is a detachable connection.

4. The eye-piece of claim 1, wherein the connection between the front camera and the head-mounted body is a detachable connection.

5. The eye-piece of claim 1, wherein the front camera comprises a visible light camera, an infrared camera, or a thermal imaging camera.

6. An image processing method for three-dimensional reconstruction using the eye tracker according to any one of claims 1 to 5, comprising the steps of:

step S11: acquiring correction image data synchronously shot by the three-dimensional scene for correcting the preset marker by at least two camera devices;

Step S12: generating imaging related information of the at least two shooting devices aiming at the same shooting scene according to the corrected image data and the pre-stored three-dimensional coordinate data of the preset marker;

step S13: acquiring corrected acquired image data of the at least two shooting devices synchronously shooting the target three-dimensional scene;

Step S14: and synthesizing the acquired image data into a three-dimensional stereoscopic image according to the imaging associated information, wherein the three-dimensional stereoscopic image is a three-dimensional reconstructed target three-dimensional scene.

7. The image processing method according to claim 6, wherein the at least two image pickup devices are pinhole cameras;

Correspondingly, step S12 is:

Eliminating distortion in the corrected image data according to the corrected image data and the pre-stored three-dimensional coordinate data of the predetermined marker respectively, and obtaining corrected image data after eliminating distortion;

Correcting the at least two shooting devices according to the corrected image data after distortion elimination and the three-dimensional coordinate data of a pre-stored predetermined marker, and generating imaging associated information of the at least two shooting devices aiming at the same shooting scene, wherein the predetermined marker comprises an identification pattern and/or an identification light marker, the identification pattern comprises a one-dimensional bar code and/or a two-dimensional bar code, and the identification light comprises fluorescence and/or invisible infrared light sources.

8. An image processing method for ranging using the eye tracker according to any one of claims 1 to 5, comprising the steps of:

acquiring a first ranging image of a ranging target shot by a first front-facing camera;

acquiring a second ranging image of the same ranging target shot by a second front-facing camera;

Taking a connecting line of the first front-end camera and the second front-end camera as an x-axis in a Cartesian coordinate system, and obtaining an x-axis coordinate xl of a connecting line of the ranging target and the first front-end camera at an intersection point of an imaging plane of the first front-end camera according to a linear distance T of the first front-end camera and the second front-end camera and a focal length f of the first front-end camera, the focal length f of the second front-end camera and the first ranging image and the second ranging image;

Obtaining a distance Z between the ranging target and an imaging plane of the first front-facing camera according to the following steps:

9. An image processing method for analyzing a target object's attention by using the eye tracker according to any one of claims 1 to 5, comprising the steps of:

Acquiring image data shot by a front camera, wherein the image data comprises image information of a preset marker and image information of a target object, and the relative position relationship between the preset marker and the target object is fixed;

Acquiring eye data of a tested person shot by an eye camera;

Determining a focusing area of the tested person in the corresponding image data according to the eye data;

Determining a position area of the target object in the image data according to the image information of the predetermined marker in the image data;

judging whether the position area falls into the focusing area, if so, determining that the tested person pays attention to the target object; if the target object does not fall into the target object, the tested person is determined to not pay attention to the target object.