CN103500061B - Control the method and apparatus of display - Google Patents

Abstract

A method and device for controlling a display. The method includes: capturing human eyes through electronic glasses to obtain human eye images; extracting human eye feature information from the human eye images; detecting the glasses position and attitude information of the electronic glasses; detecting the screen according to the human eye feature information, glasses position and attitude information eye gaze point on the screen; detect human eye movement according to human eye feature information and/or changes in human eye gaze point on the screen; control the operation of the display according to human eye movement and human eye gaze point on the screen. According to the present invention, the display can be controlled according to the human eye image.

Description

Method and device for controlling a display

technical field

The present invention relates to the field of display control, more specifically, to a method and device for controlling a display.

Background technique

In people's life, monitors have become an indispensable part. Whether it is a TV monitor or a computer monitor, users increasingly hope to use a more convenient control method to control the monitor.

Under the existing technical conditions, users usually use handheld remote control devices or buttons on the display panel to control the display, which makes the operation process inseparable from the participation of hands, and people cannot free their hands from the control device, especially for both hands It is even more inconvenient for people with disabilities.

Therefore, there is a need for a method and device for controlling a display, so that the display can be controlled without human hands.

Contents of the invention

The object of the present invention is to provide a method and device for controlling a display, which can control the display according to the images of human eyes.

The first aspect of the present invention provides a method for controlling a display, including: capturing human eyes through electronic glasses to obtain human eye images; extracting human eye feature information from the human eye images; detecting the glasses position and posture information of the electronic glasses; Eye feature information, glasses position and posture information to detect the gaze point of the human eye on the screen; detect the human eye movement according to the human eye feature information and/or the change of the human eye gaze point on the screen; The eye gaze point controls the operation of the display.

Optionally, the step of detecting the gaze point of the human eye on the screen according to the characteristic information of the human eye, the position of the glasses, and the posture information includes: obtaining the gaze point of the human eye on the electronic glasses according to the characteristic information of the human eye; The gaze point of the human eye on the screen is obtained by using point, eyeglass position and posture information.

Optionally, the human eye characteristic information is the position information of the pupil center and the corneal reflection point, and the step of obtaining the gaze point of the human eye on the electronic glasses according to the human eye characteristic information includes: calculating according to the position information of the pupil center and the corneal reflection point The gaze point of the human eye on the electronic glasses.

Optionally, the human eye action includes at least one of eye closing, gaze, saccade, and smooth movement.

Optionally, when the corneal reflection point cannot be obtained from the characteristic information of the human eye within the first predetermined time, the human eye movement is detected as eyes closed; when the gaze point of the human eye on the screen remains unchanged within the second predetermined time, The human eye movement is detected as gazing; when the human eye gaze point on the screen moves beyond a predetermined threshold within the third predetermined time, the human eye movement is detected as saccade; when the human eye gaze point on the screen moves at a predetermined speed, Human eye movements are detected as smooth movements.

Optionally, the operation of the display is controlled in different modes according to human eye movements and human gaze points on the screen.

Optionally, the step of controlling the operation of the display according to the human eye movement and the human eye gaze point on the screen includes: when the screen displays an image, controlling the display to perform the following operations according to the human eye movement and the human eye gaze point on the screen: When the eye movement is gaze, and the gaze point of the human eye on the screen is located in the middle area of the screen, the image near the gaze point of the human eye on the screen is enlarged and displayed; when the image displayed on the screen is a moving image, when the human eye When the eye movement is smooth tracking, the object tracked by the human eye is determined according to the gaze point of the human eye on the screen during the smooth tracking process, and the object tracked by the human eye is displayed in the center of the screen; when the human eye movement is gaze, and When the gaze point of the human eye on the screen is located in the border area of the screen, move the image in a direction away from the border.

Optionally, the step of controlling the operation of the display according to the human eye movement and the human eye gaze point on the screen includes: when the screen displays the user interface, controlling the display to perform the following operations according to the human eye movement and the human eye gaze point on the screen: The human eye gaze point on the screen is located in a functional area, and then the human eye action is to close the eyes, and the function corresponding to the functional area is run; when the human eye action is a smooth movement, and the human eye gaze point on the screen changes from a functional area to When moving to another functional area, switch the focus to that other area.

Optionally, the method for controlling a display further includes: detecting a viewing angle when viewing a screen according to human eye feature information, glasses position and posture information; and adjusting the depth of the screen according to the viewing angle.

Optionally, the step of adjusting the depth of the screen according to the viewing angle includes: when the viewing angle exceeds a predetermined angle threshold, reducing the depth of an area on the screen close to the electronic glasses, and adjusting the depth of an area on the screen far away from the electronic glasses, Make the viewing angle less than or equal to a predetermined angle threshold.

The second aspect of the present invention provides a device for controlling a display, including: a human eye image acquisition unit, which captures a human eye image through electronic glasses; a feature extraction unit, extracts human eye feature information from the human eye image; position and posture The detection unit detects the position and posture information of the glasses of the electronic glasses; the gaze point detection unit detects the gaze point of the human eye on the screen according to the characteristic information of the human eye, the position and posture information of the glasses; the human eye movement detection unit detects the gaze point of the human eye according to the characteristic information of the human eye and the /or detecting changes in human eye gaze points on the screen; the control unit controls the operation of the display according to the human eye movements and the human eye gaze points on the screen.

Optionally, the gaze point detection unit includes: a glasses gaze point detection unit, which obtains the gaze point of the human eye on the electronic glasses according to the human eye feature information; a screen gaze point detection unit, which obtains the gaze point of the human eye on the electronic glasses, the position of the glasses and The gesture information obtains the gaze point of the human eyes on the screen.

Optionally, the characteristic information of the human eye is the position information of the pupil center and the corneal reflection point, and the gaze point detection unit of the glasses includes: a calculation unit, which calculates the position information of the human eye line of sight on the electronic glasses according to the position information of the pupil center and the corneal reflection point point of human gaze.

Optionally, the human eye motion detection unit is used to detect at least one human eye motion among eye closing, gaze, saccade, and smooth movement.

Optionally, when the corneal reflection point cannot be obtained from the characteristic information of the human eye within the first predetermined time, the human eye movement is detected as eyes closed; when the gaze point of the human eye on the screen remains unchanged within the second predetermined time, The human eye movement is detected as gazing; when the human eye gaze point on the screen moves beyond a predetermined threshold within the third predetermined time, the human eye movement is detected as saccade; when the human eye gaze point on the screen moves at a predetermined speed, Human eye movements are detected as smooth movements.

Optionally, the control unit controls the operation of the display in different modes according to human eye movements and human gaze points on the screen.

Optionally, when the screen displays an image, the control unit controls the display to perform the following operations according to the human eye movement and the human gaze point on the screen: when the human eye movement is gazing, and the human eye gaze point on the screen is located in the middle area of the screen , to enlarge and display the image near the gaze point of the human eye on the screen; in the case that the image displayed on the screen is a moving image, when the human eye movement is smooth tracking, according to the person on the screen during the smooth tracking process The eye fixation point determines the object tracked by the human eye, and displays the object tracked by the human eye in the center of the screen; when the human eye action is gazing, and the human eye gaze point on the screen is located in the border area of the screen, the image is oriented away from this The direction of the boundary moves.

Optionally, when the user interface is displayed on the screen, the control unit controls the display to perform the following operations according to the human eye movement and the human gaze point on the screen: when the human eye gaze point on the screen is located in a functional area, then the human eye movement is closed When the eye is on, run the function corresponding to the functional area; when the human eye moves smoothly and the human gaze point on the screen moves from one functional area to another functional area, switch the focus to the other area.

Optionally, the device for controlling the display further includes: a viewing angle detection unit, which detects the viewing angle when viewing the screen according to human eye feature information, glasses position and posture information; a depth adjustment unit, which adjusts the depth of the screen according to the viewing angle.

Optionally, when the viewing angle exceeds the predetermined angle threshold, the depth adjustment unit reduces the depth of the area on the screen close to the electronic glasses, and adjusts the depth of the area on the screen far away from the electronic glasses, so that the viewing angle is less than or equal to the predetermined angle threshold .

The third aspect of the present invention provides a method for using electronic glasses to remotely control a display, including: capturing human eyes through electronic glasses to obtain human eye images; extracting human eye feature information from the human eye images; detecting the position and posture information of the electronic glasses ; Send the feature information of the captured human eye image and the glasses position and posture information of the electronic glasses to the display.

The fourth aspect of the present invention provides a device for using electronic glasses to remotely control a display, including: a human eye image acquisition unit, which obtains a human eye image by photographing the human eye; a feature extraction unit, which extracts human eye feature information from the human eye image; The posture detection unit detects the glasses position and posture information of the electronic glasses; the sending unit sends the feature information of the captured human eye image and the glasses position and posture information of the electronic glasses to the display.

A fifth aspect of the present invention provides a method for controlling a display, including: receiving from the electronic glasses the feature information of the captured human eye image and the glasses position and attitude information of the electronic glasses; The gaze point of the human eye on the screen; detect the movement of the human eye according to the characteristic information of the human eye and/or the change of the gaze point of the human eye on the screen; control the operation of the display according to the movement of the human eye and the gaze point of the human eye on the screen.

Optionally, the step of detecting the gaze point of the human eye on the screen according to the characteristic information of the human eye, the position of the glasses, and the posture information includes: obtaining the gaze point of the human eye on the electronic glasses according to the characteristic information of the human eye; The gaze point of the human eye on the screen is obtained by using point, eyeglass position and posture information.

Optionally, the human eye characteristic information is the position information of the pupil center and the corneal reflection point, and the step of obtaining the gaze point of the human eye on the electronic glasses according to the human eye characteristic information includes: calculating according to the position information of the pupil center and the corneal reflection point The gaze point of the human eye on the electronic glasses.

Optionally, the human eye action includes at least one of eye closing, gaze, saccade, and smooth movement.

Optionally, when the corneal reflection point cannot be obtained from the characteristic information of the human eye within the first predetermined time, the human eye movement is detected as eyes closed; when the gaze point of the human eye on the screen remains unchanged within the second predetermined time, The human eye movement is detected as gazing; when the human eye gaze point on the screen moves beyond a predetermined threshold within the third predetermined time, the human eye movement is detected as saccade; when the human eye gaze point on the screen moves at a predetermined speed, Human eye movements are detected as smooth movements.

Optionally, the operation of the display is controlled in different modes according to human eye movements and human gaze points on the screen.

Optionally, the step of controlling the operation of the display according to the human eye movement and the human eye gaze point on the screen includes: when the screen displays an image, controlling the display to perform the following operations according to the human eye movement and the human eye gaze point on the screen: When the eye movement is gaze, and the gaze point of the human eye on the screen is located in the middle area of the screen, the image near the gaze point of the human eye on the screen is enlarged and displayed; when the image displayed on the screen is a moving image, when the human eye When the eye movement is smooth tracking, the object tracked by the human eye is determined according to the gaze point of the human eye on the screen during the smooth tracking process, and the object tracked by the human eye is displayed in the center of the screen; when the human eye movement is gaze, and When the gaze point of the human eye on the screen is located in the border area of the screen, move the image in a direction away from the border.

Optionally, the step of controlling the operation of the display according to the human eye movement and the human eye gaze point on the screen includes: when the screen displays the user interface, controlling the display to perform the following operations according to the human eye movement and the human eye gaze point on the screen: The human eye gaze point on the screen is located in a functional area, and then the human eye action is to close the eyes, and the function corresponding to the functional area is run; when the human eye action is a smooth movement, and the human eye gaze point on the screen changes from a functional area to When moving to another functional area, switch the focus to that other area.

Optionally, the method for controlling a display further includes: detecting a viewing angle when viewing a screen according to human eye feature information, glasses position and posture information; and adjusting the depth of the screen according to the viewing angle.

Optionally, the step of adjusting the depth of the screen according to the viewing angle includes: when the viewing angle exceeds a predetermined angle threshold, the depth adjusting unit reduces the depth of the area on the screen close to the electronic glasses, and reduces the depth of the area on the screen far away from the electronic glasses. Turn it up so that the viewing angle is less than or equal to the preset angle threshold.

A sixth aspect of the present invention provides a device for controlling a display, including: a receiving unit, which receives the feature information of the human eye image taken from the electronic glasses and the glasses position and posture information of the electronic glasses; a gaze point detection unit, according to the characteristics of the human eyes information, glasses position and attitude information to detect the gaze point of the human eye on the screen; the human eye movement detection unit detects the human eye movement according to the characteristic information of the human eye and/or the change of the gaze point of the human eye on the screen; the control unit detects the human eye movement according to the Eye movements and the point of human eye gaze on the screen control the operation of the display.

Optionally, the gaze point detection unit includes: a glasses gaze point detection unit, which obtains the gaze point of the human eye on the electronic glasses according to the human eye feature information; a screen gaze point detection unit, which obtains the gaze point of the human eye on the electronic glasses, the position of the glasses and The gesture information obtains the gaze point of the human eyes on the screen.

Optionally, the characteristic information of the human eye is the position information of the pupil center and the corneal reflection point, and the gaze point detection unit of the glasses includes: a calculation unit, which calculates the position information of the human eye line of sight on the electronic glasses according to the position information of the pupil center and the corneal reflection point point of human gaze.

Optionally, the human eye motion detection unit is used to detect at least one human eye motion among eye closing, gaze, saccade, and smooth movement.

Optionally, when the corneal reflection point cannot be obtained from the characteristic information of the human eye within the first predetermined time, the human eye movement is detected as eyes closed; when the gaze point of the human eye on the screen remains unchanged within the second predetermined time, The human eye movement is detected as gazing; when the human eye gaze point on the screen moves beyond a predetermined threshold within the third predetermined time, the human eye movement is detected as saccade; when the human eye gaze point on the screen moves at a predetermined speed, Human eye movements are detected as smooth movements.

Optionally, the control unit controls the operation of the display in different modes according to human eye movements and human gaze points on the screen.

Optionally, when the screen displays an image, the control unit controls the display to perform the following operations according to the human eye movement and the human gaze point on the screen: when the human eye movement is gazing, and the human eye gaze point on the screen is located in the middle area of the screen , to enlarge and display the image near the gaze point of the human eye on the screen; in the case that the image displayed on the screen is a moving image, when the human eye movement is smooth tracking, according to the person on the screen during the smooth tracking process The eye fixation point determines the object tracked by the human eye, and displays the object tracked by the human eye in the center of the screen; when the human eye action is gazing, and the human eye gaze point on the screen is located in the border area of the screen, the image is oriented away from this The direction of the boundary moves.

Optionally, when the user interface is displayed on the screen, the control unit controls the display to perform the following operations according to the human eye movement and the human gaze point on the screen: when the human eye gaze point on the screen is located in a functional area, then the human eye movement is closed When the eye is on, run the function corresponding to the functional area; when the human eye moves smoothly and the human gaze point on the screen moves from one functional area to another functional area, switch the focus to the other area.

Optionally, the device for controlling the display further includes: a viewing angle detection unit, which detects the viewing angle when viewing the screen according to human eye feature information, glasses position and posture information; a depth adjustment unit, which adjusts the depth of the screen according to the viewing angle.

Optionally, when the viewing angle exceeds the predetermined angle threshold, the depth adjustment unit reduces the depth of the area on the screen close to the electronic glasses, and increases the depth of the area on the screen far away from the electronic glasses, so that the viewing angle is less than or equal to the predetermined angle threshold .

The seventh aspect of the present invention provides a method for using electronic glasses to remotely control a display, including: capturing human eyes through electronic glasses to obtain a human eye image; extracting human eye feature information from the human eye image; detecting the position and posture information of the electronic glasses ; Detect the gaze point of the human eye on the screen according to the characteristic information of the human eye, the position and posture information of the glasses; detect the movement of the human eye according to the characteristic information of the human eye and/or the change of the gaze point of the human eye on the screen; The human eye movement of the operation and the human eye gaze point on the screen are sent to the display.

Optionally, the step of detecting the gaze point of the human eye on the screen according to the characteristic information of the human eye, the position of the glasses, and the posture information includes: obtaining the gaze point of the human eye on the electronic glasses according to the characteristic information of the human eye; The gaze point of the human eye on the screen is obtained by using point, eyeglass position and posture information.

Optionally, the human eye characteristic information is the position information of the pupil center and the corneal reflection point, and the step of obtaining the gaze point of the human eye on the electronic glasses according to the human eye characteristic information includes: calculating according to the position information of the pupil center and the corneal reflection point The gaze point of the human eye on the electronic glasses.

Optionally, the human eye action includes at least one of eye closing, gaze, saccade, and smooth movement.

Optionally, when the corneal reflection point cannot be obtained from the characteristic information of the human eye within the first predetermined time, the human eye movement is detected as eyes closed; when the gaze point of the human eye on the screen remains unchanged within the second predetermined time, The human eye movement is detected as gazing; when the human eye gaze point on the screen moves beyond a predetermined threshold within the third predetermined time, the human eye movement is detected as saccade; when the human eye gaze point on the screen moves at a predetermined speed, Human eye movements are detected as smooth movements.

The eighth aspect of the present invention provides a device for using electronic glasses to remotely control a display, including: a human eye image acquisition unit, which obtains a human eye image by photographing the human eye; a feature extraction unit, which extracts human eye feature information from the human eye image; The posture detection unit detects the position and posture information of the glasses of the electronic glasses; the fixation point detection unit detects the gaze point of the human eye on the screen according to the characteristic information of the human eye, the position of the glasses and the posture information; the human eye movement detection unit detects the gaze point of the human eye according to the characteristic information of the human eye and/or human eye gaze changes on the screen to detect human eye movements; the sending unit sends the human eye movements for controlling the operation of the display and the human eye gaze points on the screen to the display.

Optionally, the gaze point detection unit includes: a glasses gaze point detection unit, which obtains the gaze point of the human eye on the electronic glasses according to the human eye feature information; a screen gaze point detection unit, which obtains the gaze point of the human eye on the electronic glasses, the position of the glasses and The gesture information obtains the gaze point of the human eyes on the screen.

Optionally, the characteristic information of the human eye is the position information of the pupil center and the corneal reflection point, and the gaze point detection unit of the glasses includes: a calculation unit, which calculates the position information of the human eye line of sight on the electronic glasses according to the position information of the pupil center and the corneal reflection point point of human gaze.

Optionally, the fixation point detection unit is used to detect at least one human eye action among eye closing, gaze, saccade, and smooth movement.

Optionally, when the corneal reflection point cannot be obtained from the characteristic information of the human eye within the first predetermined time, the human eye movement is detected as eyes closed; when the gaze point of the human eye on the screen remains unchanged within the second predetermined time, The human eye movement is detected as gazing; when the human eye gaze point on the screen moves beyond a predetermined threshold within the third predetermined time, the human eye movement is detected as saccade; when the human eye gaze point on the screen moves at a predetermined speed, Human eye movements are detected as smooth movements.

A ninth aspect of the present invention provides a method for controlling a display, including: receiving from electronic glasses the human eye movement for controlling the operation of the display and the human eye gaze point on the screen; controlling the display according to the human eye movement and the human eye gaze point on the screen operation of the display.

Optionally, the operation of the display is controlled in different modes according to human eye movements and human gaze points on the screen.

Optionally, the step of controlling the operation of the display according to the human eye movement and the human eye gaze point on the screen includes: when the screen displays an image, controlling the display to perform the following operations according to the human eye movement and the human eye gaze point on the screen: When the eye movement is gaze, and the gaze point of the human eye on the screen is located in the middle area of the screen, the image near the gaze point of the human eye on the screen is enlarged and displayed; when the image displayed on the screen is a moving image, when the human eye When the eye movement is smooth tracking, the object tracked by the human eye is determined according to the gaze point of the human eye on the screen during the smooth tracking process, and the object tracked by the human eye is displayed in the center of the screen; when the human eye movement is gaze, and When the gaze point of the human eye on the screen is located in the border area of the screen, move the image in a direction away from the border.

Optionally, the step of controlling the operation of the display according to the human eye movement and the human eye gaze point on the screen includes: when the screen displays the user interface, controlling the display to perform the following operations according to the human eye movement and the human eye gaze point on the screen: The human eye gaze point on the screen is located in a functional area, and then the human eye action is to close the eyes, and the function corresponding to the functional area is run; when the human eye action is a smooth movement, and the human eye gaze point on the screen changes from a functional area to When moving to another functional area, switch the focus to that other area.

The tenth aspect of the present invention provides a device for controlling a display, including: a receiving unit, which receives from electronic glasses the human eye movement for controlling the operation of the display and the human eye gaze point on the screen; The gaze point of the human eye controls the operation of the display.

Optionally, the control unit controls the operation of the display in different modes according to human eye movements and human gaze points on the screen.

Optionally, when the screen displays an image, the control unit controls the display to perform the following operations according to the human eye movement and the human gaze point on the screen: when the human eye movement is gazing, and the human eye gaze point on the screen is located in the middle area of the screen , to enlarge and display the image near the gaze point of the human eye on the screen; in the case that the image displayed on the screen is a moving image, when the human eye movement is smooth tracking, according to the person on the screen during the smooth tracking process The eye fixation point determines the object tracked by the human eye, and displays the object tracked by the human eye in the center of the screen; when the human eye action is gazing, and the human eye gaze point on the screen is located in the border area of the screen, the image is oriented away from this The direction of the boundary moves.

Optionally, when the user interface is displayed on the screen, the control unit controls the display to perform the following operations according to the human eye movement and the human gaze point on the screen: when the human eye gaze point on the screen is located in a functional area, then the human eye movement is closed When the eye is on, run the function corresponding to the functional area; when the human eye moves smoothly and the human gaze point on the screen moves from one functional area to another functional area, switch the focus to the other area.

According to the method and device for controlling a display of the present invention, the display can be controlled according to the human eye image, and the user's hands are freed from the remote control device.

Description of drawings

These and/or other aspects and advantages of the present invention will become clearer and easier to understand through the following description of embodiments in conjunction with the accompanying drawings, wherein:

FIG. 1 shows a flowchart of a method of controlling a display according to a first exemplary embodiment of the present invention.

FIG. 2 shows a block diagram of an apparatus for controlling a display according to a second exemplary embodiment of the present invention.

FIG. 3 shows a flowchart of a method for remotely controlling a display using electronic glasses according to a third exemplary embodiment of the present invention.

FIG. 4 illustrates a block diagram of an apparatus for remotely controlling a display using electronic glasses according to a fourth exemplary embodiment of the present invention.

FIG. 5 shows a flowchart of a method of controlling a display according to a fifth exemplary embodiment of the present invention.

FIG. 6 shows a block diagram of an apparatus for controlling a display according to a sixth exemplary embodiment of the present invention.

FIG. 7 shows a flowchart of a method for remotely controlling a display using electronic glasses according to a seventh exemplary embodiment of the present invention.

FIG. 8 illustrates a block diagram of an apparatus for remotely controlling a display using electronic glasses according to an eighth exemplary embodiment of the present invention.

FIG. 9 shows a flowchart of a method of controlling a display according to a ninth exemplary embodiment of the present invention.

FIG. 10 shows a block diagram of an apparatus for controlling a display according to a tenth exemplary embodiment of the present invention.

detailed description

Reference will now be made in detail to embodiments of the invention, examples of which are illustrated in the accompanying drawings. The embodiments are described below in order to explain the present invention by referring to the figures.

FIG. 1 shows a flowchart of a method of controlling a display according to a first exemplary embodiment of the present invention.

In step 101, an eye image captured by electronic glasses is obtained. The user wears electronic glasses when viewing the screen, and the user's eyes are captured by the image capture device on the electronic glasses to obtain the user's human eye image.

In step 102, human eye feature information is extracted from the human eye image. The human eye feature information is information that can represent the line of sight of the human eye. For example, the feature information of the human eye may be the position information of the pupil center and the corneal reflection point, and the sight information of the human eye can be obtained according to the position information of the pupil center and the corneal reflection point.

In step 103, the glasses position and attitude information of the electronic glasses are detected. Glasses position and attitude information of electronic glasses can be expressed using a predetermined coordinate system.

In step 104, the gaze point of the human eye on the screen is detected according to the characteristic information of the human eye, the position and posture information of the glasses. Firstly, the gaze point of the human eye on the electronic glasses is obtained according to the characteristic information of the human eye. Then, the gaze point of the human eye on the screen is obtained according to the gaze point of the human eye on the electronic glasses, the position and posture information of the glasses.

Specifically, the existing gaze tracking technology can be used to obtain the line of sight of the human eye on the electronic glasses coordinate system according to the characteristic information of the human eye, and then obtain the gaze point of the human eye on the electronic glasses according to the line of sight of the human eye on the coordinate system of the electronic glasses. Then, according to the gaze point of the human eye on the electronic glasses and the position and attitude information of the electronic glasses, a corresponding coordinate transformation is performed to obtain the gaze point of the human eye on the screen.

In one embodiment, the characteristic information of the human eye is the position information of the pupil center and the corneal reflection point, and the gaze point of the human eye on the electronic glasses is calculated according to the position information of the pupil center and the corneal reflection point. That is, use the pupil-cornea reflection vector method to analyze the human eye feature information to obtain the human eye line of sight on the electronic glasses coordinate system, and obtain the human gaze point on the electronic glasses according to the human eye line of sight on the electronic glasses coordinate system.

In step 105, the human eye movement is detected according to the characteristic information of the human eye and/or the change of the fixation point of the human eye on the screen. Including human eye movements such as eye closing, gaze, saccades, and smooth movements. When the corneal reflection point cannot be obtained from the human eye characteristic information within the first predetermined time, the human eye motion is detected as eyes closed. When the gaze point of the human eye on the screen remains unchanged within the second predetermined time, the human eye movement is detected as gazing; when the gaze point of the human eye on the screen moves beyond a predetermined threshold within the third predetermined time, the human eye movement Detected as saccades; eye movement detected as smooth motion when the point of human gaze on the screen moves at a predetermined speed. The above predetermined values can be preset or set by the user.

In step 106, the operation of the display is controlled according to the movement of the human eyes and the gaze point of the human eyes on the screen. Specifically, various operations of the display can be controlled through different combinations of human eye movements and human gaze points on the screen.

For example, the gaze point of the human eye on the screen is equivalent to the position of the cursor when using the mouse to control the display, or the position selected by the remote control when using the remote control to control the display, and the human eye action is equivalent to clicking the mouse or pressing the button of the remote control. For example, closing the eyes is equivalent to clicking the left mouse button or pressing the confirmation button on the remote control.

Preferably, the operation of the display is controlled according to human eye movements and human gaze points on the screen in different modes.

In viewing mode, when the human eye action is gazing, and the human gaze point on the screen is located in the middle of the screen, the image in the area near the gaze point will be enlarged and displayed, so that users can clearly see the subtleties they care about. image. Here, the size of the enlarged area near the gaze point can be preset or set by the user. In the case that the image displayed on the screen is a moving image, when the human eye action is smooth tracking, determine the object tracked by the human eye according to the point of focus of the human eye on the screen during the smooth tracking process, and extract the object from the TV program stream. Object information, track the object, and display the tracked object in the center of the screen. When the human eye action is gazing, and the gaze point of the human eye on the screen is located in the border area of the screen, the image is moved toward a direction away from the border. For example: when the human eye action is gazing, and the gaze point of the human eye on the screen is located in the left border area of the screen, move the image to the right to display the undisplayed image on the left.

In user interface mode, the image of the screen is divided into several areas, and each area represents a specific function. When the gaze point of the human eye on the screen is located in a functional area, and then the human eye is closed, the function corresponding to the functional area is executed.

For example: when the user gazes at the area where the page-turning function is located and then closes his eyes, the page-turning function runs. When the human eye movement is a smooth movement, and the human gaze point on the screen moves from one functional area to another functional area, switch the focus to the other area (that is, change from selecting one area to selecting another area) .

In addition, it should be understood that the function may also be calling various application software, internal functions of the application software, built-in functions of the display (for example, brightness adjustment, mode adjustment, etc.), or user-defined functions, etc.

In addition, the viewing angle when watching the screen is detected according to the human eye feature information, glasses position and posture information, and then the depth of the screen is adjusted according to the viewing angle. When the viewing angle exceeds the predetermined angle threshold, the depth of the area on the screen close to the electronic glasses is reduced, and the depth of the area on the screen far away from the electronic glasses is adjusted large, so that the viewing angle is less than or equal to the predetermined angle threshold, thereby protecting the eyes. Purpose.

It should be understood that steps 101-106 may be all implemented by electronic glasses (or a remote control device with electronic glasses) or a display; or partly by electronic glasses (or a remote control device with electronic glasses), and other parts by a display.

FIG. 2 shows a block diagram of an apparatus for controlling a display according to a second exemplary embodiment of the present invention.

As shown in Figure 2, the device 200 for controlling a display according to the present invention includes: a human eye image acquisition unit 201, a feature extraction unit 202, a position and posture detection unit 203, a gaze point detection unit 204, a human eye movement detection unit 205, a control unit 206.

The human eye image acquisition unit 201 is used to obtain the human eye image by using electronic glasses to photograph the human eye. The electronic glasses may have various imaging devices such as CMOS (Complementary Metal Oxide Semiconductor) sensors and CCD (Charge Coupled Device) sensors.

The feature extraction unit 202 is used for extracting human eye feature information from the obtained human eye image. The human eye feature information is information that can represent the line of sight of the human eye. For example, the feature information of the human eye may be the position information of the pupil center and the corneal reflection point, and the sight information of the human eye can be obtained according to the position information of the pupil center and the corneal reflection point.

The position and posture detection unit 203 is used to detect the glasses position and posture information of the electronic glasses. The position and posture detection unit 203 may be various position and posture detection devices.

The gaze point detection unit 204 is configured to detect the gaze point of the human eye on the screen according to the characteristic information of the human eye, the position and posture information of the glasses.

The gaze point detection unit 204 includes a glasses gaze point detection unit (not shown) and a screen gaze point detection unit (not shown). The gaze point detection unit of the glasses is used for obtaining the gaze point of the human eye on the electronic glasses according to the characteristic information of the human eye. Specifically, the gaze point detection unit of the glasses can obtain the human eye gaze on the electronic glasses coordinate system according to the human eye feature information through the existing gaze tracking technology, and then obtain the human eye gaze on the electronic glasses according to the human eye gaze on the electronic glasses coordinate system. The point of human gaze. The screen gaze point detection unit is used to perform corresponding coordinate transformation according to the gaze point of the human eye on the electronic glasses and the position and attitude information of the electronic glasses, so as to obtain the gaze point of the human eye on the screen.

In one embodiment, the feature information of the human eye is the position information of the pupil center and the corneal reflection point, and the gaze point detection unit of the glasses further includes a calculation unit (not shown). The calculation unit calculates the gaze point of the human eye on the electronic glasses according to the position information of the pupil center and the corneal reflection point. That is, use the pupil-cornea reflection vector method to analyze the human eye feature information to obtain the human eye line of sight on the electronic glasses coordinate system, and obtain the human gaze point on the electronic glasses according to the human eye line of sight on the electronic glasses coordinate system.

The human eye movement detection unit 205 is configured to detect the human eye movement according to the characteristic information of the human eye and/or the change of the gaze point of the human eye on the screen. Including human eye movements such as eye closing, gaze, saccades, and smooth movements. When the corneal reflection point cannot be obtained from the human eye characteristic information within the first predetermined time, the human eye motion is detected as eyes closed. When the gaze point of the human eye on the screen remains unchanged within the second predetermined time, the human eye movement is detected as gazing; when the gaze point of the human eye on the screen moves beyond a predetermined threshold within the third predetermined time, the human eye movement Detected as saccades; eye movement detected as smooth motion when the point of human gaze on the screen moves at a predetermined speed. The above predetermined values can be preset or set by the user.

The control unit 206 is used for controlling the operation of the display according to human eye movements and human gaze points on the screen. Specifically, the control unit 206 can control various operations of the display through different combinations of human eye movements and human gaze points on the screen.

For example, the gaze point of the human eye on the screen is equivalent to the position of the cursor when using the mouse to control the display, or the position selected by the remote control when using the remote control to control the display, and the human eye action is equivalent to clicking the mouse or pressing the button of the remote control. For example, closing the eyes is equivalent to clicking the left mouse button or pressing the confirmation button on the remote control.

Preferably, the control unit 206 controls the operation of the display in different modes according to human eye movements and human gaze points on the screen.

In viewing mode, when the human eye action is gazing, and the human gaze point on the screen is located in the middle of the screen, the image in the area near the gaze point will be enlarged and displayed, so that users can clearly see the subtleties they care about. image. The size of the enlarged area near the gaze point can be preset or set by the user. In the case that the image displayed on the screen is a moving image, when the human eye action is smooth tracking, determine the object tracked by the human eye according to the point of focus of the human eye on the screen during the smooth tracking process, and extract the object from the TV program stream. Object information, track the object, and display the tracked object in the center of the screen. When the human eye action is gazing, and the gaze point of the human eye on the screen is located in the border area of the screen, the image is moved toward a direction away from the border. For example: when the human eye action is gazing, and the gaze point of the human eye on the screen is located in the left border area of the screen, move the image to the right to display the undisplayed image on the left.

In user interface mode, the image of the screen is divided into several areas, and each area represents a specific function. When the gaze point of the human eye on the screen is located in a functional area, and then the human eye is closed, the function corresponding to the functional area is executed.

For example: when the user gazes at the area where the page-turning function is located and then closes his eyes, the page-turning function runs. When the human eye movement is a smooth movement, and the human gaze point on the screen moves from one functional area to another functional area, switch the focus to the other area (that is, change from selecting one area to selecting another area) .

The device 200 for controlling a display further includes a viewing angle detection unit (not shown) and a depth adjustment unit (not shown). The viewing angle detection unit detects the viewing angle when viewing the screen according to the human eye feature information, glasses position and posture information, and the depth adjustment unit adjusts the depth of the screen according to the viewing angle. When the viewing angle exceeds the predetermined angle threshold, the depth of the area on the screen close to the electronic glasses is reduced, and the depth of the area on the screen far away from the electronic glasses is increased, so that the viewing angle is less than or equal to the predetermined angle threshold, so as to protect the eyes .

It should be understood that the units 101-106 can be all arranged in the electronic glasses (or the remote control device with the electronic glasses) or the display; or some of them are arranged in the electronic glasses (or the remote control device with the electronic glasses), and the other parts are arranged in the display.

FIG. 3 shows a flowchart of a method for remotely controlling a display using electronic glasses according to a third exemplary embodiment of the present invention.

In step 301, an eye image captured by electronic glasses is obtained. The user wears electronic glasses when viewing the screen, and the user's eyes are captured by the image capture device on the electronic glasses to obtain the user's human eye image.

In step 302, human eye feature information is extracted from the human eye image. The human eye feature information is information that can represent the line of sight of the human eye. For example, the feature information of the human eye may be the position information of the pupil center and the corneal reflection point, and the sight information of the human eye can be obtained according to the position information of the pupil center and the corneal reflection point.

In step 303, the glasses position and attitude information of the electronic glasses are detected. Glasses position and attitude information of electronic glasses can be expressed using a predetermined coordinate system.

In step 304, the feature information of the captured human eye image and the glasses position and posture information of the electronic glasses are sent to the display.

It should be understood that the above steps can be implemented by electronic glasses or a remote control device with electronic glasses.

FIG. 4 illustrates a block diagram of an apparatus for remotely controlling a display using electronic glasses according to a fourth exemplary embodiment of the present invention.

As shown in FIG. 4 , the device 400 for remote display using electronic glasses according to the present invention includes: a human eye image acquisition unit 401 , a feature extraction unit 402 , a position and posture detection unit 403 , and a sending unit 404 .

The human eye image acquisition unit 401 is used to obtain human eye images by photographing human eyes. The human eye image acquisition unit 401 may be various photographing devices such as a CMOS (Complementary Metal Oxide Semiconductor) sensor, a CCD (Charge Coupled Device) sensor, and the like.

The feature extraction unit 402 is used to extract human eye feature information from the obtained human eye image. The human eye feature information is information that can represent the line of sight of the human eye. For example, the feature information of the human eye may be the position information of the pupil center and the corneal reflection point, and the sight information of the human eye can be obtained according to the position information of the pupil center and the corneal reflection point.

The position and posture detection unit 403 is used to detect the glasses position and posture information of the electronic glasses. The position and posture detection unit 403 may be various position and posture detection devices.

The sending unit 404 is used to send the feature information of the captured human eye image and the glasses position and attitude information of the electronic glasses to the display. The sending unit 404 may be various wireless data transmission devices.

It should be understood that the device 400 may be implemented as electronic glasses or a remote control device with electronic glasses.

FIG. 5 shows a flowchart of a method of controlling a display according to a fifth exemplary embodiment of the present invention.

In step 501, the feature information of the captured human eye image and the glasses position and posture information of the electronic glasses are received from the electronic glasses.

In step 502, the gaze point of the human eye on the screen is detected according to the characteristic information of the human eye, the position and posture information of the glasses. Firstly, the gaze point of the human eye on the electronic glasses is obtained according to the characteristic information of the human eye. Then, the gaze point of the human eye on the screen is obtained according to the gaze point of the human eye on the electronic glasses, the position and posture information of the glasses. Specifically, the existing gaze tracking technology can be used to obtain the line of sight of the human eye on the electronic glasses coordinate system according to the characteristic information of the human eye, and then obtain the gaze point of the human eye on the electronic glasses according to the line of sight of the human eye on the coordinate system of the electronic glasses. Then, according to the gaze point of the human eye on the electronic glasses and the position and attitude information of the electronic glasses, a corresponding coordinate transformation is performed to obtain the gaze point of the human eye on the screen.

In one embodiment, the characteristic information of the human eye is the position information of the pupil center and the corneal reflection point, and the gaze point of the human eye on the electronic glasses is calculated according to the position information of the pupil center and the corneal reflection point. That is, use the pupil-cornea reflection vector method to analyze the human eye feature information to obtain the human eye line of sight on the electronic glasses coordinate system, and obtain the human gaze point on the electronic glasses according to the human eye line of sight on the electronic glasses coordinate system.

In step 503, the human eye movement is detected according to the characteristic information of the human eye and/or the change of the gaze point of the human eye on the screen. Including human eye movements such as eye closing, gaze, saccades, and smooth movements. When the corneal reflection point cannot be obtained from the human eye characteristic information within the first predetermined time, the human eye motion is detected as eyes closed. When the gaze point of the human eye on the screen remains unchanged within the second predetermined time, the human eye movement is detected as gazing; when the gaze point of the human eye on the screen moves beyond a predetermined threshold within the third predetermined time, the human eye movement Detected as saccades; eye movement detected as smooth motion when the point of human gaze on the screen moves at a predetermined speed. The above predetermined values can be preset or set by the user.

In step 504, the operation of the display is controlled according to the movement of the human eyes and the gaze point of the human eyes on the screen. Specifically, various operations of the display can be controlled through different combinations of human eye movements and human gaze points on the screen.

For example, the gaze point of the human eye on the screen is equivalent to the position of the cursor when using the mouse to control the display, or the position selected by the remote control when using the remote control to control the display, and the human eye action is equivalent to clicking the mouse or pressing the button of the remote control. For example, closing the eyes is equivalent to clicking the left mouse button or pressing the confirmation button on the remote control.

Preferably, the operation of the display is controlled according to human eye movements and human gaze points on the screen in different modes.

In viewing mode, when the human eye action is gazing, and the human gaze point on the screen is located in the middle of the screen, the image in the area near the gaze point will be enlarged and displayed, so that users can clearly see the subtleties they care about. image. Wherein, the size of the enlarged area near the gaze point can be preset or set by the user. In the case that the image displayed on the screen is a moving image, when the human eye action is smooth tracking, determine the object tracked by the human eye according to the point of focus of the human eye on the screen during the smooth tracking process, and extract the object from the TV program stream. Object information, track the object, and display the tracked object in the center of the screen. When the human eye action is gazing, and the gaze point of the human eye on the screen is located in the border area of the screen, the image is moved toward a direction away from the border. For example: when the human eye action is gazing, and the gaze point of the human eye on the screen is located in the left border area of the screen, move the image to the right to display the undisplayed image on the left.

In user interface mode, the image of the screen is divided into several areas, and each area represents a specific function. When the gaze point of the human eye on the screen is located in a functional area, and then the human eye is closed, the function corresponding to the functional area is executed.

For example: when the user gazes at the area where the page-turning function is located and then closes his eyes, the page-turning function runs. When the human eye movement is a smooth movement, and the human gaze point on the screen moves from one functional area to another functional area, switch the focus to the other area (that is, change from selecting one area to selecting another area) .

In addition, the viewing angle when watching the screen is detected according to the human eye feature information, glasses position and posture information, and then the depth of the screen is adjusted according to the viewing angle. When the viewing angle exceeds the predetermined angle threshold, the depth of the area on the screen close to the electronic glasses is reduced, and the depth of the area on the screen far away from the electronic glasses is adjusted large, so that the viewing angle is less than or equal to the predetermined angle threshold, thereby protecting the eyes. Purpose.

It should be understood that the above steps can be implemented by the display.

FIG. 6 shows a block diagram of an apparatus for controlling a display according to a sixth exemplary embodiment of the present invention.

As shown in FIG. 6 , the device 600 for controlling a display according to the present invention includes: a receiving unit 601 , a gaze point detection unit 602 , a human eye movement detection unit 603 , and a control unit 604 .

The receiving unit 601 is used for receiving the feature information of the captured human eye image and the glasses position and posture information of the electronic glasses from the electronic glasses.

The gaze point detection unit 602 is configured to detect the gaze point of the human eye on the screen according to the characteristic information of the human eye, the position and posture information of the glasses.

The gaze point detection unit 602 includes a glasses gaze point detection unit (not shown) and a screen gaze point detection unit (not shown). The gaze point detection unit of the glasses is used for obtaining the gaze point of the human eye on the electronic glasses according to the characteristic information of the human eye. Specifically, the gaze point detection unit of the glasses can obtain the human eye gaze on the electronic glasses coordinate system according to the human eye feature information through the existing gaze tracking technology, and then obtain the human eye gaze on the electronic glasses according to the human eye gaze on the electronic glasses coordinate system. The point of human gaze. The screen gaze point detection unit is used to perform corresponding coordinate transformation according to the gaze point of the human eye on the electronic glasses and the position and attitude information of the electronic glasses, so as to obtain the gaze point of the human eye on the screen.

In one embodiment, the feature information of the human eye is the position information of the pupil center and the corneal reflection point, and the gaze point detection unit of the glasses further includes a calculation unit (not shown). The calculation unit calculates the gaze point of the human eye on the electronic glasses according to the position information of the pupil center and the corneal reflection point. That is, use the pupil-cornea reflection vector method to analyze the human eye feature information to obtain the human eye line of sight on the electronic glasses coordinate system, and obtain the human gaze point on the electronic glasses according to the human eye line of sight on the electronic glasses coordinate system.

The human eye movement detection unit 603 is configured to detect the human eye movement according to the characteristic information of the human eye and/or the change of the gaze point of the human eye on the screen. Including human eye movements such as eye closing, gaze, saccades, and smooth movements. When the corneal reflection point cannot be obtained from the human eye characteristic information within the first predetermined time, the human eye motion is detected as eyes closed. When the gaze point of the human eye on the screen remains unchanged within the second predetermined time, the human eye movement is detected as gazing; when the gaze point of the human eye on the screen moves beyond a predetermined threshold within the third predetermined time, the human eye movement Detected as saccades; eye movement detected as smooth motion when the point of human gaze on the screen moves at a predetermined speed. The above predetermined values can be preset or set by the user.

The control unit 604 is used to control the operation of the display according to the movement of the human eyes and the gaze point of the human eyes on the screen. Specifically, various operations of the display can be controlled through different combinations of human eye movements and human gaze points on the screen.

For example, the gaze point of the human eye on the screen is equivalent to the position of the cursor when using the mouse to control the display, or the position selected by the remote control when using the remote control to control the display, and the human eye action is equivalent to clicking the mouse or pressing the button of the remote control. For example, closing the eyes is equivalent to clicking the left mouse button or pressing the confirmation button on the remote control.

Preferably, the control unit 604 controls the operation of the display in different modes according to human eye movements and human gaze points on the screen.

In viewing mode, when the human eye action is gazing, and the human gaze point on the screen is located in the middle of the screen, the image in the area near the gaze point will be enlarged and displayed, so that users can clearly see the subtleties they care about. image. Wherein, the size of the enlarged area near the gaze point can be preset or set by the user. In the case that the image displayed on the screen is a moving image, when the human eye action is smooth tracking, determine the object tracked by the human eye according to the point of focus of the human eye on the screen during the smooth tracking process, and extract the object from the TV program stream. Object information, track the object, and display the tracked object in the center of the screen. When the human eye action is gazing, and the gaze point of the human eye on the screen is located in the border area of the screen, the image is moved toward a direction away from the border. For example: when the human eye action is gazing, and the gaze point of the human eye on the screen is located in the left border area of the screen, move the image to the right to display the undisplayed image on the left.

In user interface mode, the image of the screen is divided into several areas, and each area represents a specific function. When the gaze point of the human eye on the screen is located in a functional area, and then the human eye is closed, the function corresponding to the functional area is executed.

For example: when the user gazes at the area where the page-turning function is located and then closes his eyes, the page-turning function runs. When the human eye movement is a smooth movement, and the human gaze point on the screen moves from one functional area to another functional area, switch the focus to the other area (that is, change from selecting one area to selecting another area) .

The device 600 for controlling a display also includes a viewing angle detection unit (not shown) and a depth adjustment unit (not shown). The viewing angle detection unit detects the viewing angle when viewing the screen according to the human eye feature information, glasses position and posture information, and the depth adjustment unit adjusts the depth of the screen according to the viewing angle. When the viewing angle exceeds the predetermined angle threshold, the depth of the area on the screen close to the electronic glasses is reduced, and the depth of the area on the screen far away from the electronic glasses is increased, so that the viewing angle is less than or equal to the predetermined angle threshold, so as to protect the eyes .

It should be understood that device 600 may be arranged in a display.

FIG. 7 shows a flowchart of a method for remotely controlling a display using electronic glasses according to a seventh exemplary embodiment of the present invention.

In step 701, an eye image captured by electronic glasses is obtained. The user wears electronic glasses when viewing the screen, and the user's eyes are captured by the image capture device on the electronic glasses to obtain the user's human eye image.

In step 702, human eye feature information is extracted from the human eye image. The human eye feature information is information that can represent the line of sight of the human eye. For example, the feature information of the human eye may be the position information of the pupil center and the corneal reflection point, and the sight information of the human eye can be obtained according to the position information of the pupil center and the corneal reflection point.

In step 703, the glasses position and posture information of the electronic glasses are detected. Glasses position and attitude information of electronic glasses can be expressed using a predetermined coordinate system.

In step 704, the gaze point of the human eye on the screen is detected according to the characteristic information of the human eye, the position and posture information of the glasses. Firstly, the gaze point of the human eye on the electronic glasses is obtained according to the characteristic information of the human eye. Then, the gaze point of the human eye on the screen is obtained according to the gaze point of the human eye on the electronic glasses, the position and posture information of the glasses. Specifically, the existing gaze tracking technology can be used to obtain the line of sight of the human eye on the electronic glasses coordinate system according to the characteristic information of the human eye, and then obtain the gaze point of the human eye on the electronic glasses according to the line of sight of the human eye on the coordinate system of the electronic glasses. Then, according to the gaze point of the human eye on the electronic glasses and the position and attitude information of the electronic glasses, a corresponding coordinate transformation is performed to obtain the gaze point of the human eye on the screen.

In one embodiment, the characteristic information of the human eye is the position information of the pupil center and the corneal reflection point, and the gaze point of the human eye on the electronic glasses is calculated according to the position information of the pupil center and the corneal reflection point. That is, use the pupil-cornea reflection vector method to analyze the human eye feature information to obtain the human eye line of sight on the electronic glasses coordinate system, and obtain the human gaze point on the electronic glasses according to the human eye line of sight on the electronic glasses coordinate system.

In step 705, the human eye movement is detected according to the characteristic information of the human eye and/or the change of the gaze point of the human eye on the screen. Including human eye movements such as eye closing, gaze, saccades, and smooth movements. When the corneal reflection point cannot be obtained from the human eye characteristic information within the first predetermined time, the human eye motion is detected as eyes closed. When the gaze point of the human eye on the screen remains unchanged within the second predetermined time, the human eye movement is detected as gazing; when the gaze point of the human eye on the screen moves beyond a predetermined threshold within the third predetermined time, the human eye movement Detected as saccades; eye movement detected as smooth motion when the point of human gaze on the screen moves at a predetermined speed. The above predetermined values can be preset or set by the user.

In step 706, the feature information of the captured human eye image and the glasses position and posture information of the electronic glasses are sent to the display.

It should be understood that the above steps can be implemented by electronic glasses or a remote control device with electronic glasses.

FIG. 8 illustrates a block diagram of an apparatus for remotely controlling a display using electronic glasses according to an eighth exemplary embodiment of the present invention.

As shown in Figure 8, the device 800 for controlling a display according to the present invention includes: a human eye image acquisition unit 801, a feature extraction unit 802, a position and posture detection unit 803, a gaze point detection unit 804, a human eye motion detection unit 805, a sending unit 806.

The human eye image acquisition unit 801 is used to obtain human eye images by photographing human eyes. The human eye image acquisition unit 801 may be various photographing devices such as a CMOS (Complementary Metal Oxide Semiconductor) sensor, a CCD (Charge Coupled Device) sensor, and the like.

The feature extraction unit 802 is used to extract human eye feature information from the obtained human eye image. The human eye feature information is information that can represent the line of sight of the human eye. For example, the feature information of the human eye may be the position information of the pupil center and the corneal reflection point, and the sight information of the human eye can be obtained according to the position information of the pupil center and the corneal reflection point.

The position and posture detection unit 803 is used to detect the glasses position and posture information of the electronic glasses. The position and posture detection unit 803 may be various position and posture detection devices.

The gaze point detection unit 804 is configured to detect the gaze point of the human eye on the screen according to the characteristic information of the human eye, the position and posture information of the glasses.

The gaze point detection unit 804 includes a glasses gaze point detection unit (not shown) and a screen gaze point detection unit (not shown). The gaze point detection unit of the glasses is used for obtaining the gaze point of the human eye on the electronic glasses according to the characteristic information of the human eye. Specifically, the gaze point detection unit of the glasses can obtain the human eye gaze on the electronic glasses coordinate system according to the human eye feature information through the existing gaze tracking technology, and then obtain the human eye gaze on the electronic glasses according to the human eye gaze on the electronic glasses coordinate system. The point of human gaze. The screen gaze point detection unit is used to perform corresponding coordinate transformation according to the gaze point of the human eye on the electronic glasses and the position and attitude information of the electronic glasses, so as to obtain the gaze point of the human eye on the screen.

In one embodiment, the feature information of the human eye is the position information of the pupil center and the corneal reflection point, and the gaze point detection unit of the glasses further includes a calculation unit (not shown). The calculation unit calculates the gaze point of the human eye on the electronic glasses according to the position information of the pupil center and the corneal reflection point. That is, use the pupil-cornea reflection vector method to analyze the human eye feature information to obtain the human eye line of sight on the electronic glasses coordinate system, and obtain the human gaze point on the electronic glasses according to the human eye line of sight on the electronic glasses coordinate system.

The human eye movement detection unit 805 is configured to detect the human eye movement according to the characteristic information of the human eye and/or the change of the gaze point of the human eye on the screen. Including human eye movements such as eye closing, gaze, saccades, and smooth movements. When the corneal reflection point cannot be obtained from the human eye characteristic information within the first predetermined time, the human eye motion is detected as eyes closed. When the gaze point of the human eye on the screen remains unchanged within the second predetermined time, the human eye movement is detected as gazing; when the gaze point of the human eye on the screen moves beyond a predetermined threshold within the third predetermined time, the human eye movement Detected as saccades; eye movement detected as smooth motion when the point of human gaze on the screen moves at a predetermined speed. The above predetermined values can be preset or set by the user.

The sending unit 806 is used to send the feature information of the captured human eye image and the glasses position and posture information of the electronic glasses to the display. The sending unit 806 may be various wireless data transmission devices.

It should be understood that the device 800 may be implemented as electronic glasses or a remote control device with electronic glasses.

FIG. 9 shows a flowchart of a method of controlling a display according to a ninth exemplary embodiment of the present invention.

In step 901, the feature information of the captured human eye image and the glasses position and posture information of the electronic glasses are received from the electronic glasses.

In step 902, the operation of the display is controlled according to the movement of the human eyes and the gaze point of the human eyes on the screen. Specifically, various operations of the display can be controlled through different combinations of human eye movements and human gaze points on the screen.

For example, the gaze point of the human eye on the screen is equivalent to the position of the cursor when using the mouse to control the display, or the position selected by the remote control when using the remote control to control the display, and the human eye action is equivalent to clicking the mouse or pressing the button of the remote control. For example, closing the eyes is equivalent to clicking the left mouse button or pressing the confirmation button on the remote control.

Preferably, the operation of the display is controlled according to human eye movements and human gaze points on the screen in different modes.

In viewing mode, when the human eye action is gazing, and the human gaze point on the screen is located in the middle of the screen, the image in the area near the gaze point will be enlarged and displayed, so that users can clearly see the subtleties they care about. image. Wherein, the size of the enlarged area near the gaze point can be preset or set by the user. In the case that the image displayed on the screen is a moving image, when the human eye action is smooth tracking, determine the object tracked by the human eye according to the point of focus of the human eye on the screen during the smooth tracking process, and extract the object from the TV program stream. Object information, track the object, and display the tracked object in the center of the screen. When the human eye action is gazing, and the gaze point of the human eye on the screen is located in the border area of the screen, the image is moved toward a direction away from the border. For example: when the human eye action is gazing, and the gaze point of the human eye on the screen is located in the left border area of the screen, move the image to the right to display the undisplayed image on the left.

In user interface mode, the image of the screen is divided into several areas, and each area represents a specific function. When the gaze point of the human eye on the screen is located in a functional area, and then the human eye is closed, the function corresponding to the functional area is executed.

For example: when the user gazes at the area where the page-turning function is located and then closes his eyes, the page-turning function runs. When the human eye movement is a smooth movement, and the human gaze point on the screen moves from one functional area to another functional area, switch the focus to the other area (that is, change from selecting one area to selecting another area) .

It should be understood that the above steps can be implemented by the display.

FIG. 10 shows a block diagram of an apparatus for controlling a display according to a tenth exemplary embodiment of the present invention.

As shown in FIG. 10 , a device 1000 for controlling a display according to the present invention includes: a receiving unit 1001 and a control unit 1002 .

The receiving unit 1001 is used for receiving the feature information of the captured human eye image and the glasses position and posture information of the electronic glasses from the electronic glasses.

The control unit 1002 is used for controlling the operation of the display according to human eye movements and human gaze points on the screen. Specifically, various operations of the display can be controlled through different combinations of human eye movements and human gaze points on the screen.

For example, the gaze point of the human eye on the screen is equivalent to the position of the cursor when using the mouse to control the display, or the position selected by the remote control when using the remote control to control the display, and the human eye action is equivalent to clicking the mouse or pressing the button of the remote control. For example, closing the eyes is equivalent to clicking the left mouse button or pressing the confirmation button on the remote control.

Preferably, the control unit 1002 controls the operation of the display in different modes according to human eye movements and human gaze points on the screen.

In viewing mode, when the human eye action is gazing, and the human gaze point on the screen is located in the middle of the screen, the image in the area near the gaze point will be enlarged and displayed, so that users can clearly see the subtleties they care about. image. Wherein, the size of the enlarged area near the gaze point can be preset or set by the user. In the case that the image displayed on the screen is a moving image, when the human eye action is smooth tracking, determine the object tracked by the human eye according to the point of focus of the human eye on the screen during the smooth tracking process, and extract the object from the TV program stream. Object information, track the object, and display the tracked object in the center of the screen. When the human eye action is gazing, and the gaze point of the human eye on the screen is located in the border area of the screen, the image is moved toward a direction away from the border. For example: when the human eye action is gazing, and the gaze point of the human eye on the screen is located in the left border area of the screen, move the image to the right to display the undisplayed image on the left.

In user interface mode, the image of the screen is divided into several areas, and each area represents a specific function. When the gaze point of the human eye on the screen is located in a functional area, and then the human eye is closed, the function corresponding to the functional area is executed.

For example: when the user gazes at the area where the page-turning function is located and then closes his eyes, the page-turning function runs. When the human eye movement is a smooth movement, and the human gaze point on the screen moves from one functional area to another functional area, switch the focus to the other area (that is, change from selecting one area to selecting another area) .

It should be understood that device 1000 may be arranged in a display.

Furthermore, the above-described methods according to exemplary embodiments of the present invention can be implemented as a computer program so that when the program is executed, the above-described methods are implemented. Each unit in the apparatus for controlling a display according to an exemplary embodiment of the present invention may be implemented as a hardware component. Those skilled in the art may implement each unit, for example, using a Field Programmable Gate Array (FPGA) or an Application Specific Integrated Circuit (ASIC) according to the defined processing performed by each unit.

While certain embodiments of the present invention have been shown and described, it should be understood by those skilled in the art that modifications may be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the claims and their equivalents. to modify.

Claims (20)

1. A method of controlling a display, comprising: Obtain human eye images by shooting human eyes through electronic glasses; Extracting human eye feature information from the human eye image, wherein the human eye feature information is the position information of the pupil center and the corneal reflection point; Detect the position and attitude information of the glasses of the electronic glasses; Calculate the gaze point of the human eye on the electronic glasses according to the position information of the pupil center and the corneal reflection point; Obtain the gaze point of the human eye on the screen according to the gaze point of the human eye on the electronic glasses, the position and posture information of the glasses; Detect human eye movements based on human eye feature information and/or changes in human gaze points on the screen; Controls the operation of the display based on human eye movements and eye gaze points on the screen. 2. The method according to claim 1, wherein the human eye action comprises at least one of eye closing, gaze, saccade, and smooth movement. 3. The method of claim 2, wherein, When the corneal reflection point cannot be obtained from the human eye feature information within the first predetermined time, the human eye movement is detected as eyes closed; When the gaze point of the human eye on the screen remains unchanged within the second predetermined time, the human eye movement is detected as gaze; When the moving range of the gaze point of the human eye on the screen exceeds a predetermined threshold within the third predetermined time, the human eye movement is detected as saccade; When the gaze point of the human eye on the screen moves at a predetermined speed, the human eye motion is detected as a smooth motion. 4. The method of claim 1, wherein the operation of the display is controlled in different modes according to human eye movements and human eye gaze points on the screen. 5. The method as claimed in claim 1, wherein, the step of controlling the operation of the display according to the human eye movement and the human eye gaze point on the screen comprises: when the screen displays an image, according to the human eye movement and the human eye gaze point on the screen Click to control the display to perform the following operations: When the human eye movement is gazing, and the human eye gaze point on the screen is located in the middle area of the screen, the image located near the human eye gaze point on the screen is enlarged and displayed; In the case that the image displayed on the screen is a moving image, when the human eye action is smooth tracking, the object tracked by the human eye is determined according to the gaze point of the human eye on the screen during the smooth tracking process, and the object tracked by the human eye is determined. displayed in the center of the screen; When the human eye action is gazing, and the gaze point of the human eye on the screen is located in the border area of the screen, the image is moved toward a direction away from the border. 6. The method as claimed in claim 1, wherein the step of controlling the operation of the display according to the human eye movement and the gaze point of the human eye on the screen comprises: when the user interface is displayed on the screen, according to the human eye movement and the human eye gaze point on the screen The gaze point control display does the following: When the human eye gaze point on the screen is located in a functional area, and then the human eye action is to close the eyes, run the function corresponding to the functional area; When the movement of the human eye is a smooth movement, and the gaze point of the human eye on the screen moves from one functional area to another functional area, the focus is switched to the other area. 7. The method of claim 1, further comprising: Detect the viewing angle when watching the screen according to the human eye feature information, glasses position and posture information; Adjust the depth of the screen according to the viewing angle. 8. The method of claim 7, wherein the step of adjusting the depth of the screen according to the viewing angle comprises: When the viewing angle exceeds the predetermined angle threshold, the depth of the area on the screen close to the electronic glasses is adjusted smaller, and the depth of the area on the screen far away from the electronic glasses is adjusted larger, so that the viewing angle is less than or equal to the predetermined angle threshold. 9. A device for controlling a display comprising: The human eye image acquisition unit is used to capture human eye images through electronic glasses to obtain human eye images; A feature extraction unit extracts human eye feature information from the human eye image, wherein the human eye feature information is the position information of the pupil center and the corneal reflection point; The position and posture detection unit detects the glasses position and posture information of the electronic glasses; The gaze point detection unit of the glasses calculates the gaze point of the human eye on the electronic glasses according to the position information of the pupil center and the corneal reflection point; The screen gaze point detection unit obtains the gaze point of the human eye on the screen according to the gaze point of the human eye on the electronic glasses, the position and posture information of the glasses; The human eye movement detection unit detects the human eye movement according to the characteristic information of the human eye and/or the change of the gaze point of the human eye on the screen; The control unit controls the operation of the display according to human eye movements and human gaze points on the screen. 10. The device according to claim 9, wherein the human eye motion detection unit is configured to detect at least one human eye motion among eye closing, gaze, saccade, and smooth movement. 11. The apparatus of claim 10, wherein, When the corneal reflection point cannot be obtained from the human eye feature information within the first predetermined time, the human eye movement is detected as eyes closed; When the gaze point of the human eye on the screen remains unchanged within the second predetermined time, the human eye motion is detected as gaze; When the moving range of the gaze point of the human eye on the screen exceeds a predetermined threshold within the third predetermined time, the human eye movement is detected as saccade; When the gaze point of the human eye on the screen moves at a predetermined speed, the human eye motion is detected as a smooth motion. 12. The device as claimed in claim 9, wherein the control unit controls the operation of the display according to human eye movements and human eye gaze points on the screen in different modes. 13. The device as claimed in claim 9, wherein, when the screen displays an image, the control unit controls the display to perform the following operations according to the human eye movement and the gaze point of the human eye on the screen: When the human eye movement is gazing, and the human eye gaze point on the screen is located in the middle area of the screen, the image located near the human eye gaze point on the screen is enlarged and displayed; In the case that the image displayed on the screen is a moving image, when the human eye action is smooth tracking, the object tracked by the human eye is determined according to the gaze point of the human eye on the screen during the smooth tracking process, and the object tracked by the human eye is determined. displayed in the center of the screen; When the human eye action is gazing, and the gaze point of the human eye on the screen is located in the border area of the screen, the image is moved toward a direction away from the border. 14. The device as claimed in claim 9, wherein, when the screen displays the user interface, the control unit controls the display to perform the following operations according to human eye movements and human gaze points on the screen: When the human eye gaze point on the screen is located in a functional area, and then the human eye action is to close the eyes, run the function corresponding to the functional area; When the movement of the human eye is a smooth movement, and the gaze point of the human eye on the screen moves from one functional area to another functional area, the focus is switched to the other area. 15. The device of claim 9, further comprising: The viewing angle detection unit detects the viewing angle when viewing the screen according to the human eye feature information, glasses position and posture information; The depth adjustment unit adjusts the depth of the screen according to the viewing angle. 16. The device according to claim 15, wherein, when the viewing angle exceeds a predetermined angle threshold, the depth adjusting unit adjusts the depth of the area on the screen close to the electronic glasses to be smaller, and adjusts the depth of the area on the screen far away from the electronic glasses to be larger , so that the viewing angle is less than or equal to a predetermined angle threshold. 17. A method of controlling a display comprising: Receive human eye characteristic information of the human eye image taken from the electronic glasses and glasses position and attitude information of the electronic glasses, wherein the human eye characteristic information is the position information of the pupil center and the corneal reflection point; Calculate the gaze point of the human eye on the electronic glasses according to the position information of the pupil center and the corneal reflection point; Obtain the gaze point of the human eye on the screen according to the gaze point of the human eye on the electronic glasses, the position and posture information of the glasses; Detect human eye movements based on human eye feature information and/or changes in human gaze points on the screen; Controls the operation of the display based on human eye movements and eye gaze points on the screen. 18. A device for controlling a display comprising: The receiving unit receives from the electronic glasses the human eye feature information of the captured human eye image and the glasses position and posture information of the electronic glasses, wherein the human eye feature information is the position information of the pupil center and the corneal reflection point; The gaze point detection unit of the glasses calculates the gaze point of the human eye on the electronic glasses according to the position information of the pupil center and the corneal reflection point; The screen gaze point detection unit obtains the gaze point of the human eye on the screen according to the gaze point of the human eye on the electronic glasses, the position and posture information of the glasses; The human eye movement detection unit detects the human eye movement according to the characteristic information of the human eye and/or the change of the gaze point of the human eye on the screen; The control unit controls the operation of the display according to human eye movements and human gaze points on the screen. 19. A method for using electronic glasses to remotely control a display, comprising: Obtain human eye images by shooting human eyes through electronic glasses; Extracting human eye feature information from the human eye image, wherein the human eye feature information is the position information of the pupil center and the corneal reflection point; Detect the position and attitude information of the glasses of the electronic glasses; According to the position information of the pupil center and the corneal reflection point, calculate the gaze point of the human eye on the electronic glasses; Obtain the gaze point of the human eye on the screen according to the gaze point of the human eye on the electronic glasses, the position and posture information of the glasses; Detect human eye movements based on human eye feature information and/or changes in human gaze points on the screen; Human eye movements and on-screen gaze points of human eyes for controlling the operation of the display are sent to the display. 20. A device that uses electronic glasses to remotely control a display, comprising: The human eye image acquisition unit is used to capture human eye images through electronic glasses to obtain human eye images; A feature extraction unit extracts human eye feature information from the human eye image, wherein the human eye feature information is the position information of the pupil center and the corneal reflection point; The position and posture detection unit detects the glasses position and posture information of the electronic glasses; The gaze point detection unit of the glasses calculates the gaze point of the human eye on the electronic glasses according to the position information of the pupil center and the corneal reflection point; The screen gaze point detection unit obtains the gaze point of the human eye on the screen according to the gaze point of the human eye on the electronic glasses, the position and posture information of the glasses; The human eye movement detection unit detects the human eye movement according to the characteristic information of the human eye and/or the change of the gaze point of the human eye on the screen; The sending unit is configured to send the human eye movement for controlling the operation of the display and the gaze point of the human eyes on the screen to the display.