CN110268370A - Eye gaze angle feedback in teleconference - Google Patents
Eye gaze angle feedback in teleconference Download PDFInfo
- Publication number
- CN110268370A CN110268370A CN201780084186.5A CN201780084186A CN110268370A CN 110268370 A CN110268370 A CN 110268370A CN 201780084186 A CN201780084186 A CN 201780084186A CN 110268370 A CN110268370 A CN 110268370A
- Authority
- CN
- China
- Prior art keywords
- teleconference
- exhibitor
- eye gaze
- gaze angle
- eye
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
- G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
- G06V40/18—Eye characteristics, e.g. of the iris
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/82—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using neural networks
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Physics & Mathematics (AREA)
- Multimedia (AREA)
- General Health & Medical Sciences (AREA)
- Evolutionary Computation (AREA)
- Health & Medical Sciences (AREA)
- General Engineering & Computer Science (AREA)
- Human Computer Interaction (AREA)
- Medical Informatics (AREA)
- Databases & Information Systems (AREA)
- Computing Systems (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Software Systems (AREA)
- Artificial Intelligence (AREA)
- Ophthalmology & Optometry (AREA)
- User Interface Of Digital Computer (AREA)
- Two-Way Televisions, Distribution Of Moving Picture Or The Like (AREA)
- Telephonic Communication Services (AREA)
- Image Analysis (AREA)
Abstract
A kind of includes the image that the remote participant in teleconference is captured by using camera associated with the calculating equipment of content that is shown of display exhibitor to the method that exhibitor provides feedback in teleconference.The eye gaze angle information at least one remote participant is determined based on institute's captured image.The interested area of at least one of shown content is identified based on eye gaze angle information.Feedback is provided to exhibitor comprising the instruction in the interested area of at least one identified.
Description
Background technique
The virtual interacting of personalization, such as video conference are increasingly being used for completing various tasks, such as carry out
Teleconference.Video conference enables the participant being located at different location to hand over simultaneously via two-way video and audio transmission
Mutually.Video conference can be simple as the session between two participants being located at different location, or is related to being located at not
It with the discussion between many participants at place, and may include shared displaying content, such as video display or magic lantern
Piece.As high speed network connectivity is just becoming widely available at lower cost and more, and video capture and display technology at
This continues to reduce, and just becomes to become more and more popular by the video conference that network carries out between the participant in remote place.
Detailed description of the invention
Fig. 1 is the diagram illustrated according to an exemplary tele-conferencing system.
Fig. 2 is an exemplary block diagram for illustrating the remote computing device for tele-conferencing system shown in Fig. 1.
Fig. 3 is to illustrate the exemplary frame that equipment is calculated for the exhibitor of tele-conferencing system shown in Fig. 1
Figure.
Fig. 4 is the process illustrated according to an exemplary method for executing eye gaze angular estimation in teleconference
Figure.
Fig. 5 is illustrated according to an exemplary diagram in teleconference to exhibitor's offer feedback.
Fig. 6 is the diagram for illustrating the display according to an exemplary displaying content with the instruction of thermal map type.
Fig. 7 is to illustrate to be moved by force according to the eyes at any time of exemplary two participants for teleconference
The diagram of the chart of degree.
Fig. 8 is the flow chart for illustrating the method for providing feedback to exhibitor in teleconference.
Specific embodiment
In the following specific embodiments, with reference to attached drawing, the attached drawing forms a part of this paper, and wherein conduct
It illustrates and is shown in which that the particular example of present disclosure can be practiced.It is to be understood that can use other examples, and
Can make structural or logical changes without departing from scope of the present disclosure.Therefore, following specific embodiments should not be with
Restrictive sense is understood, and scope of the present disclosure be defined by the following claims.It is understood that institute herein
The various exemplary features stated can partially or entirely with combination with one another, unless specifically noted.
Some examples are related to being joined together by network to be used for multiple ginsengs of virtual interacting (such as teleconference)
With person.Teleconference as used herein is intended to refer to the interaction between at least two participants, wherein being not all of
Participant is all located at identical physical locations (i.e. at least one participant is located at long-range place).The participant of teleconference can be with
Using portable or non-portable computing device, such as, but not limited to personal computer, desktop computer, laptop computer,
Notebook computer, network computer, personal digital assistant (PDA), mobile device, handheld device or any other suitable
Calculating equipment.Some examples are related at least one exhibitor being joined together by network (such as internet) and more
A participant.It is pointed out that exhibitor is " participant " in the context of the teleconference of the property, wherein he or
She just interacts with other " participants ".
Some examples, which are directed to, pays attention to force information using gaze detection with the participant determined in teleconference, and to long-range
The exhibitor of meeting provides feedback.Sensitive interface (example is stared by using the calculating equipment with each participant is associated
Such as network cameras (webcam)) determine that meeting participant's stares angle information.Some example dependencies are in the net from consumer's grade
Network camera image obtained, and deal with free environment, may include different head poses, variable illumination with
And other factors.Some examples extract information from these images, such as detection facial landmark, end rotation, eye position and
Head angle.Eye information is determined and is input into convolutional neural networks (CNN) to extract feature, and the feature is used as
For the input of machine learning prediction module to determine eye gaze angle.Based on eye gaze angle information, in the content that is demonstrated
Interested area be identified and be provided as feedback to exhibitor.
Fig. 1 is the diagram illustrated according to an exemplary tele-conferencing system 100.System 100 is related to being respectively provided with phase
Multiple remote participants 102 (1) -102 (3) of associated remote computing device 104 (1) -104 (3) (are collectively referred to as remotely joining
With person 102) and associated exhibitor calculate equipment 108 exhibitor participant 106.Calculate equipment 104 and 108
It is communicatively coupled to each other via network 105.Calculate each of equipment 104/108 include teleconference application (such as
Lync, Skype, Webex, Google Hangouts), and video and audio stream are generated during teleconference, it is sent
To network 105, it is then supplied to each of other calculating equipment 104 and 108.
Calculating equipment 104 and 108 may include personal computer, desktop computer, personal digital assistant (PDA), movement
Equipment, handheld device or other types of calculating equipment.Network 105 can be cable network, wireless network or wired
With the combination of wireless network.In some instances, network 105 is computer network, may include that private network is (such as inline
Net) or public network (such as internet).System 100 can also be realized by using cloud computing framework.
Exhibitor participant 106 by network 105 communicated with remote participant 102 be used for virtual interacting (such as remotely
Meeting).Exhibitor participant 106 calculates equipment 108 using exhibitor and shows (such as the magic lantern of content 110 to transmit to network 105
Piece, text, image, video etc.).Remote participant 102 (1) -102 (3) uses 104 (1) -104 of remote computing device respectively
(3) come the displaying content 110 transmitted from the reception of network 105, and the displaying content 110 received is shown.In some examples
In, remote computing device 104 (1) -104 (3) is respectively that its associated remote participant 102 (1) -102 (3) determines based on eye
The feedback information 112 of eyeball angle of gaze, and the information 112 is transferred to exhibitor's meter via network 105 during teleconference
Calculate equipment 108.
Fig. 2 is an example for illustrating the remote computing device 104 for tele-conferencing system 100 shown in Fig. 1
Block diagram.Remote computing device 104 includes at least one processor 202, memory 204, input equipment 220, output equipment
222, display 224 and camera 226.Processor 202, memory 204, input equipment 220, output equipment 222, display
224 and camera 226 be communicably coupled to each other by communication link 218.Camera 226 can be embedded in display 224
Frame in, be assembled along at least one side of display 224, or be assembled in the room that display 224 is located therein
Suitable position in.
Input equipment 220 includes keyboard, mouse, data port and/or other in equipment 104 for entering information into
Suitable equipment.Output equipment 222 includes loudspeaker, data port and/or for from the other suitable of 104 output information of equipment
Equipment.
Processor 202 includes central processing unit (CPU) or another suitable processor.In one example, it stores
The storage of device 204 is executed by processor 202 with the machine readable instructions for operating equipment 104.Memory 204 includes volatibility
And/or any suitable combination of nonvolatile memory, such as random access memory (RAM), read-only memory (ROM),
The combination of flash memory and/or other suitable memories.These are the examples of non-transitory computer-readable storage media.
Memory 204 is non-temporary in the sense: it does not include temporary signal, but instead, it is deposited by least one
Reservoir component is constituted, to store the machine-executable instruction for executing technology described herein.
Memory 204 stores teleconference using 206, eye gaze angular estimation module 208 and the processing of eye gaze angle
With feedback generation module 210.Processor 202 executes teleconference using 206, eye gaze angular estimation module 208 and eyes
Angle of gaze processing and the instruction of feedback generation module 210 are to execute technology described herein.Notice teleconference application
206, some in the functionality of eye gaze angular estimation module 208 and the processing of eye gaze angle and feedback generation module 210
Or it can all be realized by using cloud computing resources.
Teleconference module 206 allows the user of remote computing device 104 to participate in teleconference, and in display 224
On check the displaying content 110 (Fig. 1) for teleconference.During teleconference, the capture of camera 226 calculates equipment 104
The image of user is provided to eye gaze angular estimation module 208.Captured image based on user, eye gaze angle
Estimation module 208 constantly estimates the current eye angle of gaze of user during teleconference.The processing of eye gaze angle and feedback
Generation module 210 receives and processes the estimated eye gaze angular data generated by module 208, and generates feedback information
(such as feedback information 112 based on eye gaze angle, be shown in Fig. 1), the feedback information is transferred to exhibitor
Calculate equipment 108.
Fig. 3 is to illustrate to calculate one of equipment 108 for the exhibitor of tele-conferencing system 100 shown in Fig. 1 and show
The block diagram of example.Exhibitor calculates equipment 108 and sets including at least one processor 302, memory 304, input equipment 320, output
Standby 322, display 324 and camera 326.Processor 302, memory 304, input equipment 320, output equipment 322, display
324 and camera 326 be communicably coupled to each other by communication link 318.Camera 326 can be embedded in display 324
Frame in, be assembled along at least one side of display 324, or be assembled in the room that display 324 is located therein
Suitable position in.
Input equipment 320 includes keyboard, mouse, data port and/or other in equipment 108 for entering information into
Suitable equipment.Output equipment 322 includes loudspeaker, data port and/or for from the other suitable of 108 output information of equipment
Equipment.
Processor 302 includes central processing unit (CPU) or another suitable processor.In one example, it stores
The storage of device 304 is executed by processor 302 with the machine readable instructions for operating equipment 108.Memory 304 includes volatibility
And/or any suitable combination of nonvolatile memory, such as random access memory (RAM), read-only memory (ROM),
The combination of flash memory and/or other suitable memories.These are the examples of non-transitory computer-readable medium.Storage
Device 304 is non-temporary in the sense: it does not include temporary signal, but instead, by least one processor
Component is constituted, to store the machine-executable instruction for executing technology described herein.
Memory 304 stores teleconference using 306, feedback processing modules 308 and shows content 110.Processor 302 is held
Row teleconference using 306 and feedback processing modules 308 instruction to execute technology described herein.It is noted that long-range meeting
View can be come real using some or all of 306 and the functionality of feedback processing modules 308 by using cloud computing resources
It is existing.
Teleconference module 306 allows the user of remote computing device 108 to participate in teleconference, and to remote participant
102 show the displaying content 110 for being used for teleconference.Show that content 110 can be checked on display 324 by exhibitor.?
During teleconference, show that content 110 is exposed to remote participant 102, and the processing of feedback processing modules 308 is joined from long-range
With the received feedback information 112 based on eye gaze angle of person.In some instances, feedback processing modules 308 are in display 324
It is upper to provide instruction to identify the interested area for showing content 110 based on the feedback information 112 received.
Fig. 4 is illustrated according to an exemplary method 400 for the execution eye gaze angular estimation in teleconference
Flow chart.In one example, remote computing device 104 (Fig. 1) can execute method 400.In method 400 402 at,
Capture the image of the participant 102 in teleconference.Image can capture (Fig. 2) by camera 226.At 404, by estimation module
The head position of 208 participant 102 to estimate in captured image.At 406, caught by estimation module 208 to detect
The face in image obtained.At 408, left eye and right eye are detected in the face detected by estimation module 208.410
Place, by estimation module 208 by using the first convolutional neural networks trained using left eye information come from the left eye detected
Middle extraction fisrt feature collection.At 412, by estimation module 208 by using the second convolution mind trained using right eye information
Second feature collection is extracted from the right eye detected through network.It is characterized in the data by machine learning model for study.?
At 414, extracted fisrt feature collection is used as the input for the first machine learning prediction module, first machine learning
Prediction module is the part of estimation module 208, and the eye gaze angle value of its first estimation of output.It is extracted at 416
Second feature collection is used as the input for the second machine learning prediction module, and the second machine learning prediction module is estimation
The part of module 208, and the eye gaze angle value of its second estimation of output.At 418, by estimation module 208 by using
The eye gaze angle value that mean value (average value) Lai Zuhe first and second of described two values estimates, to generate the eye finally estimated
Eyeball stares angle value.Compared with the solution for using single eyes, by using two eyes and staring for its estimation is combined
The accuracy at angle and mean value increase output between them.At 420, feedback information is by module 210 based on finally estimating
Eye gaze angle value generates, and is provided to the exhibitor of teleconference.
Method 400 can be executed for each participant 102 in teleconference, be participated in determining for each such
The current eye of person 102 stares angle value.Method 400 can also continue to repeat to provide and each participant 102 to exhibitor 106
The relevant continuous update of attention focusing.The time of all participants 102 during teleconference notices that force information can also
To be generated and be provided to exhibitor 106.
According in an exemplary method 400, CNN is for extracting correlated characteristic, and other machine learning prediction
Module is used for the angle of gaze of output estimation.Some examples of method 400 use the network based on high-performance convolution, such as " VGG "
CNN framework is developed by Oxford University's visual geometric group (Visual Geometry Group), and provides ratio such as
The better ability in feature extraction of AlexNet framework.Other deep neural network frameworks can be used in the other examples of method 400.
Mobile by the eyes for tracking participant 102, system 100 (Fig. 1) can determine whether individual facing away from calculating and set
For 104 or his or her eyes whether are closed.System 100 can also determine that the eyes of all participants are mobile whether
It increases or decreases.In some instances, immediate feedback is provided to exhibitor during teleconference and calculates equipment 108, and
Mark including currently receiving one or more parts of the displaying content 110 of most attentions from participant 102, allows to open up
The person of showing 106 correspondingly adapts to speech and (such as increases or slow down the speed of displaying or check whether that anyone is problematic, or change
The other characteristics shown).By knowing specific attention focusing during teleconference or particular individual (or point of individual
Group) where focus on, exhibitor 106 can carry out personalization so that exhibitor 106 is set to the specific of target to content
The attention of audience maximizes.Exhibitor 106 can also be subsequent to adapt to and to enrich using the feedback after teleconference
It shows.
System 100 can also assess the attention of participant 102 along the time shaft of displaying.Time series includes conduct
The time flow of another layer of information, and allow the visualization of different information and mode.
Fig. 5 is illustrated according to an exemplary diagram in teleconference to the offer feedback of exhibitor 106.Long-range
Session shows that content 110 is displayed on exhibitor and calculates in equipment 108.In illustrated example, content 110 is shown
Including the multiple lantern slides 502 (1) -502 (3) (being collectively referred to as lantern slide 502) shown at any time.Lantern slide 502 (1) includes
Slide title 508, text 510 and image 512.Lantern slide 502 (2) includes slide title 514, image 516 and text
518.Lantern slide 502 (3) includes slide title 520 and text 522.
During the displaying of lantern slide 502, by exhibitor calculate equipment 108 receive for each participant 102 based on
The feedback information 112 at eye gaze angle.In illustrated example, based on the feedback information based on eye gaze angle received
112, exhibitor calculates equipment 108 and provides in the shown displaying content of the current attention focusing of each participant 102
Instruction.For example, indicator 504 can indicate the attention focusing of first participant 102 (1), and indicator 506 can indicate
The attention focusing of second participant 102 (2).As by the way that shown in the indicator 504 in Fig. 5, first participant 102 (1) is focused
The text 522 in the image 516 and lantern slide 502 (3) in text 510, lantern slide 502 (2) in lantern slide 502 (1)
On.As by shown in the indicator 506 in Fig. 5, second participant 102 (2) focus on image 512 in lantern slide 502 (1),
On the slide title 520 in image 516 and lantern slide 502 (3) in lantern slide 502 (2).
In some instances, during teleconference based on receive based on the feedback information 112 at eye gaze angle come
It is continually updated the positioning of indicator 504 and 506, to provide the current attention focusing phase with participant 102 to exhibitor 106
The immediate feedback of pass.
By assessing most of stare so where for target, exhibitor 106 can be prioritized the certain of his or her material
Segmentation, changes the position of certain images, and re-organized text to be to minimize attention loss, or increases and be considered more relevant
The visibility of content.It can be visualized by different measurements and stare information, the different measurement includes thermal map, the heat
Figure informs that for participant, which area is most interested at the given displaying moment by color gradient.
Fig. 6 is the figure for illustrating the display according to an exemplary displaying content 110 with thermal map type instruction 602
Solution.Thermal map type instruction 602 is generated based on " feedback information 112 based on eye gaze angle ", and by superimposition in exhibitor
It calculates in displaying content 110 shown in equipment 108.Indicate the specific sense in the shown displaying content 110 of 602 marks
The area of interest and the intensity of participant's interest.The intensity of variation can be indicated by different colors, and can be shown
Corresponding thermal map example 604 comprising change from the left end (minimum intensity) of legend 604 to the right end (maximum intensity) of legend 604
Color.
System 100 can also assess which participant 102 has optimum focusing or when viewing is given during teleconference
It is at most interrupted when displaying.By combining the feedback information 112 from multiple participants 102, exhibitor 106 can be marked
Know the meeting moment that everyone most focuses or most disperses.
Fig. 7 is to illustrate to be moved by force according to the eyes at any time of exemplary two participants for teleconference
The diagram of the chart of degree.Chart 702 indicates the mobile intensity of the eyes of first participant 102 (1), and chart 704 indicates second
The mobile intensity of the eyes of participant 102 (2).Vertical axis in chart 702 and 704 indicates the mobile intensity of eyes, and chart 702
With the horizontal axis plots time in 704.During the period 706 and 708, exist for both participant 102 (1) and 102 (2)
Relative high levels the mobile intensity of eyes, instruction specific content shown by the time may be that dispersion attention is waited
Choosing.In contrast, during the period 710 and 712, for example, the mobile intensity of the eyes of participant 102 (1) and 102 (2) is lower,
Potentially indicate the attention of higher level.Chart 702 further includes the mobile intensity of eyes for first participant 102 (1)
The instruction 714 of average level, and chart 704 includes the average level for the mobile intensity of eyes of second participant 102 (2)
Instruction 716.
One example provides the method for feedback to exhibitor in teleconference.Fig. 8 is illustrated in teleconference
The flow chart of the middle method 800 that feedback is provided to exhibitor.In method 800 802 at, by using with display exhibitor institute
The associated camera of calculating equipment of the content of displaying captures the image of the remote participant in teleconference.At 804, base
The eye gaze angle information at least one remote participant is determined in institute's captured image.It is solidifying based on eyes at 806
Viewing-angle information identifies the interested area of at least one of shown content.At 808, feedback, packet are provided to exhibitor
Include the instruction in the interested area of at least one identified.In one example, the determination, mark and offer are by least one
Processor executes.
In method 800, determine eye gaze angle information may include detect in institute's captured image described at least
The left eye and right eye of one remote participant, and by using at least one convolutional neural networks come from the left eye that detects and
The right eye detected extracts feature.At least one described convolutional neural networks in method 800 may include multiple convolutional Neurals
Network.Method 800 may include estimating that at least one eye is solidifying using machine learning prediction module, based on extracted feature
Visual angle value.Extraction feature in method 800 may further include by using the first convolution trained using left eye information
Network from the left eye that detects extracts fisrt feature collection, and by using the second convolution net trained using right eye information
Network from the right eye that detects extracts second feature collection.Method 800, which may further include, predicts mould using the first machine learning
Block estimates that first eye stares angle value based on extracted fisrt feature collection, and using the second machine learning prediction module,
The second eye gaze angle value is estimated based on extracted second feature collection.Method 800 may further include calculating First view
Eyeball stares the mean value of angle value and the second eye gaze angle value to determine the eye gaze angle value finally estimated.Method 800 can be into
One step includes that the instruction of the current attention focusing of each remote participant is provided to exhibitor.Method 800 can be further
Including generating the instruction of thermal map type during teleconference, it is most interested in for remote participant to exhibitor's mark
Shown displaying content regions.Method 800, which may further include based on eye gaze angle information, to be generated for described at least
The chart of the mobile intensity of eyes at any time of one remote participant.
Another example is used to show the displaying by teleconference the system comprises display for a kind of system
The content and camera that person is shown are used to capture the image of remote participant during teleconference.The system is into one
Step includes at least one processor, is used for: determining the eye gaze angle for remote participant based on institute's captured image
Value；At least one interested area is identified in shown content based on the eye gaze angle value；And remotely can
It exports and feeds back to exhibitor during view comprising the instruction in the interested area of at least one identified.
The system may include portable computing device, wherein the camera is integrated into portable computing device.
At least one described processor can detecte the eyes of the remote participant in institute's captured image, by using convolutional Neural net
Network from the Extract eyes feature detected, and using machine learning prediction module, based on extracted feature estimates eye
Eyeball stares angle value.
Another example is directed to a kind of non-transitory computer-readable storage media of store instruction, and described instruction is when by extremely
A few processor makes at least one described processor when execution: generation is remotely mentioned by the exhibitor in teleconference
The display of the displaying content of confession；Receive the image of the remote participant in teleconference；Needle is determined based on the image received
To the eye gaze angle information of remote participant；It is identified at least in shown displaying content based on eye gaze angle information
One interested area；And generation will be provided to the feedback information of exhibitor comprising at least one sense identified is emerging
The instruction in the area of interest.
The non-transitory computer-readable storage media can be stored further and such as be given an order: described instruction is when by described
At least one processor makes at least one described processor when execution: detecting remote participant in the image received
Eyes；By using convolutional neural networks come from the Extract eyes feature detected；And utilization machine learning prediction module,
Eye gaze angle value is estimated based on extracted feature.
Some examples disclosed herein in teleconference by using inexpensive component (such as network cameras) from
Each participant 102 provides the feedback information of valuable instant personalization to exhibitor 106.Some examples can be dependent on commonly
Laptop computer camera, be arranged without additional hardware or environment, and provide for staring note in teleconference
The cost-effective solution of power of anticipating detection.Some examples disclosed herein are (such as empty without using the hardware of specialization
Quasi- reality headphone or depth camera) attention force follow-up is executed, and it is not related to special installation space, such as from hard
The minimum range of part, lighting condition etc..Such additional specialized hardware increases the cost of solution, and can limit
The portability and practicality of solution, and possibly cause the discomfort of participant.Some examples disclosed herein
It can be related to the participant at various diverse geographic locations, such as relative to following solution: the solution is related to object
Reason ground is present in all users in same conference room and under equal illumination constraint, and the solution is based on room
Size, capture device or the number that participant is limited by other factors.
Some examples use image procossing convolutional neural networks model, are automatically detecting for determining eye gaze angle
Classification/recurrence task correlated characteristic in be efficient.Some examples capture the eye gaze angle of each participant, and mention
For the feedback of customization, such as to the indicative information of the attention of particular participant during teleconference or which lantern slide
(or region of specific lantern slide) receives the mark of more attentions from participant during teleconference.
Although specific example has been illustrated and described herein, various interchangeable and/or equivalent reality
Shown in existing mode can substitute and described particular example without departing from scope of the present disclosure.It is intended to cover this
Any adaptation of particular example discussed in text or modification.Thus, it is intended that present disclosure only by claim and its
Equivalent limits.
Claims (15)
1. a method of it is fed back for being provided in teleconference to exhibitor, comprising:
It is captured in teleconference by using camera associated with the calculating equipment of content that is shown of display exhibitor
The image of remote participant；
The eye gaze angle information at least one remote participant is determined based on institute's captured image；
At least one interested area is identified in shown content based on eye gaze angle information；
Feedback is provided to exhibitor comprising the instruction in the interested area of at least one identified；And
Wherein the determination, mark and offer are executed by least one processor.
2. according to the method described in claim 1, wherein determining that eye gaze angle information includes:
The left eye and right eye of at least one remote participant are detected in institute's captured image；And
Feature is extracted from the left eye detected and the right eye detected by using at least one convolutional neural networks.
3. according to the method described in claim 2, wherein at least one described convolutional neural networks include multiple convolutional Neural nets
Network.
4. according to the method described in claim 2, and further comprise:
At least one eye gaze angle value is estimated using machine learning prediction module, based on extracted feature.
5. according to the method described in claim 2, wherein extraction feature further comprises:
Fisrt feature collection is extracted from the left eye detected by using the first convolutional network trained using left eye information；With
And
Second feature collection is extracted from the right eye detected by using the second convolutional network trained using right eye information.
6. according to the method described in claim 5, and further comprise:
Estimate that first eye stares angle value using the first machine learning prediction module, based on extracted fisrt feature collection；With
And
Using the second machine learning prediction module, the second eye gaze angle value is estimated based on extracted second feature collection.
7. according to the method described in claim 6, and further comprise:
It calculates first eye and stares the mean value of angle value and the second eye gaze angle value to determine the eye gaze angle value finally estimated.
8. according to the method described in claim 1, and further comprise:
The instruction of the current attention focusing of each remote participant is provided to exhibitor.
9. according to the method described in claim 1, and further comprise:
The instruction of thermal map type is generated during teleconference, is most interested in for remote participant to exhibitor's mark
Shown displaying content regions.
10. according to the method described in claim 1, and further comprise:
The figure of the mobile intensity of eyes at any time of at least one remote participant is generated based on eye gaze angle information
Table.
11. a kind of system, comprising:
Display shows content for showing as provided by the exhibitor of teleconference；
Camera, for capturing the image of the remote participant in teleconference；And
At least one processor, is used for:
The eye gaze angle value for remote participant is determined based on the image captured；
At least one interested area is identified in shown displaying content based on eye gaze angle value；And
It exports and feeds back to exhibitor during teleconference comprising the instruction in the interested area of at least one identified.
12. system according to claim 11, wherein the system comprises portable computing device, and the camera quilt
It is integrated into portable computing device.
13. system according to claim 11, wherein at least one described processor detects far in institute's captured image
The eyes of journey participant by using convolutional neural networks come from the Extract eyes feature detected, and utilize machine learning
Prediction module estimates eye gaze angle value based on extracted feature.
14. a kind of non-transitory computer-readable storage media of store instruction, described instruction is worked as to be held by least one processor
Make at least one described processor when row:
The display of the displaying content remotely provided by the exhibitor in teleconference is provided；
Receive the image of the remote participant in teleconference；
Based on received image determine the eye gaze angle information for remote participant；
At least one interested area is identified in shown displaying content based on eye gaze angle information；And
The feedback information of exhibitor will be provided to by generating comprising the instruction in the interested area of at least one identified.
15. non-transitory computer-readable storage media according to claim 14, and further store instruction, described
Instruction makes at least one described processor when being executed by least one described processor:
The eyes of remote participant are detected in the image received；
By using convolutional neural networks come from the Extract eyes feature detected；And
Using machine learning prediction module, eye gaze angle value is estimated based on extracted feature.
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| PCT/US2017/014085 WO2018136063A1 (en) | 2017-01-19 | 2017-01-19 | Eye gaze angle feedback in a remote meeting |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN110268370A true CN110268370A (en) | 2019-09-20 |
Family
ID=62908651
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201780084186.5A Pending CN110268370A (en) | 2017-01-19 | 2017-01-19 | Eye gaze angle feedback in teleconference |
Country Status (3)
| Country | Link |
|---|---|
| EP (1) | EP3548996B1 (en) |
| CN (1) | CN110268370A (en) |
| WO (1) | WO2018136063A1 (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114679437A (en) * | 2022-03-11 | 2022-06-28 | 阿里巴巴（中国）有限公司 | Teleconferencing method, data interaction method, device and computer storage medium |
Families Citing this family (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111277786B (en) * | 2020-02-28 | 2021-07-09 | 中国科学院上海微系统与信息技术研究所 | Image display system and image display method for correctly conveying the direction of human face line of sight |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102547123A (en) * | 2012-01-05 | 2012-07-04 | 天津师范大学 | Self-adapting sightline tracking system and method based on face recognition technology |
| WO2012132959A1 (en) * | 2011-03-25 | 2012-10-04 | オリンパス株式会社 | Visual display device |
| US20140184550A1 (en) * | 2011-09-07 | 2014-07-03 | Tandemlaunch Technologies Inc. | System and Method for Using Eye Gaze Information to Enhance Interactions |
| CN105009598A (en) * | 2013-03-15 | 2015-10-28 | 索尼公司 | Device for acquisition of viewer interest when viewing content |
| US20160073054A1 (en) * | 2014-09-05 | 2016-03-10 | Avaya Inc. | System and method for determining conference participation |
| CN105407791A (en) * | 2013-06-25 | 2016-03-16 | 微软技术许可有限责任公司 | Eye tracking via depth camera |
| CN105787478A (en) * | 2016-04-14 | 2016-07-20 | 中南大学 | Face direction change recognition method based on neural network and sensitivity parameter |
Family Cites Families (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US8401248B1 (en) * | 2008-12-30 | 2013-03-19 | Videomining Corporation | Method and system for measuring emotional and attentional response to dynamic digital media content |
| ES2940907T3 (en) * | 2014-06-12 | 2023-05-12 | Sr Labs S R L | Calibration device and method for an eye tracker and eye monitoring equipment comprising said calibration device |
| US9547821B1 (en) * | 2016-02-04 | 2017-01-17 | International Business Machines Corporation | Deep learning for algorithm portfolios |
| CN105760834A (en) * | 2016-02-14 | 2016-07-13 | 北京飞搜科技有限公司 | Face feature point locating method |
-
2017
- 2017-01-19 EP EP17892952.7A patent/EP3548996B1/en active Active
- 2017-01-19 CN CN201780084186.5A patent/CN110268370A/en active Pending
- 2017-01-19 WO PCT/US2017/014085 patent/WO2018136063A1/en not_active Ceased
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2012132959A1 (en) * | 2011-03-25 | 2012-10-04 | オリンパス株式会社 | Visual display device |
| US20140184550A1 (en) * | 2011-09-07 | 2014-07-03 | Tandemlaunch Technologies Inc. | System and Method for Using Eye Gaze Information to Enhance Interactions |
| CN102547123A (en) * | 2012-01-05 | 2012-07-04 | 天津师范大学 | Self-adapting sightline tracking system and method based on face recognition technology |
| CN105009598A (en) * | 2013-03-15 | 2015-10-28 | 索尼公司 | Device for acquisition of viewer interest when viewing content |
| CN105407791A (en) * | 2013-06-25 | 2016-03-16 | 微软技术许可有限责任公司 | Eye tracking via depth camera |
| US20160073054A1 (en) * | 2014-09-05 | 2016-03-10 | Avaya Inc. | System and method for determining conference participation |
| CN105787478A (en) * | 2016-04-14 | 2016-07-20 | 中南大学 | Face direction change recognition method based on neural network and sensitivity parameter |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114679437A (en) * | 2022-03-11 | 2022-06-28 | 阿里巴巴（中国）有限公司 | Teleconferencing method, data interaction method, device and computer storage medium |
| CN114679437B (en) * | 2022-03-11 | 2024-12-06 | 阿里巴巴（中国）有限公司 | Remote conference method, data interaction method, device and computer storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| EP3548996A4 (en) | 2020-07-15 |
| WO2018136063A1 (en) | 2018-07-26 |
| EP3548996A1 (en) | 2019-10-09 |
| EP3548996B1 (en) | 2024-10-09 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US10868999B2 (en) | Eye gaze angle feedback in a remote meeting | |
| TWI615776B (en) | Virtual message establishment method, search method and application system for moving objects | |
| JP7225631B2 (en) | Image processing device, camera device, and image processing method | |
| KR102506341B1 (en) | Devices, systems and methods of virtualizing a mirror | |
| US9571726B2 (en) | Generating attention information from photos | |
| JP6250819B2 (en) | Image capture feedback | |
| KR20220149503A (en) | Image capturing method and apparatus, electronic device and computer readable storage medium | |
| JP2017507557A (en) | Process for improving the quality of experience for users who view high-definition video streams on their devices | |
| CN107148628B (en) | System and method for feature-based authentication | |
| US20220214797A1 (en) | Virtual image control method, apparatus, electronic device and storage medium | |
| Anagnostopoulos et al. | Gaze-Informed location-based services | |
| Balata et al. | Blindcamera: Central and golden-ratio composition for blind photographers | |
| JP2012151547A5 (en) | ||
| US11830129B2 (en) | Object relighting using neural networks | |
| CN107018330A (en) | A kind of guidance method and device of taking pictures in real time | |
| US20140177906A1 (en) | Generating static scenes | |
| KR20190088478A (en) | Engagement measurement system | |
| CN112702521A (en) | Image shooting method and device, electronic equipment and computer readable storage medium | |
| US9141191B2 (en) | Capturing photos without a camera | |
| EP4300446A1 (en) | Methods and systems for detecting fraud during biometric identity verification | |
| CN110268370A (en) | Eye gaze angle feedback in teleconference | |
| WO2016192467A1 (en) | Method and device for playing videos | |
| EP3863275B1 (en) | System and method for smart image-capturing | |
| Metsiritrakul et al. | UP2U: program for raising awareness of phubbing problem with stimulating social interaction in public using augmented reality and computer vision | |
| US8994834B2 (en) | Capturing photos |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |