JP2025075999A - COMMUNICATION SUPPORT METHOD, COMMUNICATION SUPPORT DEVICE, AND PROGRAM - Google Patents

Abstract

To provide a communication support method capable of reporting that participants on a distal end side and a proximal end side have intentions to utter.SOLUTION: A communication support method acquires images of all the participants on a proximal end side and a distal end side in a remote conference from image data captured with cameras, detects intention display information being an index for determining whether an utterance is intended, from the images of all the participants, estimates an utterance intention holding person who has an utterance intention from among all the participants on the basis of the intention display information, and displays the information of the utterance intension holding person on a display.SELECTED DRAWING: Figure 3

Description

One embodiment of the present invention relates to a communication support method, a communication support device, and a program.

Patent Document 1 discloses speech action detection in a web conference, which detects speech from input audio, detects preparatory actions (backchannels, upward and downward hand movements, nodding, body movements) from video and audio, assigns and retains a predetermined point for each action, and shares the presence or absence of speech and the point by transmitting it to other terminals, selects a user who has spoken on each terminal, superimposes a frame on the display area of that user, calculates the total point of preparatory actions within a predetermined effective time as the speech possibility point, selects the user with the highest point, and superimposes a blinking frame on the display area of that user.

JP 2011-118632 A

For example, when multiple near-end participants are having a conversation, even if a far-end participant wishes to speak, it is difficult for the multiple near-end participants to know of that intention. Therefore, it is difficult for the far-end participant to join the conversation.

One of the embodiments aims to provide a communication support method that allows far-end and near-end participants to inform each other that they wish to speak.

The communication support method obtains images of all participants in a remote conference, including the near-end and far-end, from video data captured by a camera, detects intention-expression information, which is an index for determining whether or not there is an intention to speak, estimates which of all the participants have the intention to speak based on the intention-expression information, and displays information about the person with the intention to speak on a display.

According to one embodiment of the present invention, far-end and near-end participants can signal their intention to speak.

FIG. 1 is a block diagram of a communication support system. 1 is a schematic elevation view of a room in which a communication support system is installed at a first location 1. FIG. 1 is a schematic elevation view of a room in which a communication support system is installed at a second location 2. FIG. 13 is a flowchart showing the operation of a communication support method. 1 is an example of a screen of a remote conference application program displayed on a display of a PC 30 at a first location 1. 1 is an example of a screen of a remote conference application program displayed on a display of a PC 30 at a first location 1. 13 is an example of a screen of a remote conference application program displayed on a display of a PC 30 at a second location 2. 13 is an example of a screen of a remote conference application program displayed on a display of a PC 30 at a second location 2. 13 is an example of a screen of a remote conference application program displayed on a display of a PC 30 at a second location 2. 13 is an example of a screen of a remote conference application program displayed on a display of a PC 30 at a second location 2. 13 is an example of a screen of a remote conference application program displayed on a display of a PC 30 at a second location 2. 13 is an example of a screen of a remote conference application program displayed on a display of a PC 30 at a second location 2.

Figure 1 is a block diagram of a communication support system. The communication support system includes a microphone 10, a camera 20, and a personal computer (PC) 30, which are installed at a first location 1 and a second location 2, respectively. The microphone 10 and the camera 20 are connected to the PC 30. The PC 30 at the first location 1 and the PC 30 at the second location 2 are connected via a network.

Figure 2 is a schematic elevation diagram of the room at the first point 1. As an example, the microphone 10 is installed on the ceiling of the room. The microphone 10 has a thin, rectangular parallelepiped housing. The camera 20 is installed on a desk. The desk is placed directly below the housing of the microphone 10. In the example of Figure 1, multiple participants (participants u1, u2) are located around the desk.

Figure 3 is a schematic elevation diagram of the room at the second point 2. As an example, the microphone 10 is installed on a desk. The microphone 10 has a thin, rectangular parallelepiped housing. The camera 20 is installed on the PC 30. In the example of Figure 2, participant u3 is located in front of the PC 30 around the desk.

In this embodiment, the microphone 10, camera 20, and PC 30 installed at the first location 1 and the second location 2, respectively, have the same configuration and the same functions.

The camera 20 captures an image of the participant. The camera 20 performs a predetermined signal processing on the video signal related to the captured image, and transmits the processed video signal to the PC 30. The camera 20 performs framing processing, for example, by panning, tilting, or zooming.

The microphone 10 acquires the voice of the participant. The microphone 10 performs a predetermined signal processing on the acquired sound signal. The microphone 10 is, for example, an array microphone having multiple microphone units. The microphone 10 performs, for example, beamforming directional processing on the acquired sound signal. Beamforming is, for example, a process of forming a sound collection beam with a phase aligned in the direction of the speaker by delay-and-sum processing and high sensitivity in the direction of the speaker. The microphone 10 may obtain directional information of the speaker's voice and perform processing to direct the sound collection beam in the direction of the speaker. The microphone 10 analyzes the sound signals acquired from the multiple microphone units to estimate the direction of arrival of the sound. The method of analyzing the sound signal may be any method, such as a cross-correlation method, a delay-and-sum method, or a MUSIC (Multiple Signal Classification) method. In the cross-correlation method, the microphone 10 calculates, for example, the cross-correlation of the sound signals of multiple microphones. For example, microphone 10 finds the cross-correlation peak of the sound signals of two microphones. Furthermore, microphone 10 finds the cross-correlation peak of the sound signals of two other microphones. Based on the multiple cross-correlation peaks calculated in this way, microphone 10 estimates the direction from which the sound is coming. This allows microphone 10 to find directional information about the speaker.

The microphone 10 transmits the processed audio signal to the PC 30. The PC 30 transmits the audio signal received from the microphone 10 and the video signal received from the camera 20 to the PC 30 on the far end side. The PC 30 may display the video signal received from the camera 20 on a display (not shown) of its own device.

The PC 30 also receives a video signal and an audio signal from the PC 30 on the far end. The PC 30 outputs the received video signal to a display device (not shown). The PC 30 also outputs the received audio signal to a speaker (not shown). In this way, the communication support system functions as a component of a remote conference system for holding conferences with remote locations.

The PC 30 is a general information processing device, and is equipped with a processor that performs various processes. The PC 30 also functions as a remote controller for operating the microphone 10 or the camera 20. The processor of the PC 30 functions as a control unit of the communication support system by reading out an operation program stored in a storage device such as an internal flash memory. Note that the program does not need to be stored in the flash memory of the PC 30. The PC 30 may download the program each time, for example, from a server or the like.

Figure 4 is a flowchart showing the operation of the communication support method. First, PC 30 acquires images of all participants in the remote conference, including the near-end and far-end, from video data captured by a camera (S11). For example, PC 30 at first point 1 acquires video data captured by camera 20 at first point 1 (hereinafter referred to as first video data), and receives video data captured by camera 20 at second point 2 (hereinafter referred to as second video data). PC 30 performs processing to recognize the speaker's face in each of the first video data and the second video data using a predetermined model, for example using a neural network, and acquires images of all participants.

Next, PC 30 detects intention information, which is an index for judging whether or not there is an intention to speak, from the acquired video of all participants (S12). For example, PC 30 prepares a predetermined trained model in which the relationship between the participant's video and the intention information is trained using a DNN (Deep Neural Network) or the like. PC 30 acquires video of multiple participants in the past just before they start speaking and video at the start of speaking as training data, and trains the predetermined model on the relationship between the video just before they start speaking and the intention information. PC 30 inputs the video of each participant into the trained predetermined model to obtain the intention information of each participant. In this case, the intention information is obtained as a probability value (degree of participation in the meeting) of, for example, 0 to 100%.

Next, PC30 estimates a participant who intends to speak from among all the participants based on the intention expression information (S13). For example, when the conference participation level of the intention expression information exceeds a predetermined threshold, PC30 determines that the corresponding participant is a participant who intends to speak.

Then, the PC 30 displays information about the person with the intention to speak on the display (S14). Figures 5 and 6 are examples of screens of a remote conference application program displayed on the display of the PC 30 at the first location 1. Figures 7 and 8 are examples of screens of a remote conference application program displayed on the display of the PC 30 at the second location 2. Figures 5 and 7 are examples of screens when no one has the intention to speak, and Figures 6 and 8 are examples of screens when it is determined that participant u1 has the intention to speak.

As an example of information about a participant who intends to speak, the PC 30 displays a rectangular image surrounding the participant u2, who intends to speak, superimposed on the position of the participant u2. This allows all participants to know that there is a participant who intends to speak.

Alternatively, the PC 30 may change the image to one that highlights the participant u2 who intends to speak, as shown in FIG. 9, as an example of information on the person who intends to speak. FIG. 9 is an example of a screen of a remote conference application program displayed on the display of the PC 30 at the second location 2. In the example of FIG. 9, the PC 30 performs a framing process to enlarge the image of the participant u2 who intends to speak so that the occupancy rate of the screen of the participant u2 who intends to speak is a predetermined ratio (e.g., 50%). Alternatively, the PC 30 may perform a framing process to enlarge the image of the current speaker (e.g., participant u1) and the participant u2 who intends to speak so that the occupancy rate of the screen of the participant u2 who intends to speak is a predetermined ratio (e.g., 50%). The PC 30 may also display an enlarged image of the current speaker (e.g., participant u1) and an enlarged image of the participant u2 who intends to speak, splitting the image into two images on one screen. The PC 30 may also switch between an enlarged image of the current speaker (e.g., participant u1) and an enlarged image of the participant u2 who intends to speak every predetermined time. At this time, the PC 30 may display a long enlarged image of the current speaker (e.g., participant u1) and a short enlarged image of participant u2 who intends to speak. Alternatively, if the PC 30 has estimated multiple participants who intend to speak, it may display a relatively long image of a participant who has a high level of participation in the conference.

Because the far-end participants are displayed on a display, it is difficult for the near-end participants to understand the intentions of the far-end participants. In particular, when multiple near-end participants are talking, it is very difficult for the far-end participants to join the conversation. However, according to the communication support method of this embodiment, it is possible to inform all participants on the far-end and near-end sides that one has the intention to speak. This allows participants to obtain a new customer experience in which the psychological burden of starting to speak is reduced. In addition, by having the far-end participants inform multiple near-end participants of their intention to speak, it is possible to obtain a new customer experience in which even remote participants can easily participate in the near-end conversation. In other words, the communication support method of this embodiment can create a situation for the far-end participants that is close to an environment in which they are actually meeting and talking at the near-end.

(Variation 1)
The PC 30 may detect the gaze information of the participants as intention expression information from the acquired video of the far-end participant and the near-end participant.

PC30 calculates the direction of gaze for each participant as gaze information, and determines the gaze destination of each participant. In other words, the gaze information is expressed as a directed graph representing the participant who is the source of the gaze and the participant who is the destination of the gaze. PC30 estimates the participant who intends to speak from among all participants based on the gaze information. For example, PC30 calculates the number of gazes for each participant based on the directed graph. A participant who is gazed at many times means that multiple other participants are paying attention to him/her. In other words, the number of gazes corresponds to a participant who has a high level of participation in the conference, such as the current speaker. Therefore, when the number of gazes exceeds a predetermined threshold, PC30 determines the corresponding participant as the participant who intends to speak. Alternatively, PC30 may determine the participant who is gazed at by the current speaker as the participant who intends to speak. In this way, the communication support method of variant 1 can estimate the participant who the current speaker wants to speak next.

(Variation 2)
The PC 30 may detect, as intention expression information, motion information indicating a specific motion of the participants from the acquired video of the far-end participant and the near-end participant. The specific motion is, for example, a nodding motion, a hand movement motion, etc. Such motions are likely to indicate an attempt to participate in the conversation.

Therefore, the PC 30 may detect the above-mentioned specific motion information as intention expression information (S12), and estimate from among all the participants who have the intention to speak based on the specific motion information (S13). For example, the PC 30 may determine that a participant who is performing a specific motion is a participant who has the intention to speak.

As mentioned above, the far-end participant is displayed on the display, making it difficult for the near-end participant to notice specific actions being made by the far-end participant. However, with the communication support method of variant example 2, it is possible to inform all participants on both the far-end and near-end sides that a specific action related to speech is being made.

(Variation 3)
The PC 30 may detect intention indication information from the voice information included in the voice signal (S12), and may infer from among all the participants who have the intention to speak based on the voice information (S13). For example, when the PC 30 recognizes a specific word (such as the name of a participant or the name of a place on the far-end side), it determines that the corresponding participant is the one who has the intention to speak.

In addition, for example, when multiple people speak at the same time, PC 30 may infer that the participant who subsequently stops speaking (the participant who gives way to speaking) is the person with the intention to speak.

Alternatively, PC 30 may record the conversation history and record the relationship of the participant who will speak next for each participant as a database. Alternatively, PC 30 may infer that the participant who has just spoken is the person who intends to speak next.

The PC 30 may also use a trained model that has trained the relationship between the conversation content and speech to estimate the next person who intends to speak in relation to the current conversation content. Alternatively, the PC 30 may record information such as role (facilitator, recorder, observer, etc.), knowledge, or skills as information for each participant, and estimate a participant who possesses information such as role, knowledge, or skills that corresponds to the current conversation content as the person who intends to speak.

As described above, the communication support method of variant 3 detects intention information from voice information. Since the communication support method estimates the person with the intention to speak rather than the speaker, a participant who only utters one word that does not affect the level of participation in the conference, such as "Yes" or "That's right," is not estimated as the person with the intention to speak. Therefore, the communication support method of variant 3 does not highlight or frame such speakers.

(Variation 4)
The PC 30 may score the conversation participation of the participants. The PC 30 may estimate, based on the scores of the scores, who has the intention to speak from among all the participants (S13).

For example, PC 30 finds the number of times each participant has received gazes based on the directed graph shown in variant example 1. A participant who has received a large number of gazes means that multiple other participants are paying attention to them. Therefore, PC 30 finds the number of times each participant has received gazes as a score. PC 30 may also find the score by multiplying the number of times each participant has received gazes by a specified weighting coefficient. The weighting coefficient, for example, gives a higher score to gazes from participants who have received a large number of gazes and a lower score to gazes from participants who have received a small number of gazes. A participant who is the target of the gaze of a participant who has received a large number of gazes is likely to be the next to speak, and is thus given a high score.

In addition, PC 30 may add the points of participants who perform specific actions as action information to the number of gazes. A specific action is, for example, a nodding action or a hand movement action. These actions are actions that are likely to indicate an attempt to join a conversation. Therefore, PC 30 adds points to participants who perform specific actions as action information.

When the score exceeds a predetermined threshold, PC30 judges the corresponding participant to be a person with the intention to speak. By adding the score as movement information to the gaze information as described above, PC30 can estimate with higher accuracy the person with the intention to speak.

The PC 30 may also display the scoring results as information about participants with the intention to speak. Figure 10 is an example of a screen of a remote conference application program displayed on the display of the PC 30 at the second location 2. In the example of Figure 10, the PC 30 displays the scoring results around the video of participants whose scores exceed a predetermined threshold and who are determined to be participants with the intention to speak. This allows all participants on the far-end and near-end sides to intuitively know the strength of the intention to speak of all other participants.

The PC 30 may also highlight multiple participants with high scores who intend to speak. Figure 11 shows an example of a screen of a remote conference application program displayed on the display of the PC 30 at the second location 2. In the example of Figure 11, the PC 30 superimposes and displays a rectangular image surrounding the video of a participant who has a score exceeding a predetermined threshold and who has been determined to be a participant with an intention to speak. This allows all participants on the far-end and near-end sides to know at a glance that there is a participant who intends to speak.

(Variation 5)
The PC 30 may estimate participants who have no intention of speaking based on the intention expression information. The PC 30 may also not display information about participants who have no intention of speaking. The PC 30 may also perform a process of masking the video of participants who have no intention of speaking.

For example, when PC 30 detects a particular movement or posture, such as looking down, it infers that the participant is one who has no intention of speaking. PC 30 may also infer a participant who receives few glances as one who has no intention of speaking. Alternatively, PC 30 may infer a participant who shows little change in gaze as one who has no intention of speaking.

The PC 30 may also estimate participants who have no intention of speaking based on the audio information. The PC 30 may estimate, for example, a participant who has not spoken for a predetermined period of time or more as a participant who has no intention of speaking. The PC 30 may also estimate, for example, a participant whose name has not been called as a participant who has no intention of speaking.

(Variation 6)
The PC 30 may identify the speaker from among all the participants based on the audio signal picked up by the microphone, and display information about the speaker and the person with the intention to speak on a display.

Figure 12 is an example of a screen of a remote conference application program displayed on the display of PC 30 at second location 2. PC 30 receives speaker direction information estimated by microphone 10. In the example of Figure 12, PC 30 displays a rectangular image that is superimposed on the video of the participant corresponding to the received speaker direction information. Also, in the example of Figure 12, PC 30 displays a rectangular image that is superimposed on the video of the participant corresponding to the person with the intention to speak. However, it is preferable that PC 30 displays the speaker information and the information of the person with the intention to speak in different formats. For example, in the example of Figure 12, PC 30 displays a dashed rectangular image that is superimposed on the participant corresponding to the person with the intention to speak.

This allows all participants to see at a glance who is currently speaking and who wants to speak.

The description of the present embodiment should be considered to be illustrative in all respects and not restrictive. The scope of the present invention is indicated by the claims, not by the above-described embodiments. Furthermore, the scope of the present invention includes the scope equivalent to the claims.

10: Microphone, 20: Camera, 30: PC

Claims (15)

From the video data captured by the camera, images of all participants in the remote conference, including the near-end and far-end, are acquired,
Detecting intention expression information, which is an index for determining whether or not the participant has an intention to speak, from the images of all the participants;
predicting a person who has an intention to speak from among all the participants based on the intention expression information;
displaying information about the person with the intention to speak on a display device;
Communication support methods. Detecting gaze information or motion information showing a specific movement of the participant from the video as the intention expression information;
The communication support method according to claim 1 . The communication support method according to claim 1 or 2, wherein the intention expression information is detected from the voice information included in the voice signal. Score the conversational participation of the participants;
Displaying the result of the scoring as the information.
The communication support method according to claim 1 or 2. and highlighting and displaying the plurality of speech intention holders who have high scores in the scoring.
The communication support method according to claim 4. Inferring participants who have no intention of speaking from the intention expression information;
Do not display information about participants who have no intention of speaking;
The communication support method according to claim 1 or 2. Identifying a speaker from among all the participants based on an audio signal picked up by a microphone;
displaying information about the speaker and the speech intention holder on a display device;
The communication support method according to claim 1 or 2. From the video data captured by the camera, images of all participants in the remote conference, including the near-end and far-end, are acquired,
Detecting intention expression information, which is an index for determining whether or not the participant has an intention to speak, from the images of all the participants;
predicting a person who has an intention to speak from among all the participants based on the intention expression information;
displaying information about the person with the intention to speak on a display device;
A communication support device comprising a processor. The processor detects gaze information or motion information indicating a specific movement of the participant from the video as the intention expression information.
The communication support device according to claim 8. The communication support device according to claim 8 or 9, wherein the processor detects the intention expression information from the voice information included in the voice signal. The processor,
Score the conversational participation of the participants;
displaying the result of the scoring as the information on the display device;
The communication support device according to claim 8 or 9. the processor highlights the plurality of speech intention holders who have high scores in the scoring and displays them on the display device.
The communication support device according to claim 11. The processor,
Inferring participants who have no intention of speaking from the intention expression information;
Do not display information about participants who have no intention of speaking;
The communication support device according to claim 8 or 9. The processor,
Identifying a speaker from among all the participants based on an audio signal picked up by a microphone;
displaying information about the speaker and the speech intention holder on a display device;
The communication support device according to claim 8 or 9. From the video data captured by the camera, images of all participants in the remote conference, including the near-end and far-end, are acquired,
Detecting intention expression information, which is an index for determining whether or not the participant has an intention to speak, from the images of all the participants;
predicting a person who has an intention to speak from among all the participants based on the intention expression information;
displaying information about the person with the intention to speak on a display device;
A program that causes an information processing device to execute processing.

Images