JP2002158984A - Multicast video conference terminal device, video conference system, and control method for video conference system - Google Patents

Abstract

(57) [Problem] To provide a video conference terminal device capable of avoiding collision of audio even when multicast is used. An audio signal is digitized by an audio input unit, encoded by an audio encoding unit, and transmitted as encoded audio data. The utterance information processing unit 3 generates utterance information based on the voice level data 102 from the voice input unit 1, the lip movement information 107 from the lip movement determination unit 7, and the utterance right information 112 from the utterance right token processing unit 13. 1
13 is transmitted. The encoded audio data 116 from another terminal device is supplied to the audio decoding unit 8, the mixer unit 9, the audio output unit 10
Is sent to the speaker via. The mixer unit 9 selectively synthesizes only the audio data transmitted from the terminal device indicated by the utterance information 115 from the utterance information receiving unit 14.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a video conference terminal device, a video conference system, and a control method of the video conference system, and more particularly, to a video conference terminal device for preventing a speaker of a conference from colliding on the system. The present invention relates to a video conference system and a control method of the video conference system.

[0002]

2. Description of the Related Art As a typical configuration example of a multipoint video conference system using video and audio, a multipoint control unit (hereinafter sometimes simply referred to as "MCU") is used. Some use an intra-group broadcast function called multicast.

When using an MCU, as shown in FIG. 5, a video conference terminal device 32-1 installed at a plurality of points is used.
To 32-n are connected to the MCU 31 and the MCU 31
Receives audio and video from each of the terminal devices 32-1 to 32-n, synthesizes them, and
By transmitting to -n, a video conference can be realized. The MCU 31 and the terminal devices 32-1 to 32-32
n may be connected via a network.

When using multicast, as shown in FIG. 6, audio and video from each terminal device 41-1 to 41-n are transmitted to all other terminal devices 41-1 to 41-n.
−n via the network 42, and the terminal devices 41-1 to 41-n can realize a video conference by reproducing the received audio and video individually or in combination.

[0005] In a multipoint video conference system, delays in audio and video are unavoidable due to limitations in communication capacity.
The effect of the delay is remarkable in voice, and causes inconvenience that utterances from a plurality of terminals collide. In other words, even if one terminal starts to speak, a certain delay occurs before the sound is reproduced on the other terminal, so that the speaker on the other terminal assumes that there is no other speaker. You may start speaking. As a result, voices may collide with each other, making it difficult to hear, and the speakers of both terminal devices may stop speaking, and the conference may not proceed smoothly. In a system that uses an MCU, the MCU
However, such inconvenience is avoided by controlling the speech timing of each terminal device.

[0006]

However, when multicast is used, the above disadvantages cannot be avoided since each terminal device transmits independently. The present invention is directed to a video conference terminal device that can avoid collision of audio, particularly a collision at the start of speaking even when using multicast, a video conference system using such a video conference terminal device, and a video conference system. It is an object to provide a control method.

[0007]

SUMMARY OF THE INVENTION A multicast video conference terminal apparatus of the present invention is used in a multipoint video conference system for transmitting and receiving audio information and video information between multipoints. An utterance information processing unit that transmits utterance information indicating that information is valid, and an audio information reproduction processing unit that performs selective reproduction of audio information received based on the utterance information from another multicast video conference terminal device, A voice token processing unit for performing transmission and reception processing of a voice token circulated between a plurality of multicast video conference terminal devices participating in the video conference; andthe voice information processing unit, when holding the voice token, Outputting the utterance information when an utterance voice input of the terminal device user is provided, wherein the utterance right token processing unit includes the utterance information When the management unit transmits the utterance information, the circulating of the floor token is stopped, and the utterance information is being received from another multicast video conference terminal device, and the patrol of the floor token is stopped. Is continued for a certain period of time, the circulation of the floor right token is restarted between the other multicast video conference terminals except the multicast video conference terminal transmitting the utterance information.

[0008] Further, in the case where the speech of the terminal user is input and the speech information is not output, a warning indicating that the speech is not valid is output.

The utterance information indicating that the transmitted voice information is valid is transmitted when the voice input of the terminal device user has been stopped for a certain period of time.

Another multicast video conference terminal of the present invention is used in a multipoint video conference system for transmitting and receiving audio information and video information between multipoints, and is transmitting audio information. A voice information processing unit that transmits voice information indicating the following, a voice information transmission control unit that transmits voice information only during the voice information transmission, and a plurality of multicast video conference terminal devices participating in the video conference. A voice right token processing unit for performing a transmission / reception process of the voice right token, wherein the voice information processing unit is configured to output the voice information when the terminal device user receives a voice voice input while holding the voice right token. The speaking right token processing unit, when the utterance information processing unit transmits the utterance information, stops the circulation of the speaking right token Also, while receiving the utterance information from another multicast video conference terminal device, when the stop of the circulation of the floor right token has continued for a certain period, the multicast video conference terminal device transmitting the utterance information is The loop of the floor token is resumed between the other multicast video conference terminal devices except the above.

[0011] Further, in a case where the voice of the terminal device user has been input and the voice information is not output, a warning indicating that the voice information has not been transmitted is output.

The speech information indicating that the speech information is being transmitted is transmitted when the speech input by the user of the terminal device has been stopped for a certain period of time.

Further, the warning is performed by displaying the warning in the field of view of the speaker.

[0014] The multicast video conference terminal device of the present invention may also determine whether or not the terminal device user has input a speech voice, by determining an input level of an input voice signal to the terminal device and the utilization obtained from an input video signal. This is performed based on the lip movement information of the person.

[0015] The circulation of the floor token is performed by unicast or multicast.

[0016] Further, the fixed period until the circulation of the floor right token is resumed is at least the maximum delay time from the time of voice input to the time of reproduction on another multicast video conference terminal or the maximum delay time. It is the time obtained by adding the time that the contents can be grasped.

A video conference system according to the present invention comprises a plurality of the above-mentioned multicast video conference terminal devices connected to each other.

Further, the interconnection uses a network.

A method of controlling a video conference system according to the present invention uses a plurality of multicast video conference terminal devices, wherein the floor right token for controlling the floor of the plurality of multicast video conference terminal devices is transmitted to the terminal. By circulating between the devices, the utterance from the terminal device having the floor token is stopped by the utterance from the terminal device for a certain period of time, and after the lapse of the certain period, between the terminal devices other than the terminal device having the utterance This restarts the circulation of the floor token.

[0020]

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Embodiments of the present invention will be described below with reference to FIGS.

(First Embodiment) FIG. 1 is a schematic functional block diagram of a first embodiment of a multicast video conference terminal device (hereinafter sometimes simply referred to as a "terminal device") of the present invention. FIG. This terminal device is used by being connected to a network as in the system shown in FIG. 6, and receives an audio signal 100 from a microphone, a video signal 104 from a camera, and encoded audio data 1.
03 and the video data 108 are transmitted to another terminal device via the network. Further, it receives encoded audio data 116 and video data 120 from another terminal device via a network, and
6 is transmitted to the speaker, and the video data 1
The video signal 122 based on the video signal 20 is output to a display monitor. Furthermore, while transmitting and receiving the floor right tokens 110 and 111, the terminal device transmits voice information 113 indicating that the terminal device holds the floor and outputs voice information to another terminal device. The same utterance information 114 is received.

An audio signal 100 input from a microphone is digitized by an audio input unit 1, and digitized audio data 101 is sent to an audio encoding unit 2. The audio encoding unit 2 encodes the audio data 101 by a predetermined encoding method, and transmits the encoded audio data 101 as encoded audio data 103. Further, the voice input unit 1 sends voice level data 102 indicating the voice level of the voice signal from the microphone to the utterance information processing unit 3.

The video signal 104 input from the camera is digitized by the video input unit 4, and the digitized video data 105 is encoded by the video encoding unit 5 by a predetermined encoding method. Then, it is transmitted as encoded video data 108. The video data 105 is also sent to the lip extracting unit 6, and the lip is extracted. The extracted lip image data 106 is sent to the lip movement discriminating unit 7 and
When a movement of a predetermined size is detected, the lip movement information 10
7 is sent to the utterance information processing unit 3. The movement of the lips is performed by detecting changes in a plurality of lip image frames.

The utterance information processing unit 3 receives the utterance level data 102 from the voice input unit 1, the lip movement information 107 from the lip movement discriminating unit 7, and the utterance right information 112 from the utterance right token processing unit 13 described later. The utterance information 113 is transmitted based on the utterance information. The speech information 113 indicates to other terminal devices that the encoded speech data 103 transmitted from the speech encoding unit 2 is valid, and that the speaker holds the right to speak and includes speech information during speech. It is shown.

The utterance information 113 is also sent to the floor token processing unit 13 to stop the circulation of the floor token. The utterance information 113 is also sent to the video encoding unit 5 and changes the video encoding process. The video coding process is changed by changing the coding portion, for example, by changing the image of the speaker to a zoomed image or by increasing the frame rate from the time of non-speech. The reason for changing the video encoding process is to make it easy to distinguish the video of the terminal device that is speaking from the video of the terminal device that is not speaking, and to easily identify the speaker.

The coded voice data 103 and the utterance information 11 transferred to another terminal device via the network
3, and the encoded video data 108 include the ID of the terminal device.
Is added.

The floor token processing unit 13 receives the token 110 circulating through a plurality of terminal devices participating in the video conference at a predetermined timing, and transmits the token 111. While the token is being acquired, the right to speak 112 is output to the speech information processing unit 3. When the voice is detected during the acquisition of the token and the voice information 113 is transmitted, the voice right token processing unit
Stop sending. Therefore, as will be described later, at least for a certain period, the other terminal cannot obtain the right to speak, and the uttered voice is transmitted to the other terminal device as coded voice data, and voice information with less collision and noise is generated. Will be played.

As a method of circulating the floor token, there is a unicast method as shown in FIG. 2A or a multicast method as shown in FIG. 2B. The unicast method is to receive a floor token from a predetermined terminal device and transmit the floor token to a predetermined terminal device. For example, if the terminal device of FIG. 1 is a terminal device C that circulates by unicast, the terminal device B receives the floor right token 110 from the terminal device B, and
The floor token 111 is transmitted. Unicast transmission has a small amount of communication and a simple configuration. However, when one terminal device does not operate normally, a token is not transmitted at all and the system is completely stopped. On the other hand, the multicast method sends a token to all terminal devices,
The example of FIG. 2B shows a state in which the terminal device B transmits that "the token has been turned from B to C" to all other terminal devices. This method is resistant to network failures, and all terminal devices can grasp the current state of the system, so that more complicated control can be handled.

Encoded audio data 11 from another terminal device
6 is sent to a speaker via an audio decoding unit 8, a mixer unit 9 and an audio output unit 10. The utterance information receiving unit 14
Then, the utterance information 114 transmitted by another terminal device is received. The encoded audio data 116 is decoded by the audio decoding unit 8 and sent to the mixer unit 9 to synthesize audio signals from a plurality of terminal devices.
Only the voice data transmitted from the terminal device indicated by the utterance information 115 from No. 4 is selectively synthesized. Therefore,
Therefore, since only the audio signal obtained from the effective encoded audio data of the speaker drives the speaker via the audio output unit 10, noise is effectively reduced and the performance of the video conference terminal is improved.

With such a configuration, it is possible to eliminate the collision of the voice, but since the speaking right token does not circulate and other speakers cannot speak, the speaking token processing unit performs the speaking based on a certain standard. Resumes patrol of rights token.

The circulation of the floor token will be described in more detail with reference to FIG. The figure shows an example of patrol by unicast. FIG. 3A shows a case where no one is speaking, and the speaking right token circulates in the order of A, B, C, D, E, A,. FIG.
(b) is a case in which C speaks while acquiring the speaking right token. At this time, the circulation of the speaking right token is stopped, and an utterance signal is transmitted from C to another terminal device. FIG. 3C shows a state where the circulation of the floor token has been resumed. In this case, while A holds the right to speak and continues to speak, D, E, A,
B, D,... The patrol is resumed by the terminal device that is to acquire the invention right token next to the terminal device that has stopped the floor right token, in this case, the floor right token processing unit 13 of D. D's voice token processing unit,
The fact that the floor right token from C is not circulated and the fact that the voice information from C has been transmitted allow the user to recognize that the floor right token is stopped at C. If this state continues for a certain period of time, the floor token is
To restart the tour. In this tour, B
Can recognize that the previous talk right token has been stopped at C, which is a normal talk right token transmission destination, and therefore sends the talk right token to D.

In consideration of prevention of collision of speech, a certain period of time until the resumption of the floor token must be at least longer than the maximum delay time from the time of voice input to the time of reproduction on another multicast video conference terminal. is there. Preferably, it is desirable that the maximum delay time be added to the time required to grasp the content of the utterance of the speaker.

When the circulation of the floor token is resumed in such a manner, voice information from a plurality of terminal devices may collide with each other. However, unlike the collision at the beginning of the speech, the conference does not proceed smoothly. .

(Second Embodiment) A second embodiment of a multicast video conference terminal device (hereinafter, sometimes simply referred to as a "terminal device") of the present invention is a first embodiment. And the transmission processing of the encoded audio data and the synthesis processing of the audio signal based on the received encoded audio data are different. FIG. 4 shows a main part of a schematic functional block diagram of the second embodiment. In the terminal device of FIG. 4, the encoded audio data from the audio encoding unit 2 is transmitted via the audio information transmission control unit 15. The voice information transmission control unit 15 controls the utterance information 11
3 transmits the encoded audio data from the audio encoding unit 2 only during transmission. With this configuration, since only the encoded audio data from the terminal device that is transmitting the utterance information is input to the audio decoding unit 8, the mixer unit 9 needs to perform the selective synthesis based on the received utterance information 115. Disappears.

According to the present invention, the voice information spoken only at the time of acquiring the floor right token can be reproduced on another terminal. Therefore, even if the speaker speaks, it is not recognized by another terminal device. Things happen. If such a situation occurs frequently, the smooth progress of the conference is hindered. Therefore, it is effective to warn the speaker that other terminal devices have not recognized the situation. This warning is issued when the sound level data 102 is greater than a certain value and the lip movement information 107
This is performed when the utterance information 113 is not transmitted even though is detected. The warning is preferably an image which can be given by various means, or a display in a visual field of a speaker, for example, a display on a display monitor.

[0036]

As is apparent from the above description, according to the present invention, even when multicast is used, it is possible to prevent a collision at the start of a utterance with a simple configuration and to hold a smooth multipoint video conference. can do.

[Brief description of the drawings]

FIG. 1 is a schematic functional block diagram of a first embodiment of the present invention.

FIG. 2 is a diagram for explaining a token patrol method;

FIG. 3 is a diagram illustrating a method of circulating a floor token

FIG. 4 is a functional block diagram of a main part according to a second embodiment of the present invention;

FIG. 5 is a configuration example of a multipoint video conference system using an MCU.

FIG. 6 is a configuration example of a multipoint video conference system using a multicast terminal device.

[Explanation of symbols]

 DESCRIPTION OF SYMBOLS 1 ... Audio input part 2 ... Audio encoding part 3 ... Speech information processing part 4 ... Video input part 5 ... Video encoding part 6 ... Lip extraction part 7 ... Lip movement discriminating unit 8 ... Audio decoding unit 9 ... Mixer unit 10 ... Audio output unit 11 ... Video decoding unit 12 ... Video output unit 13 ... Speaking right token processing unit 14 ... .Speech information receiving unit 15: Voice information transmission control unit

Continued on the front page (72) Inventor Toshio Oka 1006 Kazuma Kadoma, Osaka Pref.Matsushita Electric Industrial Co., Ltd. Person Satoshi Chiga 1006 Kazuma Kadoma, Kadoma City, Osaka Prefecture F-term in Matsushita Electric Industrial Co., Ltd.

Claims (15)

[Claims] 1. A multicast video conference terminal device used in a multipoint video conference system for transmitting and receiving audio information and video information between multipoints, wherein voice information indicating that the transmitted audio information is valid. An audio information processing unit for transmitting audio information, an audio information reproduction processing unit for selectively reproducing audio information received based on the audio information from another multicast video conference terminal device, and a plurality of multicast video conferences participating in the video conference. A voice token processing unit for performing a transmission / reception process of a voice token circulated between terminal devices, wherein the voice information processing unit receives a voice voice input of the terminal user when holding the voice token. And outputting the utterance information when the utterance information processing unit transmits the utterance information. Occasionally, to stop the circulation of the talk burst token, even during receiving the voicing information from other multicast teleconference terminal device, when the cyclic stopping of the voice token has continued for a predetermined period of time,
A multicast video conference terminal device for resuming the circulation of the floor token among other multicast video conference terminal devices except the multicast video conference terminal device transmitting the utterance information. 2. The multicast video conference terminal device according to claim 1, wherein the speech is not valid when the speech information of the terminal device user is input and the speech information is not output. Multicast video conference terminal device that outputs a warning that indicates. 3. The multicast video conference terminal device according to claim 1, wherein the utterance information indicating that the transmitted voice information is valid is a voice input by the terminal device user for a predetermined time. A multicast video conference terminal device whose transmission is stopped when it runs out. 4. A multicast video conference terminal device used in a multipoint video conference system for transmitting and receiving audio information and video information between multipoints, wherein said terminal transmits voice information indicating that voice information is being transmitted. A voice information transmission unit that transmits voice information only during the transmission of the voice information, and a transmission / reception process of a floor token that is circulated among a plurality of multicast video conference terminal devices participating in the video conference A speech right token processing unit for performing the following, wherein the speech information processing unit outputs the speech information when a speech voice input of the terminal device user is received at the time of holding the speech right token. The speaking right token processing unit, when the speaking information processing unit transmits the utterance information, stops the circulation of the speaking right token and performs another token. A receiving the voicing information from a multicasting teleconference terminal apparatus, when the cyclic stopping of the voice token has continued for a predetermined period of time,
A multicast video conference terminal device for resuming the circulation of the floor token among other multicast video conference terminal devices except the multicast video conference terminal device transmitting the utterance information. 5. The multicast video conference terminal device according to claim 4, wherein the voice information is transmitted when the voice information of the terminal device user is input and the voice information is not output. A multicast videoconferencing terminal device that outputs a warning indicating that there was no message. 6. The multicast video conference terminal device according to claim 4, wherein the utterance information indicating that the voice information is being transmitted is such that the terminal user has stopped inputting utterance voice for a certain period of time. A multicast teleconferencing terminal device whose transmission is to be stopped when. 7. The multicast video conference terminal device according to claim 2, wherein the warning is based on a display in a field of view of a speaker. 8. The multicast video conference terminal device according to claim 1, wherein the determination as to whether or not the terminal user has input a speech voice is performed based on the input voice signal to the terminal device. A multicast video conference terminal device which is performed based on an input level and lip movement information of the user obtained from an input video signal. 9. The multicast video conference terminal device according to claim 1, wherein the floor token is a multicast routed between the plurality of multicast video conference terminal devices by unicast. Video conference terminal device. 10. The multicast video conference terminal device according to claim 1, wherein the floor token is circulated between the plurality of multicast video conference terminal devices by multicast. Conference terminal device. 11. The multicast video conference terminal device according to claim 1, wherein at least a certain period of time from the time of voice input to another multicast video conference is continued until the circulation of the floor token is restarted. A multicast video conference terminal device that is the maximum delay time up to the point of playback on the terminal device. 12. The multicast video conference terminal device according to claim 11, wherein, for a certain period of time before resuming the circulation of the floor token, the maximum delay time is such that the content of the voice of the speaker can be grasped. Multicast video conference terminal device that is the added time. 13. A video conference system in which a plurality of the multicast video conference terminal devices according to claim 1 are connected to each other. 14. The video conference system according to claim 13, wherein the interconnection uses a network. 15. A method of controlling a video conference system using a plurality of multicast video conference terminal devices, wherein a talk right token for controlling a voice right of the plurality of multicast video conference terminal devices is circulated between the terminal devices. By means of a statement from the terminal device having the floor right token, the circulation of the floor right token is stopped for a certain period of time, and after the certain period of time has passed, between the terminal devices other than the terminal device that has said the line, A method for controlling a video conference system for resuming the circulation of right tokens.