JP5436743B2 - Communication terminal device and communication control device - Google Patents

Description

  The present invention relates to a communication terminal device and a communication control device capable of making a call with a plurality of communication terminal devices.

In the conventional video conference system, as shown in the system configuration diagram of FIG. 1, a terminal device having a function capable of connecting to a WAN or the like uses the HVoIP / H320 VoIP protocol and uses an IP network or ISDN network. Then, the audio data and the video data are transferred to the server, and the server relays the video to each terminal device and transmits the audio data to each terminal device. There is also a system that creates and displays a document from conversation in a conference (see Patent Document 1).
JP-A-5-12246 Japanese Patent Laid-Open No. 11-154998 Japanese Patent Laid-Open No. 5-75745

  In the conventional video conference system, the audio of each terminal device is distributed with the same audio level. For example, in a conventional MCU (multi-point conference system), the speech spoken by each terminal device is synthesized as it is and distributed to each terminal device. When talking from each terminal device at the same time, since the sound level of each terminal device is the same, there is a problem that it is difficult to hear the sound of the terminal device being noticed. In addition, as a means for solving the problem, there is a method of delivering voices at different times when each terminal device has a conversation at the same time, but if each terminal device can hear conversation continuously at the same sound level, Since it is difficult to distinguish the voice of interest, there is a problem that it is missed.

  Patent Document 2 proposes a method of removing ambient noise and extracting only a speaker's voice. The purpose of making it easier to hear the voice of the person who speaks is the same, but it is not assumed that many people speak at the same time, and the same problem occurs when the voices overlap.

  Patent Document 3 proposes a method of detecting the mute state of a speaker who is talking on the phone and muting. The MCU has a function of detecting a terminal device in a silent state and silences the corresponding terminal device, but the same problem occurs when voices overlap.

  The present invention has been made in view of such problems, and an object of the present invention is to provide a communication terminal device and communication control that do not miss the voice from the communication terminal device of interest in N-to-N communication. To provide an apparatus.

In order to achieve the above object, a communication terminal device according to the present invention includes a transmission means for transmitting voice and image data in a communication terminal device capable of calling with a plurality of communication terminal devices, and the plurality of communication terminal devices. Receiving means for receiving the voice and image data, output means for outputting the voice and image data received by the receiving means, and specifying means for designating at least one communication terminal apparatus from the plurality of communication terminal apparatuses; The output of at least the image data received by the receiving means is different between when the designation means designates another communication terminal apparatus and when the other communication terminal apparatus designates its own communication terminal apparatus. and a control means for controlling the output of the output means, the control means, the other as specifying the other communication terminal apparatus by said specifying means At least image data received by the receiving means when the communication terminal device is not one of the cases where the communication terminal device is designated and when the communication terminal device is designated by the other communication terminal device The output of the output means is controlled so as to vary the output of .

Further, the communication control device of the present invention is a communication control device that controls communication between a plurality of communication terminal devices, receiving means for receiving audio and image data from the plurality of communication terminal devices, and receiving by the receiving means. A synthesizing unit that synthesizes the voice and image data, a transmission unit that transmits the voice and image data synthesized by the synthesizing unit to the plurality of communication terminal devices, and a first of the plurality of communication terminal devices. When the information indicating that the second communication terminal device among the plurality of communication terminal devices is designated is received from the communication terminal device, and the first communication terminal device is designated from the second communication terminal device The voice and the image data are synthesized so that at least the output of the image data in the first communication terminal apparatus is different from the case where the information to the effect is received, and is transmitted to the first communication terminal apparatus. And a control means for controlled so, the said control means, said from the first when the communication terminal device receives information to the effect that specifies the second communication terminal apparatus and the second communication terminal device When the information indicating that the first communication terminal apparatus is specified is not received, and when the information indicating that the first communication terminal apparatus is specified is received from the second communication terminal apparatus. The voice and the image data are combined so that at least the output of the image data in the first communication terminal apparatus is different, and control is performed so as to transmit to the first communication terminal apparatus .

The output control method for a communication terminal device according to the present invention includes a case where a predetermined communication terminal device is specified among a plurality of communication terminal devices, and a case where a self communication terminal device is specified from the predetermined communication terminal device. And controlling at least the output of the received image data to be different , both when the predetermined communication terminal device is designated and when the own communication terminal device is designated from the predetermined communication terminal device Control is performed so that the output of at least the received image data differs depending on whether the communication terminal device is specified from the predetermined communication terminal device .

The communication control method of the communication control device of the present invention is a communication control method of a communication control device for controlling communication between a plurality of communication terminal devices, from the first communication terminal device among the plurality of communication terminal devices, When receiving information indicating that the second communication terminal apparatus is designated from among the plurality of communication terminal apparatuses, and receiving information indicating that the first communication terminal apparatus is specified from the second communication terminal apparatus. In this case, the first communication terminal apparatus is controlled to synthesize voice and image data so that at least the output of the image data is different, and to transmit to the first communication terminal apparatus . Either when the information indicating that the second communication terminal device is designated is received from the communication terminal device or when the information indicating that the first communication terminal device is designated is received from the second communication terminal device No place And when the information indicating that the first communication terminal device is designated is received from the second communication terminal device, at least the output of the image data in the first communication terminal device is different from the sound and the image. Control is performed such that data is synthesized and transmitted to the first communication terminal device .

  In the present invention, in N-to-N communication, the voice data of the communication terminal device of interest is controlled to be at a relatively high level with respect to the voice of the entire communication terminal device, and the voices of the other communication terminal devices are relative to each other. Therefore, the voice from the communication terminal device of interest is not missed.

  Embodiments of the present invention will be described with reference to the drawings. In the N-to-N communication system according to the present invention, as shown in FIG. 1, a communication terminal device transfers audio data and video data to a server which is a communication control device using an IP network, and passes through the IP network. The relayed video and audio are distributed to each communication terminal device. In the present invention, when a user selects a communication terminal device that the user wants to pay attention to, the user of each communication terminal device has a conversation at the same time by performing control to emphasize the voice of the communication terminal device of interest compared to the voice of other communication terminal devices. Sometimes it will not be missed.

  FIG. 2 is a block diagram of the communication terminal apparatus of the present invention. The communication terminal device 1 includes a camera unit 2 that captures user images, a transmission / reception unit 3 that transmits and receives data using an IP network, information necessary for transferring IP data, user attention selection parameter information, audio A storage unit 4 for storing display methods of data and image data, a control unit 7 for processing the data according to settings stored in the storage unit 4 and controlling each unit, and an image controlled by the control unit 7 The display unit 5 is a data output unit that displays data, caption data, and the like, and an operation unit 6 that designates at least one communication terminal device from among a plurality of communication terminal devices.

  FIG. 3 is a block diagram of a server which is a communication control device that controls communication between a plurality of communication terminal devices. The server 11 stores a storage unit 13 that stores an IP address, a distribution method, a data control method, and the like of each terminal device, and a control unit 14 that processes the data of each terminal device according to settings stored in the storage unit 13. Receiving and transmitting data necessary for voice, video, and IP communication of each terminal device from the IP network, and distributing video, voice data, and data necessary for IP communication controlled by the control unit 14 to each terminal device The unit 12 is configured.

  FIG. 4 is a protocol configuration diagram of the present invention. In the protocol configuration diagram of FIG. 4, when comparing the conventional protocol configuration diagram of FIG. 5, the protocol layer of the attention selection parameter is added above the RTP (Real-time Transport Protocol), and it is used for encoding and decoding of audio and video. It can be seen that it is configured to influence.

  FIG. 6 is a configuration diagram of an attention selection header in which the above-described attention selection parameter is stored. Information of the attention selection header is stored in the storage unit 4 of the communication terminal device 1 and is transmitted to other communication terminal devices by a packet. The attention selection header information can be transmitted to other communication terminal devices by providing a special message for transmitting the attention selection header information or by providing an identifier of the attention selection header information in the header field of the packet. preferable.

  In FIG. 6, the terminal number indicates the number of the terminal device assigned to each terminal device, the attention number indicates the number of the terminal device selected by the terminal device, and the target number is the number of the terminal device. The number of the terminal device that is attracting attention is indicated, and the group number indicates to which conversation group the terminal device currently belongs. Here, since the header is divided into 4 bytes, a maximum of 32 people can be connected at the same time. The size of the header information does not necessarily have to be divided into 4 bytes.

  FIG. 7 is a diagram illustrating the state of the attention selection header of each terminal when terminal A and terminal B form a conversation group. Terminal A and terminal B form a conversation group when they pay attention to each other and the attention number matches the attention target number. When composing a conversation group, terminal A and terminal B have higher voice levels of the other party's voice. Between the terminal A and the terminal C, the terminal C pays attention to the terminal A, but since the terminal A does not pay attention to the terminal C, the voice level of the terminal A is high at the terminal C. At terminal A, the sound of terminal C can be heard at a slightly lower level.

  Next, a first embodiment of the present invention will be described. The first embodiment is a case where the volume is controlled by synthesizing voice at the server. FIG. 8 is a timing chart when the volume is controlled by synthesizing voices at the server.

  When the terminal device B is selected for attention in the terminal device A, the terminal device A adds the terminal device B to the attention number of the attention selection header shown in FIG. 6 and notifies the server of the addition request. The server delivers the synthesized voice data to the terminal device A so that the volume of the terminal device B is emphasized as compared with the other terminal devices, and further, the attention number of the attention selection header of the packet delivered from the terminal device A The terminal device B is confirmed, and the relay video data of the changed terminal device A and the voice data synthesized so that the volume of the terminal device A is emphasized compared to other terminal devices are distributed to the terminal device B. The terminal device A is added to the target number of the attention selection header to notify the terminal device B that the terminal device B has been noticed.

  In the terminal device B, the attention number of the attention selection header of the packet distributed from the server is confirmed, and it is determined whether the terminal device A is put in the conversation group. When permitted, that is, when the terminal device A is selected for attention, the terminal device B adds the terminal device A to the attention number of the attention selection header and notifies the server of the addition request. The server notifies the terminal device A that the terminal device A has received attention, and distributes the changed relay video of the terminal device B to the terminal device A. At this time, the terminal device B is added to the target number of the attention selection header of the terminal device A, and a group number indicating the same group as the terminal device is added. After that, the target number and the target number match and are in the same group, so the sound from the terminal device of interest is distributed at a high volume.

  In this state, when attention is paid from another terminal device, the voice is delivered with a sound that is lower than that of the terminal device being watched but has no hindrance to hear. For example, when the terminal device C pays attention to the plurality of terminal devices A and B, the synthesized voice data is distributed to the terminal devices A and B so that the volume of the terminal device C is emphasized as compared with other terminal devices. Audio from other terminal devices (other than A, B, and C) is distributed at a low volume (including silence). As described above, when information indicating that the own communication terminal device is designated is received from at least one communication terminal device among the plurality of communication terminal devices, the voice and / or image data of the designated communication terminal device is emphasized and output. To do. When the server synthesizes the sound, the distributed sound is reproduced as it is, so that it is not necessary to control the volume on the terminal device side.

  Next, a second embodiment of the present invention will be described. In the second embodiment, the volume is controlled on the terminal device side. FIG. 9 is a timing chart in the case where the volume is controlled on the terminal device side without synthesizing the voice at the server.

  When the terminal device B is selected for attention in the terminal device A, the terminal device A adds the terminal device B to the attention number of the attention selection header shown in FIG. 6 and notifies the server of the addition request. Thereafter, in the terminal device A, when the audio data of the terminal device B that is its attention number comes, the audio level is emphasized and the video of the terminal device B is changed. The server confirms the terminal device B of the attention number of the attention selection header of the packet distributed from the terminal device A, adds the terminal device A to the attention number of the attention selection header, and relays the terminal device A and the terminal device. The voice data of A is distributed to the terminal device B. Basically, the server manages the attention selection header and distributes the audio and the relay video to each terminal device as they are.

  In the terminal device B, the attention selection header of the packet delivered from the server is processed to emphasize the voice level of the terminal device A and change the video of the terminal device A. When the terminal device B pays attention to the terminal device A, the server notifies the terminal device A that the terminal device B has paid attention. Upon receiving the notification, the terminal device A further changes the video of the terminal device B. At this time, the server adds a conversation group number indicating the same group as the terminal device B. When the terminal device C pays attention to the terminal devices A and B, the terminal devices A and B emphasize the volume of the terminal device C compared to other terminal devices. Audio from other terminal devices is distributed at a low volume (including silence). As described above, when information indicating that the own communication terminal device is designated is received from at least one of the plurality of communication terminal devices, the voice and / or image data of the designated communication terminal device is emphasized.

  FIG. 10 is a flowchart when the terminal device B shown in FIG. 9 processes the attention selection header of the packet distributed from the server.

  When the terminal device receives the packet having the attention selection header from the server (step 100), the terminal device confirms the terminal number of its attention selection header stored in the storage unit of the terminal device (step 110). Next, it is determined whether or not the attention number of the received attention selection header matches the terminal number of its own attention selection header (step 120). Is set to a non-attention level (step 200), and if they match, it is determined whether or not the attention number of its own attention selection header matches the terminal number of the received attention selection header (step 130). If the attention number of its own attention selection header does not match the attention number of the received attention selection header, the voice level of the terminal device that has been noticed is set to the target voice level (step 140).

  Next, it is determined whether or not to pay attention to the terminal device that has been noticed (step 150). When not paying attention to the terminal device that has been noticed, the terminal number of the received attention selection header is added to the attention target number of its own attention selection header (step 190). When attention is paid to the terminal device that has been noticed, the terminal number of the received attention selection header is added to the attention number of its own attention selection header (step 160). Next, the noticed terminal device is notified (step 170), and the voice level of the noticed terminal device is set to the noticed voice level.

  The ratio of the sound level is determined by the relationship of attention sound level> target sound level> non-notice level (including silence). The target voice level is a level that is increased so as not to hinder the listening of the voice level of the terminal device being noticed by a terminal device that has been noticed by a terminal device that has not been noticed.

  Next, the present invention will be described with reference to specific examples. FIG. 11 is a diagram for explaining N-to-N communication. Here, it is assumed that the users A to F make videophone calls at the same time. The audio data of the users A to F, video data, data necessary for IP communication, and the attention selection header of each user are sent to the server. The attention selection header is a setting value indicating which terminal device the user is paying attention to and which terminal device is receiving attention. A conversation group is formed when the number of interest and the number of interest match at each terminal device, and the audio data of the terminal device outside the group is, for example, silence. When attention selection is received from a terminal device that has not been noticed even when outside the group, the voice of the terminal device is at the target voice level. Audio of a terminal device that has not been selected for attention can also be displayed in subtitles.

  FIG. 12 is a diagram showing a communication terminal device when making a videophone call. The communication terminal device includes a reception unit 20, a display unit 21, a camera 23, an attention selection key 24, and a transmission unit 25. The attention selection key 24 can select and determine which user is focused. The display unit 21 includes an attention selection area 22 whose display changes when attention is selected. Here, “notice” is displayed when a user designates a certain terminal device, and attention is paid by other users. “Attention” is displayed when the user has been selected, and “Attention” is displayed as a mesh when the users pay attention to each other.

  FIG. 13 is a diagram illustrating the transition of the conversation group when the user C pays attention to the user A when the user A and the user B pay attention to form a conversation group. Here, the conversation viewable area indicates the outside of the conversation group of the user A. Outside the conversation group in which the user is present is called a conversation viewable area, and the voice of the terminal device in the conversation viewable area is synthesized as a low level or silence. By using the conversation group and the conversation viewable area in this way, even if the majority speaks at the same time and the voice overlaps, the target target or group voice to be noticed is not missed. Using the same line, it is possible to discuss without any discomfort even if multiple groups exist.

  FIG. 14 is a diagram illustrating the transition of the sound level and the screen transition of the terminal devices of the users A and B when the user A and the user B pay attention to each other. In the initial state, the sound level of all users is uniform. When the user A selects attention to the user B from the initial state, the terminal device of the user B determines whether to add the user A to the conversation group. At this time, since none of the terminal devices of the user B is paying attention, the display of the user A is changed, and the volume of the user A is emphasized as compared with other terminal devices. Since the terminal device of user A is paying attention to the terminal device of B, the voice level of user B is high after the attention selection.

  When the user B pays attention to the terminal device A, the video of both the A and B terminal devices changes, and conversation can be performed at a relatively higher voice level than other terminal devices. Others are distributed to the users A and B without sound because they are in the conversation viewing area. At this time, the terminal device in the conversation viewable area is displayed in subtitles, and even a terminal device not paying attention can reliably obtain information.

  FIG. 15 is a diagram illustrating the transition of the sound level and the screen transition of the terminal devices of the users A and B when the user C pays attention to the user A when the user A and the user B form a conversation group. . When the user C is in the initial state in which the users A and B are talking, when the user C pays attention to the user A, the video of the terminal device of the user A changes so that the user C can be noticed, It becomes audible with the voice of the attention level. There is no change because Mr. B who is the same conversation group is not attracting attention. At this time, the terminal device A can hear the voice of C at a slightly higher voice level.

  When the user A authenticates the user C, the user A can hear the voices at the same level as the users B and C. The video is changed so that the user A can understand that the user A and the conversation group have been formed with the user C, and the audio of the user C is distributed so as to be heard at the level of interest. At this time, it is possible to determine whether or not the user B is also interested in the user C. This is because user A formed a conversation group with user B.

  Although the present invention has been described with reference to the above-described embodiment, the present invention is not limited to this embodiment. Therefore, what is implemented as a change aspect without deviating from the meaning of the present invention is also included.

It is a system configuration figure of a video conference system. It is a block diagram of the communication terminal device of this invention. It is a block diagram of the server which is a communication control apparatus. It is a protocol block diagram of this invention. It is a conventional protocol block diagram. It is a block diagram of an attention selection header. It is a figure which shows the state of the attention selection header of each terminal. It is a timing chart in the case of controlling the volume by synthesizing voice at the server. It is a timing chart in the case of controlling the volume on the terminal device side. It is a flowchart when the terminal device B processes the attention selection header of the packet delivered from the server. It is a figure explaining N to N communication. It is a figure which shows a communication terminal device when carrying out the videophone. It is a figure which shows transition of a conversation group in case the user C pays attention to the user A when the user A and the user B are paying attention and comprise the conversation group. It is a figure showing the transition of the audio | voice level of the terminal device of the user A and B when the user A and the user B mutually pay attention, and a screen transition. It is a figure showing the transition of the audio | voice level of the terminal device of the user A and B, and a screen transition when the user C pays attention to the user A when the user A and the user B comprise the conversation group.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 Communication terminal device 2 Camera part 3,12 Transmission / reception part 4,13 Storage part 5,21 Display part 6 Operation part 7,14 Control part 11 Server 20 Reception part 22 Attention selection area 23 Camera 24 Attention selection key 25 Transmission part

Claims (4)

In a communication terminal device capable of calling with a plurality of communication terminal devices,
Transmission means for transmitting audio and image data;
Receiving means for receiving audio and image data of the plurality of communication terminal devices;
Output means for outputting audio and image data received by the receiving means;
Designating means for designating at least one communication terminal device from the plurality of communication terminal devices;
The output of at least the image data received by the receiving unit is different between when the other communication terminal device is specified by the specifying unit and when the own communication terminal device is specified by the other communication terminal device. Control means for controlling the output of the output means;
Equipped with a,
The control means is not the case where the designation means designates another communication terminal apparatus or the case where the communication terminal apparatus is designated from the other communication terminal apparatus; A communication terminal apparatus that controls the output of the output means so that at least the output of the image data received by the receiving means is different when the own communication terminal apparatus is designated . In a communication control device that controls communication between a plurality of communication terminal devices,
Receiving means for receiving audio and image data from the plurality of communication terminal devices;
Synthesizing means for synthesizing audio and image data received by the receiving means;
Transmitting means for transmitting the voice and image data synthesized by the synthesizing means to the plurality of communication terminal devices;
A case where information indicating that a second communication terminal device among a plurality of communication terminal devices is designated is received from a first communication terminal device among the plurality of communication terminal devices; and the second communication terminal device The voice and image data are synthesized so that at least the output of the image data in the first communication terminal apparatus is different from the case where the information indicating that the first communication terminal apparatus is designated is received from the first communication terminal apparatus. Control means for controlling to transmit to the communication terminal device;
Equipped with a,
The control means receives the information indicating that the second communication terminal apparatus is specified from the first communication terminal apparatus, and specifies the first communication terminal apparatus from the second communication terminal apparatus At least in the first communication terminal apparatus, and when the information indicating that the first communication terminal apparatus is designated is received from the second communication terminal apparatus. A communication control apparatus that controls to synthesize voice and image data so that output of image data is different and to transmit to the first communication terminal apparatus. The output of at least received image data differs between when a predetermined communication terminal device is specified from among the plurality of communication terminal devices and when the communication terminal device is specified from the predetermined communication terminal device. And when the predetermined communication terminal device is designated or when the communication terminal device is designated by the predetermined communication terminal device, the communication from the predetermined communication terminal device An output control method for a communication terminal device , wherein control is performed so that at least received image data is output differently when a terminal device is designated . In a communication control method of a communication control device for controlling communication between a plurality of communication terminal devices,
A case where information indicating that a second communication terminal device among a plurality of communication terminal devices is designated is received from a first communication terminal device among a plurality of communication terminal devices; The voice and image data are synthesized so that at least the output of the image data in the first communication terminal apparatus is different from the case where the information indicating that the first communication terminal apparatus is designated is received, Control is performed to transmit to the communication terminal device, and when the information indicating that the second communication terminal device is designated is received from the first communication terminal device, and the first communication terminal device receives the first information from the second communication terminal device. When the information indicating that the communication terminal apparatus is designated is not received, and when the information indicating that the first communication terminal apparatus is specified is received from the second communication terminal apparatus, the first 1 communication terminal device At least the output of the image data is synthesized voice and image data to be different, the communication control method of a communication control device and controls to transmit to the first communication terminal device takes.