JPH07245665A - Multimedia terminal - Google Patents

Abstract

(57) [Abstract] [Purpose] It is an object of the present invention to provide a multimedia terminal device capable of realizing smooth communication. [Structure] Therefore, since the voice data is delayed only during the moving image transmission, the responsiveness of the conversation is improved when exchanging still images or when making a voice-only call, and smooth communication is achieved. You can do it. Also, because the audio data is delayed only when the image of a person is being transmitted, when a person is not included in the image, when exchanging still images, and when making a call using only audio In this case, the responsiveness of the conversation is improved and smooth communication can be performed.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a multimedia terminal device capable of transmitting and receiving at least moving image data and audio data.

[0002]

2. Description of the Related Art In recent years, as terminals using ISDN,
At least a multimedia terminal device capable of transmitting and receiving moving image data and audio data, such as a video conference communication terminal, has been widely used, and has become an important item as an application using ISDN. The device function of such a multimedia terminal device is
For example, CCITT Recommendation H.264. 320 (videophone / videoconference terminal).

In this way, while displaying the image of a person on a multimedia terminal device using ISDN,
When the voice spoken by the person is output, the following inconvenience occurs.

That is, since the transmission capacity that can use ISDN is limited (for example, 128 Kbps in the basic interface), it is not realistic to exchange moving image information and audio information with raw data. Therefore, the moving image information and the audio information are encoded and compressed by an appropriate encoding method, and the encoded and compressed moving image data and audio data are exchanged in the transmission path.

Here, since the moving image information has a remarkably larger amount of information per unit time than the audio information, the processing time required for encoding and decoding the moving image information is generally the same as the encoding and decoding of the audio information. It is longer than the processing time required for conversion.

In this way, since the processing time required for encoding and decoding the moving image information and the audio information is different, the video information and the audio information obtained at the same time on the transmitting side are output on the receiving side at different times. To be done.

As a result, the movement of the person's lips in the image viewed on the receiving side and the sound output on the receiving side are not synchronized,
Unnatural video / audio can be obtained. Here, such an inconvenience is called a lip sync problem.

In order to solve this lip sync problem, the following method has been conventionally applied.

That is, by delaying the audio information by a delay time corresponding to the processing time required for encoding and decoding the moving image information and the audio information, the movement of the lips of the person in the image seen on the receiving side and the receiving side. The sound output in the same was matched so that a natural image / sound could be obtained.

[0010]

However, such a conventional method has the following disadvantages.

For example, in the video conference communication terminal, the object to be photographed may be not only a person but also a still object.
That is, the video conference communication terminal exchanges not only moving images but also still images with the other terminal. In this way, even when exchanging still images, a voice call is made. In some cases, only voice communication may be performed with the image erased.

Therefore, when voice information is delayed as described above even when a still image is transmitted / received or a voice-only communication is performed, a conversation with the user of the partner terminal may occur. The responsiveness is deteriorated, and smooth video conference communication is hindered.

The present invention has been made in view of such circumstances, and an object thereof is to provide a multimedia terminal device capable of realizing smooth communication.

[0014]

The present invention provides at least the following:
In a multimedia terminal device capable of transmitting and receiving moving image data and audio data, a delay correction means for delaying the audio data by a predetermined delay time corresponding to the difference between the modulating and demodulating time of the moving image data and the modulating and demodulating time of the audio data, and transmitting and receiving the moving image data. The control means delays the audio data by the delay correction means when the delay time is set, and prevents the delay data from being delayed by the delay correction means when the moving image data is not transmitted or received.

Further, at least in a multimedia terminal device capable of transmitting and receiving moving image data and audio data, a predetermined delay time audio data corresponding to a deviation between the modulation time of transmission moving image data and the modulation time of transmission audio data is delayed. A first delay correction means, and a second delay correction means for delaying the audio data by a predetermined delay time corresponding to the deviation between the modulation time of the received video data and the modulation time of the received audio data,
When transmitting moving image data, the first delay correcting unit transmits the transmitted audio data in a delayed state, while when receiving moving image data, the second delay correcting unit delays the received audio data. It is equipped with means.

Further, at least in a multimedia terminal device capable of transmitting and receiving moving image data and audio data, a delay correction for delaying the audio data by a predetermined delay time corresponding to the difference between the modulating and demodulating time of the moving image data and the modulating and demodulating time of the audio data. Means and the moving image data based on the person's image are transmitted and received, the delay correction means delays the audio data. When the moving image data based on the person's image is not transmitted and received, It is provided with a control means that does not delay.

Further, at least in a multimedia terminal device capable of transmitting and receiving moving image data and audio data, a predetermined delay time audio data is delayed corresponding to a deviation between the modulation time of transmission moving image data and the modulation time of transmission audio data. A first delay correction means, and a second delay correction means for delaying the audio data by a predetermined delay time corresponding to the deviation between the modulation time of the received video data and the modulation time of the received audio data,
When transmitting the moving image data based on the image of the person, the transmission sound data is transmitted in a delayed state by the first delay correction means, while when the moving image data based on the image of the person is received, the second delay is transmitted. The control means delays the received voice data by the correction means.

[0018]

Therefore, since the voice data is delayed only during the transmission of the moving image, the responsiveness of the conversation is improved when exchanging a still image or when making a call using only the voice, and smooth communication is possible. You can do it.

Further, since the audio data is delayed only when the image of the person is being transmitted, the case where the person is not included in the image, the case of exchanging a still image, and the case of using only the audio The responsiveness of conversation when making a call is improved, and smooth communication can be performed.

[0020]

Embodiments of the present invention will now be described in detail with reference to the accompanying drawings.

FIG. 1 shows a video conference terminal device according to an embodiment of the present invention.

In FIG. 1, a microphone 1 is for inputting voice information, a speaker 2 is for outputting voice information, and a monitor device 3 is for video information (moving image and still image). , The main camera device 4 is mainly for shooting a moving image, and the document camera device 5 is for shooting a stationary object such as a document, and a video. The tape reproducing device 6 is for reproducing video / audio of a video tape recorded by another means.

A voice signal SA1 output from the microphone 1,
The audio signal SA2 output from the video tape reproducing device 6 is applied to one input terminal of the switching mixer 7. The switching mixer 7 selects either one of the audio signal SA1 and the audio signal SA2 or mixes the audio signal SA1 and the audio signal SA2, and the output thereof is the analog transmission audio signal SAt. CO
It is added to the analog audio signal input terminal of the DEC 8.

The audio CODEC 8 converts the analog transmission audio signal SAt applied to the analog audio signal input terminal into corresponding digital transmission audio data DAt, and
The received digital received voice data DAr is converted into a corresponding analog received voice signal SAr and output to the speaker 2. The audio CODEC 8 has a plurality of types of conversion functions.

The digital transmission audio data DAt output from the audio CODEC 8 passes through the delay circuit 9 and is converted into the digital transmission audio data DAt 'by the multiplexing / demultiplexing device 10.
The digital transmission voice data has been added to the input end. In addition, the digital reception audio data output from the digital reception audio data output terminal of the multiplexer / demultiplexer 10 passes through the delay circuit 11 and the digital reception audio data DAr.
Has been added to the voice CODEC8.

The moving image signal SVm output from the main camera device 4, the still image signal SVs output from the document camera device 5, and the reproduced video signal SVp output from the video tape reproducing device 6 are respectively output from the switching device 12. It is added to one input terminal.

The switch 12 selects one of the moving image signal SVm, the still image signal SVs, and the reproduced video signal SVp, and the output thereof is the transmission video signal SV.
It is added to the analog transmission video signal input end of the moving image CODEC 13 as t. The moving image signal and the still image signal are collectively referred to as a video signal (hereinafter the same).

The moving picture CODEC 13 converts the NTSC type transmission video signal applied to the analog transmission video signal input terminal into corresponding digital data video data, and converts the video data into predetermined CIF type video data.
Furthermore, CCITT Recommendation H.264. In addition to encoding and compressing by the encoding method of H.261, the transmission video information DVt is formed, the reception video information in the coded and compressed state output from the digital reception video data output terminal of the multiplexing / demultiplexing device 10 is used as the original. The video signal is converted into CIF type video data, the video data is converted into a corresponding analog signal video signal, and the video signal is converted into an NTSC type video signal.

The video signal converted and formed by the moving picture CODEC 13 in this manner is the received video signal S.
It is output to the monitor device 3 as Vr, so that, for example, the state of the partner terminal side is displayed and output to the monitor device 3. The transmission video information DVt formed by the moving image CODEC 13 is added to the digital transmission video data input terminal of the multiplexing / separating device 10.

The multiplexing / demultiplexing control unit 10 operates according to CCITT Recommendation H.264. 221 is provided with a data multiplexing / separation function compliant with 221 and a data frame synchronization function, and the separation side thereof has a digital transmission data input end, a digital reception data output end, a digital transmission video data input end, A digital reception video data output terminal and a FAS / BAS input / output terminal are provided, and the multiplexing side is connected to the ISDN interface circuit 14.

The ISDN interface circuit 14 connects the multimedia terminal device to the ISDN, and also combines the signal processing function of layer 1, the D channel (signal channel) signal and the two B channel (information channel) signals. The signal connection end on the D channel side is connected to the D channel transmission control unit 15, and the two B channel connection ends are connected to the multiplexing / demultiplexing device 10. The D channel transmission controller 15
This is for realizing the call control procedure function performed in the signal channel and the multiple call control function.

The control unit 16 is for performing operation control of each unit of this multimedia terminal device, processing of transmission control procedure of the upper layer in the information channel (B channel), and the like, and the operation display unit 17 is For operating a multimedia terminal device, for example, a dial key for specifying a destination, a video source selection key (including video off), and an audio source selection key (audio off (mute) Various operation keys such as (including) and a display (for example, a liquid crystal display).

The delay circuit 9 and the delay circuit 11 are
When the delay function is on / off controlled by the control unit 16 and the delay function is on, the input signal is delayed by the set delay time TTs, and when the delay function is off, the input signal is delayed. Is output without delay.

Now, in this multimedia terminal device,
In the B channel, the TTC standard JT-H. 221
Data is exchanged in a multi-frame format defined by (national standard corresponding to CCITT recommendation H.221).

As shown in FIG. 2, one multi-frame MFL includes eight sub-multi-frames SMF1 to SMF.
8 sub-multiframes SMF1 to 8
Each SMF 8 is composed of two frames. That is, one multi-frame MFL is composed of 16 frames FLM0 to FLM15.

Each frame FLM0 to FLM15
As shown in FIG. 3, it is composed of data of 80 octets, and the respective bit positions in which these octets are arranged in the bit order form subchannels SCH1 to SCH8.

The eighth bit of the first octet to the eighth octet is the frame synchronization signal (Frame Ali).
The 8th bit of the 9th octet to the 16th octet constitutes a bit rate allocation signal (Bit rate Allocatio).
n Signal) BAS. The part of the eighth bit from the 17th octet to the 80th octet of the sub-channel SCH8 is set with a part of the data of the encryption channel for exchanging key information for encrypting the data. Sometimes (optional).

In this way, the frame synchronization signal FAS
8 bits are arranged in one frame FLM0 to FLM15, and the bit allocation is configured in units of multiframe MFL as shown in FIG.

That is, even frames FLM0, FLM
2, ..., The second to eighth octets of the FLM 14, and the odd frames FLM1, FLM3 ,.
A horizontal synchronization signal composed of an 8-bit data pattern "001101011" is arranged in the FLM 15, and a vertical synchronization composed of a 6-bit data pattern "001011" is arranged in the first octet of the odd-numbered frames FLM1, FLM3, ..., FLM11. A sync signal is placed.

By detecting the horizontal synchronizing signal and the vertical synchronizing signal, it is possible to detect the synchronization of one multi-frame MFL.

The 0th frame, the 2nd frame, the 4th frame
Bits N1, N2, N3, N4, N5 of the first octet of the frame, the sixth frame and the eighth frame are used to indicate the multi-frame number. Of these, the bit N5 is used to indicate whether or not the multiframe number is used. In this way, since the data used for the multi-frame number is 4 bits, the multi-frame number changes in descending order with a value of 0 to 15, and the same multi-frame number appears every 16 multi-frames.

The tenth frame, the twelfth frame,
And bit L of the 1st octet of the 13th frame
1, L2, L3 are used to display the connection number indicating the order in which the information channels carrying the frame among the currently used information channels are connected. The bit R of the first octet of the fifteenth frame is reserved (reserved) for future recommendation and is normally set to 0.

The bit TEA of the first octet of the 14th frame is used to indicate that data transmission is impossible due to an internal failure of the data terminal device.

Also, odd-numbered frames FLM1, FLM3
.., bit A of the third octet of the FLM 15 is used to indicate whether frame synchronization or multi-frame synchronization is established or out of synchronization.

Also, odd-numbered frames FLM1, FLM3
..., bit C1 of the fifth octet, sixth octet, seventh octet, and eighth octet of FLM15
C2, C3 and C4 are for displaying a CRC (Cyclic Redundancy Check) code that is referred to for data error detection (that is, channel quality detection) of two consecutive frames (that is, sub-multiframe). , And the bit E of the fourth octet of the odd frames FLM1, FLM3, ..., FM15 is used to indicate that a transmission error has been detected on the receiving side.

The bit rate allocation signal BAS is, as shown in FIG. 5, even-numbered frames FLM0, FLM2.
.., FLM14, 8-bit data representing the capability BAS or BAS command is arranged, and in the subsequent odd numbered frames FLM1, FLM3 ,.
A double error correction code for error correcting the value of the capability BAS or BAS command transmitted in the immediately preceding frame is arranged. Here, the capability BAS is used, for example, to notify the partner terminal of the device function of the own terminal, and the BAS command is used to instruct the partner terminal to switch the communication function. The signal format of a predetermined format is specified for each. Further, for example, an extended BAS format applied for notifying a non-standard function is defined.

Data transmission of the multi-frame MFL is performed in the order of frame numbers, and each frame FLM0 to FLM15 is transmitted in the octet order from the first octet to the 80th octet, as shown in FIG. The first bit of the octet is sent first.

That is, each frame FLM0 to FLM0
In the FLM 15, the first bit of the first octet is transmitted first and the eighth bit of the 80th octet is transmitted last.

Further, in this multimedia terminal device,
Usually, data transmission is performed using two information channels, and an example of the transmission procedure at that time is shown in FIG. Note that this transmission procedure is based on the TTC standard JT-H. 242 (C
CITT Recommendation H.264 242) and standard JT-H.242. 320 (national standard corresponding to CCITT Recommendation H.320).

First, the calling terminal makes a call to the destination terminal and performs a call setup procedure on the signaling channel to secure one information channel (hereinafter referred to as the first channel) (Phase A). Frame synchronization is performed while exchanging frame data in which PCM audio data (A-law or μ-law, 64 Kbps) is set on one channel (frame mode). When frame synchronization is established, the capability BAS data and command BAS data are mutually exchanged. Exchanges (phase B1-1), decides a transmission mode to be used at that time, and initiates an additional call setup request for securing the second information channel (phase B1-2).

Then, from the contents of the exchange of the transmission modes at that time, the common and highest function mode is selected (phase B1-3), and the calling terminal functions as the called terminal in the selected operation mode. A BAS command for designating a terminal function is operated to operate, and parameters common to the device functions of the calling terminal and the called terminal are set (phase B2). Thereby, in the first channel, data transmission corresponding to the transmission mode selected at that time, for example,
Data transmission of audio data (16 Kbps) and data transmission of moving image data (46.4 Kbps) are performed (phase C).

When the first channel starts data transmission in the frame mode, a call setup procedure is performed on the signal channel for the second channel in order to secure the second information channel (hereinafter referred to as the second channel) ( Phase C
A).

When the second channel is established, frame data including only the frame synchronization signal FAS and the bit allocation signal BAS is exchanged using the second channel to establish frame synchronization and multiframe synchronization (Phase C).
B1-11), and then synchronization between the first channel and the second channel is established (phase CB1-12).

When the synchronization of the two information channels is complete,
The BAS command is sent from the calling terminal side to set the transmission mode (phase CB1-2), the transmission mode is switched to the set contents (phase CB1-3), and the common parameter is set (phase CB3).

When the initialization of the second channel is completed in this way, after that, the frame data exchanged on the first channel and the frame data exchanged on the second channel are synchronized with each other. Data transmission of audio data, general-purpose data, and moving image data is executed using the information channel. 56 Kbps for audio data, general-purpose data, and video data,
FIG. 8 shows a variation in allocation of various data to the first channel and the second channel when transmission rates of 6.4 Kbps and 62.4 Kbps are allocated.

To end such data transmission, first, the second channel is disconnected. At this time, the first
For voice data transmission on the channel only, the procedure for setting the common mode is performed (Phase CD
1), the mode of the second channel is switched to the mode 0F of the frame mode (phase CD2). At this time, the first channel and the second channel are asynchronous, and the second channel is in a state where the call is held in the transmission state of only the frame synchronization signal FAS and the bit allocation signal BAS, and the call disconnection of the signal channel is performed. The call for the second channel can be released by the release procedure.

In the first channel, phase CD1
During the phase CD2, the audio data and the moving image data are transmitted at the transmission rate of 62.4 Kbps in total in the frame mode.

When the operator of one of the terminals ends the call from this state, first, the transmission of the moving image data is ended, and the transmission capacity of all the first channels including the transmission capacity of this moving image data is transmitted by voice. Mode 0 for use in
Switch to F (phase D2). Here, the call of the first channel can be released by the call disconnection release procedure of the signaling channel.

As a result, the call disconnection release procedure is executed on the signaling channel for the first channel and the second channel (phase E), and the multimedia transmission between the two terminals is completed.

As described above, in the multimedia terminal device, when data transmission is started, first, one information channel (first channel) is first secured to establish the frame mode, and then the voice is transmitted on the first channel. If the second information channel (second channel) can be set at the same time while allocating the data and video data transmission rates and performing data transmission in the transitional mode, the call setup procedure for the signal channel is used. Reserve the second channel.

Then, the first channel transmitted in the transient mode and the newly secured second channel are channel-synchronized, and when the channel synchronization is established, the transmission rates of the audio data and the moving image data are assigned. In order to utilize the increased transmission capacity of the information channel by changing the coding rule (coding method) of audio data and moving image data, and exchanging higher quality audio data and moving image data. I am trying.

Further, when the data transmission is completed, it is necessary to once change from the mode in which the transmission path in which both the first channel and the second channel are synchronized is used to the mode in which only the first channel is used. is there. Therefore, first, each coding rule of the audio data and the moving image data is set to a method that optimizes the transmission capacity of the first channel of 62.4 Kbps, and the transmission mode of only the first channel is changed.
At this time, the second channel stops the synchronization state with the first channel, shifts to the mode 0F in which the user data is in the idle transmission state, and disconnects / releases the call by the call disconnection release procedure of the signal channel.

On the other hand, in the first channel, after changing from two types of media transmission of voice data and moving image data to mode 0F which is a transmission mode of only voice data, the call is disconnected by the call disconnection release procedure of the signal channel. /release.

At the end of the data transmission, the first channel can be disconnected / released after the mode is directly changed to the mode 0F without switching the mode to the two types of media transmission of the audio data and the moving image data. . In addition, the billing information obtained in the call disconnection / release procedure is managed for each channel.

With the above-mentioned configuration, the control section 16 controls when the self terminal sends the transmission video signal SVt to the partner terminal.
The operation of the delay circuit 9 is set to the ON state. As a result, as shown in FIGS. 9C and 9E, the audio CODE
The analog transmission voice signal SAt input to C8 is the digital transmission voice data D corresponding to the voice CODEC 8.
After being converted into At, it is added to the multiplexer / demultiplexer 10 as digital transmission voice data DAt ′ in a state of being delayed by the delay time TTs set in the delay circuit 9.

On the other hand, the transmission video signal S input to the moving picture CODEC 13 at the same time as the analog transmission audio signal SAt.
The Vt is converted into corresponding transmission video information DVt by the moving image CODEC 13, and this transmission image information DVt is
In terms of time, it is in a state of being delayed from the transmission video signal SVt by the processing time TTv at the time of encoding the moving image CODEC 13 (see FIGS. 9D and 9F).

Therefore, the digital audio data DAt 'applied to the digital audio data input terminal of the multiplexer / demultiplexer 10 is a signal in a state in which the delay time TTs has elapsed since the original transmission audio signal SAt was output. The transmission video information DVt applied to the digital transmission video data input terminal of the multiplexer / demultiplexer 10 is a signal in a state in which the delay time TTv has elapsed since the original transmission video signal SVt was output.

Thus, at the time of being applied to the input terminal of the multiplexer / demultiplexer 10, the time difference between the audio data and the moving image data is an amount corresponding to the difference between the delay time TTs and the delay time TTv.

That is, in this case, the time difference between the audio data and the moving image data transmitted from the multimedia terminal device to the partner terminal is the time corresponding to the difference between the delay time TTs and the delay time TTv. Is shortened to.

Further, the control unit 16 sets the operation of the delay circuit 11 to the ON state when the self terminal receives the moving image data. As a result, the digital reception data output from the digital reception data output terminal of the multiplexer / demultiplexer 10 is delayed by the delay time TTs set in the delay circuit 11, and is output as the digital reception voice data DAr. Added to CODEC8, voice CODEC8
Is converted into a corresponding analog reception voice signal SAr and output from the speaker 2.

On the other hand, the received video information output from the digital received data output terminal of the multiplexer / demultiplexer 10 at the same time as the digital received data is converted into a corresponding received video signal SVr by the moving picture CODEC 13, and this received video signal is received. The SVr is output to the monitor device 3 and the corresponding received video is displayed and output. At this time, the received video signal SVr
In terms of time, is in a state of being delayed from the received video information by the processing time TTt at the time of decoding the moving image CODEC 13.

Therefore, the sound output from the speaker 2 is in a state in which the delay time TTs has elapsed since the original digital received sound data was output from the multiplexing / demultiplexing device 10, and the monitor device 3 outputs the display. The video to be displayed is in a state in which the delay time TTt has elapsed since the reception / video information was output from the multiplexer / demultiplexer 10.

As a result, the time lag between the voice heard by the user and the video to be viewed becomes an amount corresponding to the difference between the delay times TTs and TTt.

At the time of application to the input terminal of the multiplexer / demultiplexer 10, the time difference between the audio data and the moving image data is an amount corresponding to the difference between the delay time TTs and the delay time TTv.

In other words, in this case, the time difference between the audio data and the moving image data that is output to the user after being received by this multimedia terminal device corresponds to the difference between the delay time TTs and the delay time TTt. It is shortened to the time to do.

Therefore, the audio and video input from the multimedia terminal device on the transmission side are added with the above-mentioned time difference on the transmission side and the time difference on the reception side from the multimedia terminal device on the reception side. The signals are output from the speaker 2 and the monitor device 3, respectively, with a time lag of about that.

As a result, for example, the movement of the person's lips in the image viewed on the receiving side and the audio output on the receiving side are almost synchronized, and a natural image / audio is obtained. Further, even when the video / audio reproduced by the video tape reproducing apparatus 6 is transmitted, since the video and the audio are output in a state of being substantially synchronized with each other at the receiving side, a natural video / audio can be obtained.

Further, the control unit 16 turns off the delay operation of the delay circuit 9 when the moving image data is not being transmitted.
As a result, when moving image data is not being transmitted, the digital transmission audio data DAt output from the audio CODEC 8 is added to the multiplexer / demultiplexing device 10 as digital transmission audio data DAt ′ without being delayed by the delay circuit 9.

At the same time, the control unit 16 turns off the delay operation of the delay circuit 11 when the moving image data is not received. As a result, when the moving image data is not received, the digital received audio data output from the multiplexer / demultiplexer 10 is added to the audio CODEC 8 without being delayed by the delay circuit 11.

Therefore, in this case, the voice inputted by the multimedia terminal device on the transmitting side is delayed by the delay circuit 9,
Since it is output from the multimedia terminal device on the receiving side without being delayed by 11, the responsiveness of the voice call is improved.

In this way, according to the present embodiment, it is possible to solve the lip sync problem when moving image data is exchanged, and it is possible to make a call with good responsiveness when moving image data is not exchanged. , The usability of multimedia terminal device is greatly improved.

Here, the delay time TT of the delay circuits 9 and 11
The value of s is preferably set to an integral multiple of 20 (milliseconds) of the sub-multiframe period described above (FIG. 9).
(See (a) and (b)).

That is, the time required for encoding the moving image data and the time required for decoding the moving image data are
Since the amount of delay varies depending on the content of the moving image data at that time, and the range of variation is relatively large, it is generally difficult to predict beforehand the magnitude of the delay time caused by these processes.

Therefore, the encoding processing time and the decoding processing time are measured for a specific moving image (for example, a scene where a person is talking), and the average value of the measured values and the fluctuation range of the measured values are measured. According to the delay circuits 9 and 1
It is preferable to set the delay time TTs of 1.

Therefore, the delay time TTs can be set with a certain width. Further, the sub-multiframe cycle is 20 (milliseconds), and when the delay time TTs is set to an integral multiple of the sub-multiframe cycle close to the average value of the actual measurement values of the delay times of the actual video data, the delay of the video data is delayed. Since it is expected that the difference from the time will be about ± 20 (milliseconds), it is appropriate to set the delay time TTs to an integral multiple of the sub-multiframe period.

Further, since the data of two information channels are exchanged in the sub-multiframe period, it may be preferable to give the transmission data periodicity according to the sub-multiframe period.

For example, CCITT Recommendation G. 728 (16
Kbps; LD-CELP (Low-Delay Co
de Excited Linear Predict
Ion; low delay code excitation type linear prediction) method),
Audio COD with framed data format
When exchanging audio data of the EC system (described later), it is necessary to establish frame synchronization and subframe synchronization.

In this case, if the delay time TTs of the delay circuits 9 and 22 is set to an integral multiple of the sub-multiframe period, the sub-multiframe period is synchronized with the frame period, so the delay time TTs As a result, the synchronization deviation of frame synchronization does not occur due to the addition of
The circuit configuration of audio data processing including the delay correction element (in this case, the delay circuits 9 and 11) can be simplified.

FIG. 10, FIG. 11 and FIG. 12 show an example of processing when the user operates the operation display unit 17 to instruct the source switching. Here, the source is
Refers to audio or video sources. The audio sources are the microphone 1 and the video tape reproducing device 6, and the video sources are the main camera device 4, the document camera device 5,
And the video tape reproducing device 6.

First, when the source switching operation is instructed (the result of judgment 101 is YES), it is checked whether or not the video source switching is instructed (judgment 102). Judgment 1
When the result of 02 is YES, it is checked whether or not the main camera device 4 is selected (decision 103).

When the result of judgment 103 is YES, it is checked whether or not the flag FV1 for storing that any video is being transmitted is set (decision 104).

When the image is not being transmitted at that time, and the flag FV1 is not set (the result of the judgment 104 is NO), a command to indicate that the terminal itself is transmitting video data, video on BAS. A command is sent to the partner terminal (process 105), the multiplexer / demultiplexing device 10 is instructed to turn on the video on the transmission side (process 106), and the flag FV1 is set (process 107). The video signal SVm output from the main camera device 4 is selected by the switch 12 (process 108), the video source is switched to the main camera device 4, and the process returns to the determination 101.

When the result of the determination 104 is YES, moving image data is already being transmitted and the image source has been switched from the other image source to the main camera device 4. The source is switched to the main camera device 4.

When the result of the judgment 103 is NO, it is checked whether or not the document camera device 5 is selected at that time (judgment 109).

When the result of judgment 109 is YES, it is checked whether or not a flag FV1 for storing that any image is being transmitted is set (decision 110).

When the image is not being transmitted at that time and the flag FV1 is not set (the result of the judgment 110 is NO), a command to indicate that the self terminal transmits video data is a video on BAS. The command is sent to the partner terminal (process 111), the multiplexing / demultiplexing device 10 is instructed to turn on the video on the transmission side indicating that the video data is to be transmitted (process 112), and the flag FV1 is set (process 113). The still image signal SVs output from the document camera device 5 is selected by the switch 12 (process 114), the video source is switched to the document camera device 5, and the process returns to the determination 101.

When the result of judgment 110 is YES, the moving picture data is already being transmitted and the picture source has been switched from the other picture source to the document camera device 5, so that the processing shifts to the processing 114 The source is switched to the document camera device 5.

When the result of judgment 109 is NO,
At that time, it is checked whether the video tape reproducing device 6 has been selected (decision 115).

When the result of judgment 115 is YES, it is checked whether or not a flag FV1 for storing that any image is being transmitted is set (decision 116).

When the image is not being transmitted at that time, and the flag FV1 is not set (the result of the judgment 116 is NO), the command is the video on BAS which is a command indicating that the self terminal transmits the video data. The command is sent to the partner terminal (process 117), the multiplexing / separation apparatus 10 is instructed to turn on the video on the transmitting side, which represents the transmission of moving image data (process 118), and the flag FV1 is set (process 119). The reproduction video signal SVp output from the video tape reproducing device 6 is selected by the switch 12 (process 120), and the video source is switched to the video tape reproducing device 6.

When the result of the determination 116 is YES, the moving picture data is already being transmitted, and the picture source has been switched from the other picture source to the video tape reproducing device 6, so that the process proceeds to step 120. The video source is switched to the video tape reproducing device 6.

When the process 120 is completed, it is next checked whether or not the flag FA1 for storing that any one of the voices is being transmitted is set (decision 121).

When the voice is not being transmitted at that time and the flag FA1 is not set (the result of the determination 121 is NO), the voice ON BAS which is a command indicating that the terminal itself transmits the voice data. A command is sent to the partner terminal (process 122), the multiplexing / demultiplexing device 10 is instructed to turn on the voice on the transmission side indicating that voice data is to be transmitted (process 123), and the flag FA1 is set (process 124). The reproduced audio signal SA2 output from the video tape reproducing device 6 is selected by the switching mixer 7 (process 125), and the audio source is switched to the video tape reproducing device 6.

When the result of the judgment 121 is YES, the audio data is already being transmitted and the audio source has been switched from the other audio source to the video tape reproducing device 6. Therefore, the processing shifts to the processing 125. The video source is switched to the video tape reproducing device 6. Then, the processing 125
When is finished, the process returns to the determination 101.

When the result of judgment 115 is NO,
At that time, it is checked whether or not a video off command has been issued (decision 126). When the result of the determination 126 is YES, the video off BAS command, which is a command indicating to stop the transmission of the video data of the own terminal, is sent to the partner terminal (process 127), and the multiplexing / demultiplexing device 10
The video off of the transmission side, which indicates to stop the transmission of the moving image data, is designated (process 128), the flag FV1 is cleared (process 129), and the process returns to the determination 101.

When the result of the judgment 126 is NO, it means that the switching operation of the video source has been canceled, and at that point, the process immediately returns to the judgment 101.

When the switching of the audio source is instructed and the result of judgment 102 is NO, it is checked whether or not the microphone 1 is selected (judgment 130).

When the result of judgment 130 is YES, it is checked whether or not a flag FA1 for storing that any voice is being transmitted is set (decision 131).

When the voice is not being transmitted at that time and the flag FA1 is not set (the result of the judgment 131 is NO), the voice ON BAS which is a command indicating that the own terminal transmits the voice data. The command is sent to the partner terminal (process 132), the multiplexing / demultiplexing device 10 is instructed to turn on the transmission side voice indicating that the voice data is to be transmitted (process 133), and the flag FA1 is set (process 134). The audio signal SA1 output from the microphone 1 is selected by the switching mixer 7 (process 135),
The audio source is switched to the microphone 1, and the process returns to the judgment 101.

If the result of judgment 131 is YES, it means that the audio data is already being transmitted and the audio source has been switched from another audio source to the microphone 1.
The process moves to processing 135, and the audio source is switched to the microphone 1.

When the result of the judgment 130 is NO, it is checked whether or not the video tape reproducing device 6 is selected (decision 136).

When the result of judgment 136 is YES, it is checked whether or not the flag FA1 for storing that any voice is being transmitted is set (decision 137).

When the voice is not being transmitted at that time and the flag FA1 is not set (the result of the judgment 137 is NO), the voice ON BAS which is a command indicating that the own terminal transmits the voice data. The command is sent to the partner terminal (process 138), the multiplexing / demultiplexing device 10 is instructed to turn on the voice on the transmitting side indicating that the voice data is to be transmitted (process 139), and the flag FA1 is set (process 140). The audio signal SA2 output from the video tape reproducing device 6 is selected by the switching mixer 7 (process 141), and the audio source is selected as the audio source.
, And the process returns to judgment 101.

When the result of the judgment 137 is YES, the audio data is already being transmitted and the audio source has been switched from the other audio source to the video tape reproducing device 6, so the processing shifts to the processing 141. The audio source is switched to the video tape reproducing device 6.

When the result of judgment 136 is NO,
At that time, it is checked whether or not a voice off command has been issued (decision 142). When the result of the determination 142 is YES, the voice off (mute) BAS command, which is a command indicating to stop the transmission of the voice data of the own terminal, is sent to the partner terminal (process 143), and the multiplexer / demultiplexer 10 is connected. Is designated to be OFF on the transmitting side (processing 144), the flag FA1 is cleared (processing 145), and the process returns to the determination 101.

When the result of judgment 142 is NO,
At that time, it is checked whether a voice mix is instructed (decision 146). When the result of determination 146 is YES, the switching mixer 7 mixes the audio signal SA1 output from the microphone 1 and the audio signal SA2 output from the video tape reproducing device 6 (process 147), and the determination 10 is performed.
Return to 1. Further, when the result of the judgment 146 is NO, it means that the switching operation of the audio source has been canceled, and therefore the process immediately returns to the judgment 101.

In this way, when the switching operation of the video source and the audio source is performed, the transmission video information corresponding to the video of the video source after the switching is output to the multiplexing / separating apparatus 10 and the video after the switching is performed. The digital transmission audio data corresponding to the audio of the audio source (including the audio after mixing) is output to the multiplexing / separation device 10. Further, when the video off is operated, the video transmission is stopped, and when the audio off is operated, the audio transmission is stopped.

At the same time, the flag FV1 for storing the video transmission state and the flag FA1 for storing the voice transmission state are appropriately set according to the video transmission state and the voice transmission state at that time. Is turned on and off.

13 and 14 show an example of processing when the command BAS is received from the partner terminal.

When a batch of BAS signals is received (the result of judgment 201 is YES), B received at that time is received.
It is checked whether the AS signal is a command for instructing video-on (decision 202).

When the result of judgment 202 is YES, it is checked at that time whether or not the flag FV2 for storing that any image is being received is set (decision 203).

When the image is not being received at that time and the flag FV2 is not set (the result of the judgment 203 is NO), the flag FV2 is set (process 204) and the multiplexer / demultiplexer 10 is set. , A command to turn on the video on the receiving side, which represents reception of moving image data (step 205).

Next, it is checked whether or not the flag FA2 for storing that any voice is being received is set (decision 206). Judgment 206
If the result is YES, a flag FC2 for storing that the delay correction on the receiving side is being performed
Is checked (decision 207).

When the result of the judgment 207 is NO,
In this case, since the delay correction on the receiving side is to be executed, the flag FC2 is set (process 208), the delay operation of the delay circuit 11 is turned on (process 209), and the determination 201 is made.
Return to.

If the image is already received at that time and the result of the determination 203 is YES, the process immediately proceeds to the immediate determination 206. Further, when the result of the judgment 206 is NO, and when the result of the judgment 207 is YES, the immediate judgment 20 is made.
Move to 1.

When the result of judgment 202 is NO, it is checked whether the BAS signal received at that time is a command for instructing voice-on (decision 210).

When the result of the judgment 210 is YES, it is checked at that time whether or not the flag FA2 for storing that any voice is being received is set (decision 211).

When the voice is not being received at that time and the flag FA2 is not set (the result of the judgment 211 is NO), the flag FA2 is set (process 212) and the multiplexer / demultiplexer 10 is set. , Commanding reception side voice ON indicating reception of voice data (Process 2
13).

Then, it is checked whether or not the flag FV2 for storing that any one of the images is being received is set (decision 214). Judgment 214
If the result is YES, a flag FC2 for storing that the delay correction on the receiving side is being performed
Is checked (decision 215).

When the result of judgment 215 is NO,
In this case, since the delay correction on the receiving side is executed, the flag FC2 is set (process 216), the delay operation of the delay circuit 11 is turned on (process 217), and the determination 201
Return to.

Further, at that time, the voice is already being received, and when the result of the judgment 211 is YES, at that time, the process shifts to the immediate judgment 214. Further, when the result of the determination 214 is NO, and when the result of the determination 215 is YES, the immediate determination 20 is made.
Move to 1.

When the result of judgment 210 is NO, it is checked whether the BAS signal received at that time is a command for instructing video off (decision 21).
8).

When the result of judgment 218 is YES, it is checked at that time whether or not the flag FV2 for storing that any image is being received is set (decision 219).

At that time, the image is being received and the flag FV2 is set (determination 2
If the result of 19 is YES), the flag FV2 is cleared (process 220), and the multiplexing / demultiplexing device 10 is instructed to turn off the video on the receiving side, which indicates to stop receiving the moving image data (process 221).

Then, it is checked whether or not the flag FC2 for storing that the delay correction on the receiving side is being performed is set (decision 222).

When the result of determination 222 is NO,
In this case, since the delay correction on the receiving side is not performed, the flag FC2 is cleared (process 223), the delay operation of the delay circuit 11 is turned off (process 224), and the determination 201
Return to.

When the image is not already received at that time and the result of judgment 219 is NO,
At that point, the process moves to immediate determination 222. Also, the judgment 22
When the result of 2 becomes NO, the immediate judgment 2
Move to 01.

When the result of judgment 218 is NO, it is checked whether or not the BAS signal received at that time is a command for instructing voice-on (decision 225).

When the result of judgment 225 is YES, it is checked at that time whether or not the flag FA2 for storing that any voice is being received is set (decision 226).

At that time, the voice is being received, and when the flag FA2 is set (determination 2
The result of step 26 is YES), the flag FA2 is cleared (step 227), and the multiplexing / separation apparatus 10 is instructed to turn off the receiving side voice, which indicates to stop the reception of voice data (step 228).

Then, it is checked whether or not the flag FC2 for storing that the delay correction on the receiving side is being performed is set (decision 229).

When the result of judgment 229 is NO,
In this case, since the delay correction on the receiving side is not performed, the flag FC2 is cleared (process 230), the delay operation of the delay circuit 11 is turned off (process 231), and the determination 201 is made.
Return to.

Further, at that time, the voice is not already received, and when the result of the judgment 226 is YES, at that time, the process shifts to the immediate judgment 229. Further, when the result of the determination 229 is NO, at that point, the process immediately shifts to the determination 201.

If another BAS signal is received and the result of determination 225 is NO, the process corresponding to the received BAS signal is executed (process 23).
2) Return to decision 201.

In this way, B received from the partner terminal
In accordance with the AS command, the received video is turned on / off and the received voice is turned on / off, and when the video reception state and the voice reception state are performed, the audio delay correction on the receiving side is performed.

FIG. 15 shows an example of ON / OFF control processing for transmission side delay correction.

First, it is checked whether or not the flag FA1 for storing the transmission side audio transmission state is set (decision 301). When the result of the determination 301 is YES, the transmission side video transmission state is stored. Flag FV1
Is checked (decision 302).

When the result of judgment 302 is YES, it is checked whether or not the flag FC1 for storing that the transmission side delay correction is being performed is set (decision 303). When the result of the judgment 303 is NO, the flag FC1 is set (process 304), the delay operation of the delay circuit 9 is turned on (process 305), and it is checked whether or not the communication is in the end state (judgment 306). Thirty
When the result of 6 is NO, it means that communication is in progress, and therefore the determination 3
Return to 01. Further, when the result of the determination 303 is YES, the process proceeds to the immediate determination 306.

Further, when the result of the judgment 301 is NO, and when the result of the judgment 302 is NO,
This is a state in which the transmission side delay correction is not performed. At this time,
It is checked whether or not the flag FC1 is set (decision 307), and when the result of the decision 307 is YES,
The flag FC1 is cleared (process 308), the delay operation of the delay circuit 9 is turned off (process 309), and the process proceeds to the determination 306. When the result of determination 307 is NO,
Immediate determination 306 is performed.

Further, in the state where the communication is completed, the judgment 306
When the result of is YES, this process ends.

In this way, the image is transmitted by the own terminal,
In addition, when transmitting voice, delay correction of voice on the transmitting side is performed.

By the way, in the above-described embodiment, when the video is transmitted and the audio is transmitted, the audio delay of the transmitting side is corrected, and when the video is received and the audio is received, Although the audio delay correction is performed, if the audio delay correction is performed mainly to solve the lip sync problem, the audio delay correction is not necessary at least when transmitting a still image.

Even if a still image is transmitted / received, if the audio delay correction is performed, the delay overcorrection state is set, which may cause a situation in which the responsiveness of the voice when talking while referring to the still image is deteriorated.

In order to eliminate such a situation, it is preferable not to perform voice delay correction on the transmitting side when transmitting a still image and not perform voice delay correction on the receiving side when receiving a still image. .

In this case, processing examples when the user operates the operation display unit 17 to instruct the source switching are shown in FIGS. 16, 17, and 18.

First, when the source switching operation is instructed (the result of judgment 401 is YES), it is checked whether or not the video source switching is instructed (judgment 402). Judgment 4
When the result of 02 is YES, it is checked whether the main camera device 4 has been selected (decision 403).

When the result of the judgment 403 is YES, a video-on extended BAS command (main camera) which is a command indicating that the terminal itself transmits video data and the video source is the main camera device 4 is issued. It is sent to the partner terminal (process 404), and the multiplexing / separation apparatus 10 is instructed to turn on the video on the transmission side, which indicates that the moving image data is to be sent (process 405), and one of the images is being sent. The flag FV1 for storing the information is set (process 406), the video signal SVm output from the main camera device 4 is selected by the switch 12 (process 407), and the video source is switched to the main camera device 4, Return to determination 401.

When the result of the judgment 403 is NO, it is checked whether or not the document camera device 5 is selected at that time (judgment 408).

When the result of the judgment 408 is YES, a video-on extended BAS command (document camera) which is a command indicating that the terminal itself is transmitting video data and the image source is the document camera device 5. It is sent to the partner terminal (process 409) and the multiplexing / separation device 1
0 is instructed to turn on the video on the transmitting side, which indicates that video data is to be transmitted (process 410), and a flag FV1 for storing that any video is being transmitted.
Is set (process 411), the still image signal SVs output from the document camera device 5 is selected by the switch 12 (process 412), the video source is switched to the main camera device 4, and the process returns to the step 401.

When the result of judgment 408 is NO,
At that time, it is checked whether the video tape reproducing device 6 has been selected (decision 413).

When the result of the judgment 413 is YES, the video-on extended BAS command (playback video) which is a command indicating that the own terminal transmits video data and the video source is the video tape playback device 6 Is sent to the partner terminal (process 414), and the multiplexing / separation apparatus 10 is instructed to turn on the video on the transmitting side, which indicates that video data is to be sent (process 415), and any video is being sent. The flag FV1 for storing the existence is set (step 416), and the reproduced video signal S output from the video tape reproducing device 6 by the switching device 12 is set.
Vp is selected (process 417) and the video source is switched to the video tape reproducing device 6.

When the process 417 is completed, it is next checked whether or not the flag FA1 for storing that any voice is being transmitted is set (decision 418).

When the voice is not being transmitted at that time and the flag FA1 is not set (the result of the judgment 418 is NO), the voice ON BAS which is a command indicating that the own terminal transmits the voice data. The command is sent to the partner terminal (process 419), the multiplexing / demultiplexing device 10 is instructed to turn on the voice on the transmitting side indicating that the voice data is to be transmitted (process 420), and the flag FA1 is set (process 421). The switching mixer 7 selects the reproduced audio signal SA2 output from the video tape reproducing device 6 (process 422) and switches the audio source to the video tape reproducing device 6.

When the result of the judgment 418 is YES, it means that the audio data is already being transmitted and the audio source has been switched from the other audio source to the video tape reproducing device 6, so that the processing shifts to the processing 422. The video source is switched to the video tape reproducing device 6. Then, the processing 422
When is completed, the process returns to the determination 401.

When the result of judgment 413 is NO,
At that time, it is checked whether or not a video off command has been issued (decision 423). When the result of determination 423 is YES, the video off BAS command, which is a command indicating to stop the transmission of the video data of the own terminal, is sent to the partner terminal (process 424), and the multiplexing / demultiplexing device 10
The transmission side video off, which indicates to stop the transmission of the moving image data, is designated (process 425), the flag FV1 is cleared (process 426), and the process returns to the step 401.

When the result of judgment 423 is NO, it means that the video source switching operation has been canceled, and at that point, the process immediately returns to judgment 401.

When the switching of the audio source is instructed and the result of the judgment 402 is NO, it is checked whether or not the microphone 1 is selected (judgment 427).

When the result of judgment 427 is YES, it is checked whether or not the flag FA1 for storing that any one of the voices is being transmitted is set (decision 428).

When the voice is not being transmitted at that time and the flag FA1 is not set (the result of the judgment 428 is NO), the voice ON BAS which is a command indicating that the own terminal transmits the voice data. A command is sent to the partner terminal (process 429), the multiplexing / demultiplexing device 10 is instructed to turn on the voice on the transmission side indicating that the voice data is to be transmitted (process 430), and the flag FA1 is set (process 431). The audio signal SA1 output from the microphone 1 is selected by the switching mixer 7 (process 432),
The sound source is switched to the microphone 1, and the process returns to the determination 401.

If the result of the judgment 428 is YES, it means that the audio data is already being transmitted and the audio source has been switched from another audio source to the microphone 1.
The process moves to process 432 and the audio source is switched to the microphone 1.

When the result of the judgment 427 is NO, it is checked whether or not the video tape reproducing device 6 is selected (decision 433).

When the result of judgment 433 is YES, it is checked whether or not a flag FA1 for storing that any one of the voices is being transmitted is set (decision 434).

When the voice is not being transmitted at that time and the flag FA1 is not set (the result of the judgment 434 is NO), the voice ON BAS which is a command indicating that the own terminal transmits the voice data. The command is sent to the partner terminal (process 435), the multiplexing / separation apparatus 10 is instructed to turn on the voice on the transmitting side indicating that the voice data is to be transmitted (process 436), and the flag FA1 is set (process 437). The audio signal SA2 output from the video tape reproducing device 6 is selected by the switching mixer 7 (process 438), and the audio source is selected as the audio source.
, And the process returns to judgment 401.

When the result of the judgment 434 is YES, the audio data is already being transmitted and the audio source has been switched from the other audio source to the video tape reproducing device 6, so the processing shifts to the processing 438. The audio source is switched to the video tape reproducing device 6.

When the result of judgment 433 is NO,
At that time, it is checked whether a voice off command has been issued (decision 439).

When the result of the judgment 439 is YES, a voice off (mute) BAS command, which is a command indicating to stop the transmission of the voice data of the own terminal, is sent to the partner terminal (step 440), and multiplexing / multiplexing is performed. The separation device 10 is designated to turn off the transmission-side voice that indicates to stop the transmission of voice data (process 441), clears the flag FA1 (process 442), and returns to the determination 401.

When the result of judgment 439 is NO,
At that time, it is checked whether a voice mix is instructed (decision 443). When the result of determination 443 is YES, the switching mixer 7 mixes the audio signal SA1 output from the microphone 1 and the audio signal SA2 output from the video tape reproducing device 6 (processing 444), and the determination 40
Return to 1. Further, when the result of the judgment 443 is NO, it means that the switching operation of the audio source has been canceled, and therefore, the process immediately returns to the judgment 401 at that time.

In this way, when the switching operation of the video source and the audio source is performed, the transmission video information corresponding to the video of the video source after switching is output to the multiplexing / separating device 10 and also after switching. The digital transmission audio data corresponding to the audio of the audio source (including the audio after mixing) is output to the multiplexing / separation device 10. Further, when the video off is operated, the video transmission is stopped, and when the audio off is operated, the audio transmission is stopped. At the same time, the extended BAS command indicating the transmission / stop of the video and the BAS command indicating the transmission / stop of the audio are notified to the other terminal so that the transmission status of the video and the audio of the own terminal is notified. I have to.

Further, as the BAS command representing the video transmission, an extended BAS command extended so that the type of the video source can be displayed is used,
The type of video source can be notified. Here, the extended BAS command is a BAS command that can arbitrarily set an information element in a predetermined format.

At the same time, the flag FV1 for storing the video transmission state and the flag FA1 for storing the voice transmission state are appropriately set according to the video transmission state and the voice transmission state at that time. Is turned on and off.

FIGS. 19 and 20 show an example of processing when the command BAS is received from the partner terminal in this case.

When a batch of BAS signals is received (the result of judgment 501 is YES), B received at that time is received.
It is checked whether the AS signal is an extended BAS command for instructing video-on (decision 502).

When the result of judgment 502 is YES, it is checked at that time whether or not the flag FV2 for storing that any video is being received is set (decision 503).

When the image is not being received at that time and the flag FV2 is not set (the result of the determination 503 is NO), the flag FV2 is set (process 504) and the multiplexer / demultiplexer 10 is set. , A command to turn on the video on the receiving side, which represents reception of moving image data (process 505).

Next, it is checked whether or not the flag FA2 for storing that any voice is being received is set (decision 506). Judgment 506
When the result is YES, it is checked whether or not the video source notified at that time is the document camera device (decision 507).

When the result of the judgment 507 is NO,
It is checked whether or not the flag FC2 for storing that the reception side delay correction is being performed is set (decision 508).

When the result of the judgment 508 is NO,
In this case, since the delay correction on the receiving side is executed, the flag FC2 is set (process 509), the delay operation of the delay circuit 11 is turned on (process 510), and the determination 501 is made.
Return to.

If the image is already being received at that time and the result of determination 503 is YES, then the process immediately shifts to immediate determination 506. When the result of the determination 506 is NO, the result of the determination 507 is Y.
When it becomes ES and when the result of the judgment 508 becomes YES, the processing shifts to the judgment 501 at that time.

When the result of judgment 502 is NO, it is checked whether the BAS signal received at that time is a command for instructing voice-on (decision 511).

When the result of judgment 511 is YES, it is checked at that time whether a flag FA2 for storing that any voice is being received is set (decision 512).

When the voice is not being received at that time and the flag FA2 is not set (the result of the judgment 512 is NO), the flag FA2 is set (process 513) and the multiplexer / demultiplexer 10 is set. , Instructing to turn on the voice on the receiving side, which indicates reception of voice data (Process 5
14).

Next, it is checked whether or not the flag FV2 for storing that any one of the images is being received is set (decision 515). Judgment 515
When the result is YES, it is checked whether or not the video source notified at that time is the document camera device (decision 516).

When the result of judgment 516 is NO,
It is checked whether or not the flag FC2 for storing that the delay correction on the receiving side is being performed is set (decision 517).

When the result of judgment 517 is NO,
In this case, since the delay correction on the receiving side is executed, the flag FC2 is set (process 518), the delay operation of the delay circuit 11 is turned on (process 519), and the determination 501 is made.
Return to.

[0195] Further, at that time, the voice is already being received, and when the result of the judgment 512 is YES, at that time, the process shifts to the immediate judgment 515. When the result of the determination 515 is NO, the result of the determination 516 is Y.
When it becomes ES and when the result of the judgment 517 is YES, at that time, the processing shifts to the judgment 501.

When the result of judgment 511 is NO, it is checked whether or not the BAS signal received at that time is a command for instructing video off (decision 52).
0).

When the result of judgment 520 is YES, it is checked at that time whether a flag FV2 for storing that any video is being received is set (decision 521).

When the image is being received at that time and the flag FV2 is set (determination 5
If the result of step 21 is YES), the flag FV2 is cleared (step 522), and the multiplexer / demultiplexer 10 is instructed to turn off the video on the receiving side, which indicates to stop receiving the moving image data (step 523).

Then, it is checked whether or not the flag FC2 for storing that the reception side delay correction is being performed is set (decision 524).

When the result of the judgment 524 is NO,
In this case, since the delay correction on the receiving side is not performed, the flag FC2 is cleared (process 525), the delay operation of the delay circuit 11 is turned off (process 526), and the determination 501 is made.
Return to.

Further, when the image is not already received at that time and the result of the judgment 521 is NO,
At that point, the process moves to immediate determination 524. Also, the judgment 52
When the result of 4 is NO, the immediate judgment is 5 at that time.
Move to 01.

When the result of judgment 520 is NO, it is checked whether or not the BAS signal received at that time is a command for instructing voice-on (decision 527).

When the result of judgment 527 is YES, it is checked at that time whether a flag FA2 for storing that any voice is being received is set (decision 528).

At that time, the voice is being received, and when the flag FA2 is set (determination 5
If the result of step 28 is YES), the flag FA2 is cleared (step 529), and the multiplexing / separating apparatus 10 is instructed to turn off the receiving side voice, which indicates to stop the reception of voice data (step 530).

Next, it is checked whether or not the flag FC2 for storing that the delay correction on the receiving side is being performed is set (decision 531).

When the result of the judgment 531 is NO,
In this case, since the delay correction on the receiving side is not performed, the flag FC2 is cleared (process 532), the delay operation of the delay circuit 11 is turned off (process 533), and the determination 501 is made.
Return to.

[0207] Further, at that time, the voice is not already received, and when the result of the determination 528 is YES, the process immediately shifts to the immediate determination 531 at that time. Further, when the result of the judgment 531 is NO, at that point, the process immediately shifts to the judgment 501.

If another BAS signal is received and the result of determination 527 is NO, the process corresponding to the received BAS signal is executed (process 53).
4) Return to decision 501.

[0209] In this way, B received from the other terminal
In accordance with the AS command, the received video is turned on / off and the received voice is turned on / off, and when the video is being received by a video source other than the document camera device and the audio is being received, the audio delay of the receiving side is corrected. I have to.

FIG. 21 shows an example of ON / OFF control processing for the transmission side delay correction in this case.

First, it is checked whether or not the flag FA1 for storing the transmission side audio transmission state is set (decision 601). When the result of the determination 601 is YES, the transmission side video transmission state is stored. Flag FV1
Is checked (decision 602).

When the result of judgment 602 is YES, it is checked whether or not the document camera device 5 is selected as the video source at that time (judgment 603).

When the result of judgment 603 is NO,
It is checked whether or not the flag FC1 for storing that the transmission side delay correction is being performed is set (decision 604).

When the result of judgment 604 is NO,
When the flag FC1 is set (process 605), the delay operation of the delay circuit 9 is turned on (process 606), it is checked whether the communication is in the end state (judgment 607), and when the result of the judgment 607 is NO, the communication is terminated. Since it is medium, the process returns to the determination 601. When the result of determination 604 is YES,
Immediate determination 607 follows.

When the result of the determination 601 is NO, when the result of the determination 602 is NO, and when the result of the determination 603 is YES, the transmission side delay correction is not performed. At this time, it is checked whether or not the flag FC1 is set (decision 608). When the result of the decision 608 is YES, the flag FC1 is cleared (process 609) and the delay operation of the delay circuit 9 is turned off (process 609). 610), and shifts to the judgment 607. Further, when the result of the judgment 608 is NO, the process moves to the immediate judgment 607.

Further, in the state where the communication is completed, the judgment 607 is made.
When the result of is YES, this process ends.

In this way, in this embodiment, the terminal itself transmits the image obtained by other than the document camera device 5, and
When transmitting audio, delay correction of audio on the transmitting side is performed.

FIG. 22 shows a case where data is transmitted using two information channels, in which voice data is transmitted at a transmission rate of 16 Kbps and moving image data is transmitted at a transmission rate of 108.8 Kbps. An example of channel allocation of data is shown.

The audio data in this case is CCITT Recommendation G.264. It is formed by applying the audio CODEC method of 728.

This CCITT Recommendation G. Voice CO of 728
The DEC method is an encoding method called a low-delay code excitation type linear prediction method, and the audio data formed by the DEC method is framed with a processing unit of 10 (milliseconds). Subchannel S of one information channel
It is transmitted using CH1 and SCH2.

That is, as shown in FIG.
The voice data for one frame is composed of four voice code frames, each voice code frame consists of 40 bits, and the data of the odd-numbered bits is in the sub-channel SCH1, as shown in FIG. Data of even-numbered bits are arranged in the sub-channel SCH2.

In the above-mentioned embodiment, the case where the present invention is applied to the video conference terminal device has been described, but the present invention can be similarly applied to the multimedia terminal device having other device functions. You can

[0223]

As described above, according to the present invention,
Since the audio data is delayed only while the video is being sent, the response of the conversation is good when exchanging still images or when making a voice-only call, and there is an effect that smooth communication can be performed. .

Further, since the audio data is delayed only in the state where the image of the person is being transmitted, the case where the person is not included in the image, the case of exchanging a still image, and the case of only the audio are used. When the call is made, the responsiveness of the conversation is improved, and there is an effect that smooth communication can be performed.

[Brief description of drawings]

FIG. 1 is a block diagram showing an example of a video conference terminal device according to an embodiment of the present invention.

FIG. 2 is a schematic diagram showing an example of a frame configuration of a multi-frame.

FIG. 3 is a schematic diagram showing an example of a signal configuration of one frame.

FIG. 4 is a schematic diagram showing an example of a frame adjustment signal.

FIG. 5 is a schematic diagram showing an example of a bit allocation signal.

FIG. 6 is a schematic diagram for explaining a signal transmission order.

FIG. 7: TTC Recommendation JT-H. 242 (a domestic standard corresponding to CCITT Recommendation H.242) is a time chart showing an example of a general procedure.

FIG. 8 is a schematic diagram showing an example of bit allocation of audio data, general-purpose data, and moving image data when two information channels are used.

FIG. 9 is a timing chart for explaining audio delay correction.

FIG. 10 is a flowchart showing a part of an example of processing at the time of a source switching operation.

FIG. 11 is a flowchart showing another part of an example of processing at the time of source switching operation.

FIG. 12 is a flowchart showing still another portion of an example of processing at the time of a source switching operation.

FIG. 13 is a flowchart showing a part of an example of processing when a BAS signal is received.

FIG. 14 is a flowchart showing another part of an example of processing when a BAS signal is received.

FIG. 15 is a flowchart showing an example of processing at the time of transmission side delay correction.

FIG. 16 is a flowchart showing a part of another example of processing at the time of source switching operation.

FIG. 17 is a flowchart showing another part of another example of the processing at the time of the source switching operation.

FIG. 18 is a flowchart showing still another part of another example of the processing at the time of the source switching operation.

FIG. 19 is a flowchart showing a part of another example of processing when a BAS signal is received.

FIG. 20 is a flowchart showing another part of another example of processing when a BAS signal is received.

FIG. 21 is a flowchart showing another example of processing at the time of transmission side delay correction.

FIG. 22: Audio data 16 using two information channels
FIG. 3 is a schematic diagram showing an example of bit allocation when performing data transmission of Kbps and moving image data of 108.8 Kbps.

FIG. 23 CCITT Recommendation G. The schematic diagram for explaining the audio data of 728.

[Explanation of symbols]

 9, 11 Delay circuit 16 Control unit

Claims (4)

[Claims] 1. At least a multimedia terminal device capable of transmitting and receiving video data and audio data, a delay correction for delaying audio data according to a predetermined delay time corresponding to a difference between a modulation / demodulation time of video data and a modulation / demodulation time of audio data. And means for delaying the audio data by the delay correction means when moving image data is being transmitted and received, and controlling means for not delaying the audio data by the delay correcting means when the moving image data is not being transmitted and received. A multimedia terminal device characterized by the above. 2. At least in a multimedia terminal device capable of transmitting and receiving moving image data and audio data, a predetermined delay time audio data corresponding to a deviation between a modulation time of transmission moving image data and a modulation time of transmission audio data is delayed. A first delay correction means, a second delay correction means for delaying the audio data by a predetermined delay time corresponding to the deviation between the modulation time of the received video data and the modulation time of the received audio data, and when transmitting the video data While the transmission audio data is transmitted while being delayed by the first delay correction means, a control means for delaying the reception audio data by the second delay correction means when receiving moving image data is provided. Characteristic multimedia terminal device. 3. A delay correction for delaying audio data at least in a multimedia terminal device capable of transmitting and receiving video data and audio data, a predetermined delay time corresponding to a difference between the modulation and demodulation time of video data and the modulation and demodulation time of audio data. Means, and when moving image data based on a person's image is transmitted and received, the delay correction means delays the audio data,
A multimedia terminal device, comprising: a control unit that does not delay the audio data by the delay correction unit when moving image data based on a person's image is not transmitted / received. 4. At least in a multimedia terminal device capable of transmitting and receiving moving image data and audio data, delay a predetermined delay time audio data corresponding to a deviation between a modulation time of transmission moving image data and a modulation time of transmission audio data. A first delay correction means, a second delay correction means for delaying the audio data by a predetermined delay time corresponding to the deviation between the modulation time of the received video data and the modulation time of the received audio data, and a video based on a person's video When transmitting data, the first delay correcting means transmits the transmitted voice data in a delayed state, while when receiving moving image data based on the image of a person, the second delay correcting means receives the received voice data. A multimedia terminal device comprising a control means for delaying the.