KR102770762B1 - Audio encoding/decoding apparatus and method using vector quantized residual error feature - Google Patents

Abstract

An audio encoding/decoding device and method using vector quantized residual error features are disclosed. The audio encoding method may include a step of encoding an original signal to output a bitstream of a main codec; a step of decoding the bitstream of the main codec; a step of determining a residual error feature vector from a feature vector of a decoded signal and a feature vector of the original signal; and a step of encoding the residual error feature vector to output a bitstream of additional information.

Description

{AUDIO ENCODING/DECODING APPARATUS AND METHOD USING VECTOR QUANTIZED RESIDUAL ERROR FEATURE}

The present invention relates to a device and method capable of improving coded sound quality by compressing vector quantized residual error features using a neural network and using them as additional information.

When audio coding technology operates at low bit rates, coding artifacts such as pre-echo and quantization noise may occur, which may degrade audio quality. Various pre- and post-processing techniques are being developed to improve audio quality by removing these coding artifacts.

Ghido, Florin, et al. "Coding of fine granular audio signals using High Resolution Envelope Processing (HREP)." 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2017 is a method of improving sound quality by using the gain value of the high-frequency component of the signal as additional information. It detects a transient signal that may cause pre-echo, flattens the envelope, and uses the additional information transmitted at the decoding stage to return the flattened component to the original component.

Conventional coding sound quality enhancement techniques using additional information have the limitation of limiting the extent of sound quality enhancement because they limit the additional information to the presence or absence of voice, the presence or absence of transient signals, or the gain value of signals in the time-frequency domain.

Therefore, there is a need for a method to improve the sound quality of audio coding without limiting the range of sound quality improvement.

The present invention provides an audio encoding device that encodes residual error features using a neural network and vector quantizes them to transmit them as additional information, and an audio decoding device and method that provide backward compatibility with existing codecs and improve the sound quality of audio signals decoded using existing codecs by post-processing the received additional information using a neural network.

In addition, the present invention performs end-to-end deep learning to jointly train a deep learning model that encodes a residual error feature vector in a side information encoder, a deep learning model that restores the residual error feature vector in a side information decoder, and a deep learning model that estimates a feature vector of an original signal in a post-processing processor, thereby providing a device and method that prevent errors in a side information encoder, a side information decoder, and a post-processing process that use deep learning models from sequentially accumulating.

An audio encoding method according to one embodiment of the present invention may include a step of encoding an original signal to output a bitstream of a main codec; a step of decoding the bitstream of the main codec; a step of determining a residual error feature vector from a feature vector of a decoded signal and a feature vector of the original signal; and a step of encoding the residual error feature vector to output a bitstream of additional information.

The step of outputting the additional information bitstream of the audio encoding method according to one embodiment of the present invention may include the steps of: mapping the residual error feature vector to a latent space; encoding the residual error feature vector corresponding to the latent space by assigning it as a code vector for vector quantization; and outputting the additional information bitstream by quantizing the encoded residual error feature vector.

The additional information encoder for encoding the residual error feature vector of the audio encoding method according to one embodiment of the present invention may be trained according to a loss function determined according to a loss due to encoding of the additional information encoder, a loss due to vector quantization of the additional information decoder for decoding the additional information bitstream, and a difference between the feature vector of the original signal and the feature vector of the original signal estimated from the bitstream of the main codec and the bitstream of the additional information.

An audio encoding method according to one embodiment of the present invention may further include a step of training a side information encoder that encodes the residual error feature vector together with a side information decoder that decodes the side information bitstream and a post-processing processor that estimates a feature vector of an original signal based on a bitstream of the main codec and a bitstream of the side information.

The training step of the audio encoding method according to one embodiment of the present invention may train the additional information encoder, the additional information decoder, and the post-processing processor using a loss function based on a mean squared error (MSE) function and a loss function of a Vector Quantized Variational AutoEncoder (VQ-VAE).

An audio encoding method according to one embodiment of the present invention may further include a step of extracting a feature vector of a decoded signal from acoustic features included in the decoded signal; and a step of extracting a feature vector of an original signal from acoustic features included in the original signal.

An audio decoding method according to one embodiment of the present invention may include the steps of: receiving a bitstream of a main codec and a bitstream of additional information; decoding the bitstream of the main codec; extracting a feature vector of the decoded signal from acoustic features included in the decoded signal; decoding the bitstream of the additional information to restore a residual error feature vector; and estimating a feature vector of an original signal from the feature vector of the decoded signal and the residual error feature vector.

The step of estimating the feature vector of the original signal in the audio decoding method according to one embodiment of the present invention can estimate the feature vector of the original signal by combining the feature vector of the decoded signal and the residual error feature vector.

An audio decoding method according to one embodiment of the present invention may further include a step of converting a feature vector of an estimated original signal into a time domain representation and outputting the converted feature vector.

An audio decoding method according to one embodiment of the present invention may further include a step of training, in an encoding device, an additional information decoder for decoding the additional information bitstream and a post-processing processor for estimating a feature vector of an original signal based on the bitstream of the main codec and the bitstream of the additional information, together with an additional information encoder for encoding the residual error feature vector.

An audio encoding device according to one embodiment of the present invention may include a main codec encoder which encodes an original signal and outputs a bitstream of a main codec; a main codec decoder which decodes the bitstream of the main codec; and an additional information encoder which determines a residual error feature vector from a feature vector of a decoded signal and a feature vector of the original signal, and encodes the residual error feature vector and outputs a bitstream of additional information.

The additional information encoder of the audio encoding device according to one embodiment of the present invention can correspond the residual error feature vector to a latent space, encode the residual error feature vector corresponding to the latent space by assigning it as a code vector for vector quantization, and output an additional information bitstream by quantizing the encoded residual error feature vector.

The additional information encoder of the audio encoding device according to one embodiment of the present invention may be trained according to a loss function determined according to a loss due to encoding of the additional information encoder, a loss due to vector quantization of the additional information decoder, and a difference between a feature vector of the original signal and a feature vector of the original signal estimated from a bitstream of the main codec and a bitstream of the additional information.

The additional information encoder of the audio encoding device according to one embodiment of the present invention can estimate a feature vector of an original signal based on a bitstream of the main codec, a bitstream of the additional information, and an additional information decoder that decodes the additional information bitstream.

The additional information encoder of the audio encoding device according to one embodiment of the present invention can be trained together with the additional information decoder and the post-processing processor using a loss function based on a mean squared error (MSE) function and a loss function of a Vector Quantized Variational AutoEncoder (VQ-VAE).

An audio decoding device according to one embodiment of the present invention may include: a main codec decoder which receives a bitstream of a main codec and decodes the bitstream of the main codec; a feature extractor which extracts a feature vector of the decoded signal from acoustic features included in the decoded signal; a side information decoder which receives a bitstream of side information and decodes the bitstream of the side information to restore a residual error feature vector; and a post-processing processor which estimates a feature vector of an original signal from the feature vector of the decoded signal and the residual error feature vector.

The post-processing processor of the audio decoding device according to one embodiment of the present invention can estimate the feature vector of the original signal by combining the feature vector of the decoded signal and the residual error feature vector.

The post-processing processor of the audio decoding device according to one embodiment of the present invention can convert the feature vector of the estimated original signal into a time domain representation and output it.

The additional information decoder and the post-processing processor of the audio decoding device according to one embodiment of the present invention may be trained together with an additional information encoder that encodes the residual error feature vector in an encoding device and outputs a bitstream of the additional information.

According to one embodiment of the present invention, an audio encoding device encodes residual error features using a neural network, vector quantizes the encoded features, and transmits them as additional information, and an audio decoding device post-processes the received additional information using a neural network, thereby providing backward compatibility with an existing codec and improving the sound quality of an audio signal decoded using an existing codec.

In addition, according to one embodiment of the present invention, by performing end-to-end deep learning that jointly trains a deep learning model that encodes a residual error feature vector, a deep learning model that restores the residual error feature vector, and a deep learning model that estimates a feature vector of an original signal, errors in an additional information encoder, an additional information decoder, and a post-processing process using deep learning models can be prevented from sequentially accumulating.

In addition, the present invention performs end-to-end deep learning that jointly trains deep learning models, thereby effectively training a code vector that quantizes a compressed latent vector, thereby extracting additional information for improving sound quality during an audio encoding process.

FIG. 1 is a drawing showing an audio encoding device and an audio decoding device according to one embodiment of the present invention.
FIG. 2 is an example of the operation of an audio encoding device and an audio decoding device according to an embodiment of the present invention.
FIG. 3 is an example of performance evaluation for the output of an audio decoding device according to an embodiment of the present invention.
FIG. 4 is an example of sound quality evaluation for the output of an audio decoding device according to an embodiment of the present invention.
FIG. 5 is an example of a spectrogram of a signal output by an audio decoding device according to an embodiment of the present invention.
FIG. 6 is a flowchart illustrating an audio encoding method according to an embodiment of the present invention.
FIG. 7 is a flowchart illustrating an audio decoding method according to an embodiment of the present invention.

Hereinafter, an embodiment of the present invention will be described in detail with reference to the attached drawings. An audio encoding method and an audio decoding method according to an embodiment of the present invention can be performed by an audio encoding device (110) and an audio decoding device (120).

FIG. 1 is a drawing showing an audio encoding device and an audio decoding device according to one embodiment of the present invention.

The audio encoding device (110) may include a main codec encoder (111), a main codec decoder (112), a feature extractor (113), a feature extractor (114), and an additional information encoder (115), as illustrated in FIG. 1. At this time, the main codec encoder (111), the main codec decoder (112), the feature extractor (113), the feature extractor (114), and the additional information encoder (115) may be different processes, or may be each module included in one process.

The main codec encoder (111) is the original signal can output the bitstream of the main codec by encoding it. For example, the main codec can be a legacy codec such as HE-AAC (High-Efficiency Advanced Audio Coding).

The main codec decoder (112) decodes the bitstream of the main codec and outputs the decoded signal. can output.

The feature extractor (113) decoded the signal Signal decoded from acoustic features included in The feature vector X _d can be extracted.

The feature extractor (114) extracts the original signal Original signal from acoustic features included in The feature vector X _o can be extracted. For example, the feature extractor (113) and the feature extractor (114) can extract the feature vector X _d and the feature vector X _o using at least one of various types of acoustic features, such as LPS (log power spectra).

The additional information encoder (115) decoded the signal Feature vector X _d and original signal The residual error feature vector X _r can be determined from the feature vector X _o . Then, the additional information encoder (115) can encode the residual error feature vector X _r to output a bitstream of the additional information. At this time, the residual error feature vector X _r can satisfy mathematical expression 1.

At this time, the additional information encoder (115) can map the residual error feature vector X _r to a latent space having a smaller dimension than the dimension of the residual error feature vector. For example, the dimension of the residual error feature vector X _r input to the additional information encoder (115) may be 257, and the dimension of the bitstream output from the additional information encoder (115) may be 32. In addition, the latent space may be a space expressing latent information inherent in observed data.

Next, the additional information encoder (115) can encode the residual error feature vector corresponding to the latent space by assigning it as a code vector for vector quantization. Next, the additional information encoder (115) can quantize the encoded residual error feature vector to output the additional information bitstream.

At this time, the additional information encoder (115) may be trained according to a loss function determined according to a loss due to encoding and vector quantization of the additional information encoder (115), and a difference between the feature vector of the original signal and the feature vector of the original signal estimated from the bitstream of the main codec and the bitstream of the additional information by the post-processing processor (1240). At this time, the additional information encoder (115) may be trained together with the additional information decoder (123) and the post-processing processor (124). For example, the additional information encoder (115), the additional information decoder (123), and the post-processing processor (124) may perform joint training according to a loss function L expressed as in mathematical expression 2.

At this time, is the loss due to encoding and vector quantization for optimization of the additional information encoder (115), may be a loss due to decoding and vector quantization of the additional information decoder (123). In addition, is the feature vector X _o of the original signal in the post-processing processor (124) and the feature vector of the original signal estimated by the post-processing processor (124). It can be a difference between the loss function and the mean squared error (MSE). In addition, the loss function can be trained using various optimization methods, such as the mean squared error (MSE).

In summary, the additional information encoder (115) can be trained together with the additional information decoder (123) and the post-processing processor (124) using a loss function based on the mean squared error (MSE) function and the loss function of the VQ-VAE (Vector Quantized Variational AutoEncoder). For example, the additional information encoder (115), the additional information decoder (123), and the post-processing processor (124) can be trained using the MSE function as shown in Equation 2. ) and the function of VQ-VAE( , ) can be trained using a loss function that combines

The audio decoding device (110) may include a main codec decoder (121), a feature extractor (122), an additional information decoder (123), and a post-processing processor (124) as illustrated in FIG. 1. At this time, the main codec decoder (121), the feature extractor (122), the additional information decoder (123), and the post-processing processor (124) may be different processes, or may be individual modules included in one process.

The main codec decoder (121) can receive the bitstream of the main codec from the main codec encoder (111) of the audio encoding device (110). Then, the main codec decoder (121) decodes the received bitstream of the main codec and outputs the decoded signal. can output. In addition, the main codec decoder (121) can operate in the same manner as the main codec decoder (112) of the audio signal encoding device (110).

The feature extractor (122) decodes the signal decoded by the main codec decoder (121). Signal decoded from acoustic features included in The feature vector X _d of the audio signal encoding device (110) can be extracted. In addition, the feature extractor (122) can operate in the same manner as the feature extractor (112) of the audio signal encoding device (110).

The additional information decoder (123) can receive a bitstream of additional information from the additional information encoder (115) of the audio encoding device (110). Then, the additional information decoder (123) can decode the received bitstream of additional information to restore a residual error feature vector.

The post-processing processor (124) decrypts the signal The feature vector X _d and the residual error feature vector restored by the additional information decoder (123) The feature vector of the original signal can be estimated from the post-processing processor (124). Then, the feature vector of the estimated original signal Representing the time domain can be converted into and output. At this time, the post-processing processor (124) outputs the feature vector X _d and the residual error feature vector can be combined to estimate the feature vector of the original signal.

The audio encoding device (110) according to the present invention encodes residual error features using a neural network, vector quantizes them, and transmits them as additional information, and the audio decoding device (120) post-processes the received additional information using a neural network, thereby providing backward compatibility with existing codecs and improving the sound quality of audio signals decoded with existing codecs.

FIG. 2 is an example of the operation of an audio encoding device and an audio decoding device according to an embodiment of the present invention.

Original signal can be input to the main codec encoder (111) and feature extractor (114) as shown in Fig. 2.

At this time, the main codec encoder (111) It can be encoded and transmitted to the main codec decoder (112) of the audio encoding device (120) and the main codec decoder (121) of the audio decoding device (120).

And, the main codec decoder (112) and the main codec decoder (121) each decode the received bitstream and output the decoded signal. can output.

The feature extractor (113) decoded the signal Signal decoded from acoustic features included in The feature vector X _d of the original signal can be extracted. In addition, the feature extractor (114) Original signal from acoustic features included in The feature vector X _o can be extracted.

At this time, the additional information encoder (115) The feature vector X _o and the decoded signal The residual error feature vector X _r , which is the difference between the feature vectors X _{d ,} can be determined. Then, the additional information encoder (115) can encode the residual error feature vector X _r to output a bitstream of the additional information. For example, the neural network that the additional information encoder (115) uses to encode the residual error feature vector X _r can be formed as a deep learning model having the structure (210). In addition, the output code vector of the additional information encoder (115) can be assigned as a representative code vector of the VQ codebook (220). At this time, the representative code vector can be a code vector having the closest distance between vectors among the vectors included in the VQ codebook (220). For example, the distance between vectors can be calculated using the Euclidean distance, etc.

Next, the additional information encoder (115) can quantize the encoded residual error feature vector to output an additional information bitstream. At this time, the additional information bitstream can include a codebook index (code vector index) of the additional information. Then, the additional information encoder (115) can transmit the codebook (220) and the additional information bitstream to the additional information decoder (123).

The additional information decoder (123) can decode the bitstream of the additional information received from the additional information encoder (115) of the audio encoding device (110) to restore the residual error feature vector. For example, the neural network used by the additional information decoder (123) to decode the residual error feature vector X _r can be formed as a deep learning model having a structure (230). At this time, the additional information decoder (123) can restore the residual error feature vector using the code vector of the code book (220).

The feature extractor (122) decodes the signal decoded by the main codec decoder (121). Signal decoded from acoustic features included in The feature vector X _d can be extracted.

The concatenate operator (201) combines the feature vector X _d with the residual error feature vector restored by the additional information decoder (123). The result of performing a join operation on can be input to the post-processing processor (124). Then, the post-processing processor (124) uses a deep learning module having a structure (240). The feature vector of the original signal can be estimated from the post-processing processor (124). Then, the feature vector of the estimated original signal At this time, the waveform restorer (202) can output the feature vector of the estimated original signal. Representing the time domain It can be converted to and printed.

The audio encoding device (110) and the audio decoding device (120) perform end-to-end deep learning by jointly training a deep learning model that encodes a residual error feature vector in the additional information encoder (115), a deep learning model that restores the residual error feature vector in the additional information decoder (123), and a deep learning model that estimates a feature vector of an original signal in the post-processing processor (124), thereby preventing errors in the additional information encoder (115), the additional information decoder (123), and the post-processing process (124) that use deep learning models from sequentially accumulating.

In addition, the audio encoding device (110) and the audio decoding device (120) perform end-to-end deep learning by jointly training a deep learning model having a structure (210), a deep learning model having a structure (230), and a deep learning module having a structure (240), thereby effectively training a code vector that quantizes a compressed latent vector and extracting additional information for improving sound quality in an audio encoding process. Specifically, the audio encoding device (110) and the audio decoding device (120) train the deep learning model having a structure (210), the deep learning model having a structure (230), and the deep learning module having a structure (240) to minimize the loss function of mathematical expression 2, thereby optimizing the additional information encoder (115), the additional information decoder (123), the codebook (220), and the post-processing processor (124).

FIG. 3 is an example of performance evaluation for the output of an audio decoding device according to an embodiment of the present invention.

The performance evaluation of an audio decoding device using the NeroAAC codec among MPEG-4 high-efficiency advanced audio coding (HE-AAC) v1 (NeroAAC), the performance evaluation of an audio decoding device adding a post-processor to the NeroAAC codec (+PP only), and the performance evaluation of an audio decoding device (120) using the NeroAAC codec as the main codec (Prop. (+0.6 kbps)) may be as shown in the upper table of Fig. 3. The table of Fig. 3 is an example of performance measured using the ITU-T Recommendation P.862.2 wideband perceptual evaluation of speech quality (PESQ), which is a standardized speech quality evaluation tool.

In addition, the performance evaluation of an audio decoding device using the QAAC codec (QAAC), the performance evaluation of an audio decoding device adding a post-processor to the QAAC codec (+PP only), and the performance evaluation of an audio decoding device (120) using the QAAC codec as the main codec (Prop. (+0.6 kbps)) may be as shown in the lower table of FIG. 3.

As illustrated in FIG. 3, the audio encoding device (110) and the audio decoding device (120) according to one embodiment of the present invention can have a higher average PESQ score than a method that uses only a post-processing module in a main codec operating at a higher bit rate, even though the additionally used bit rate is about 0.6 kbps.

FIG. 4 is an example of sound quality evaluation for the output of an audio decoding device according to an embodiment of the present invention.

The graph (410) can represent the degree to which the quality of a signal decoded by an audio decoding device with a post-processor added to the NeroAAC codec is improved compared to a signal decoded by an audio decoding device using the NeroAAC codec operating at 16 kbps (+PP only), and the degree to which the quality of a signal decoded by an audio decoding device (120) using the NeroAAC codec as the main codec is improved compared to a signal decoded by an audio decoding device using the NeroAAC codec operating at 16 kbps (Prop. (+0.6 kbps)).

In addition, the graph (420) can represent the degree to which the quality of a signal decoded by an audio decoding device with a post-processor added to the QAAC codec is improved compared to a signal decoded by an audio decoding device using the QAAC codec operating at 16 kbps (+PP only), and the degree to which the quality of a signal decoded by an audio decoding device (120) using the QAAC codec as the main codec is improved compared to a signal decoded by an audio decoding device using the QAAC codec operating at 16 kbps (Prop. (+0.6 kbps)).

At this time, the graph (410) and the graph (420) may be results measured according to the MUltiple Stimuli with Hidden Reference and Anchor (MUSHRA) test, which is one of the methods for performing a codec listening test to evaluate the quality of a codec output signal.

According to graph (410) and graph (420), it can be confirmed that the signal decoded by the audio decoding device (120) according to one embodiment of the present invention is improved by 9.73 points in NeroAAC and 7.93 points in QAAC compared to the signal decoded by the audio decoding device that only uses post-processing for the main codec.

FIG. 5 is an example of a spectrogram of a signal output by an audio decoding device according to an embodiment of the present invention.

The spectrogram (510) of FIG. 5 can represent an original signal (a), a signal decoded by an audio decoding device (120) using the NeroAAC codec as the main codec (b), a signal decoded by an audio decoding device adding a post-processor to the NeroAAC codec (c), and a signal decoded by an existing audio decoding device using the NeroAAC codec (d).

In addition, the spectrogram (520) of FIG. 5 can represent an original signal (a), a signal decoded by an audio decoding device (120) using the QAAC codec as the main codec (b), a signal decoded by an audio decoding device adding a post-processor to the QAAC codec (c), and a signal decoded by an existing audio decoding device using the QAAC codec (d).

According to spectrogram (510) and spectrogram (520), it can be confirmed that the high-frequency band that is not well restored in (c) is well restored in (b).

FIG. 6 is a flowchart illustrating an audio encoding method according to an embodiment of the present invention.

At step (610), the main codec encoder (111) The bitstream of the main codec can be output by encoding it. At this time, the main codec encoder (111) can transmit the bitstream of the main codec to the audio decoding device (120).

In step (620), the main codec decoder (112) decodes the bitstream of the main codec output in step (610) and outputs the decoded signal. can output.

At step (630), the feature extractor (113) decrypts the signal decoded at step (620). Signal decoded from acoustic features included in The feature vector X _d can be extracted.

At step (640), the feature extractor (114) extracts the original signal Original signal from acoustic features included in The feature vector X _o can be extracted.

In step (650), the additional information encoder (115) decoded the signal Feature vector X _d and original signal The residual error feature vector X _r can be determined from the feature vector X _o .

In step (660), the additional information encoder (115) can output a bitstream of the additional information by encoding the residual error feature vector X _r . At this time, the additional information encoder (115) can correspond the residual error feature vector X _r to a latent space. Next, the additional information encoder (115) can encode the residual error feature vector corresponding to the latent space by assigning it as a code vector for vector quantization. Next, the additional information encoder (115) can quantize the encoded residual error feature vector and output a bitstream of the additional information.

FIG. 7 is a flowchart illustrating an audio decoding method according to an embodiment of the present invention.

In step (710), the main codec decoder (121) can receive a bitstream of the main codec from the main codec encoder (111) of the audio encoding device (110). Then, the main codec decoder (121) decodes the received bitstream of the main codec and outputs a decoded signal. can output.

In step (720), the feature extractor (122) decode the signal decoded in step (710). Signal decoded from acoustic features included in The feature vector X _d can be extracted.

In step (730), the additional information decoder (123) can receive a bitstream of additional information from the additional information encoder (115) of the audio encoding device (110). Then, the additional information decoder (123) can decode the received bitstream of additional information to restore a residual error feature vector.

In step (740), the post-processing processor (124) decrypts the signal The feature vector X _d and the residual error feature vector restored by the additional information decoder (123) The feature vector of the original signal can be estimated from the feature vector X d . At this time, the post-processing processor (124) calculates the feature vector X _d and the residual error feature vector can be combined to estimate the feature vector of the original signal.

In step (750), the post-processing processor (124) estimates the feature vector of the original signal. Representing the time domain It can be converted to and printed.

The audio encoding device (110) of the present invention encodes residual error features using a neural network, vector quantizes them, and transmits them as additional information, and the audio decoding device (120) post-processes the received additional information using a neural network, thereby providing backward compatibility with existing codecs and improving the sound quality of audio signals decoded with existing codecs.

In addition, the present invention performs end-to-end deep learning by jointly training a deep learning model that encodes a residual error feature vector in an additional information encoder (115), a deep learning model that restores the residual error feature vector in an additional information decoder (123), and a deep learning model that estimates a feature vector of an original signal in a post-processing processor (124), thereby preventing errors in the additional information encoder (115), the additional information decoder (123), and the post-processing process (124) that use deep learning models from sequentially accumulating.

In addition, the present invention performs end-to-end deep learning that jointly trains deep learning models, thereby effectively training a code vector that quantizes a compressed latent vector, thereby extracting additional information for improving sound quality during an audio encoding process.

Meanwhile, the method according to the present invention can be written as a program that can be executed on a computer and can be implemented in various recording media such as a magnetic storage medium, an optical reading medium, and a digital storage medium.

The implementations of the various technologies described herein may be implemented as digital electronic circuitry, or as computer hardware, firmware, software, or combinations thereof. The implementations may be implemented as a computer program product, i.e., a computer program tangibly embodied in an information carrier, e.g., a machine-readable storage medium (computer-readable medium) or a radio signal, for processing by the operation of a data processing device, e.g., a programmable processor, a computer, or multiple computers, or for controlling the operation thereof. A computer program, such as the computer program(s) described above, may be written in any form of programming language, including compiled or interpreted languages, and may be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. The computer program may be deployed to be processed on one computer or multiple computers at a single site, or distributed across multiple sites and interconnected by a communications network.

Processors suitable for processing a computer program include, for example, both general-purpose and special-purpose microprocessors, and any one or more processors of any type of digital computer. Typically, the processor will receive instructions and data from a read-only memory or a random access memory or both. Elements of the computer may include at least one processor for executing instructions and one or more memory devices for storing instructions and data. Typically, the computer may include, or be coupled to receive data from, transmit data to, or both, one or more mass storage devices for storing data, such as magnetic, magneto-optical disks, or optical disks. Information carriers suitable for embodying computer program instructions and data include, by way of example, semiconductor memory devices, magnetic media such as hard disks, floppy disks, and magnetic tape, optical media such as CD-ROMs (Compact Disk Read Only Memory) and DVDs (Digital Video Disk), magneto-optical media such as floptical disks, ROMs (Read Only Memory), RAMs (Random Access Memory), flash memory, EPROMs (Erasable Programmable ROM), EEPROMs (Electrically Erasable Programmable ROM), etc. The processor and memory may be supplemented by, or included in, special purpose logic circuitry.

Additionally, the computer-readable medium can be any available medium that can be accessed by a computer, and can include both computer storage media and transmission media.

While this specification contains details of a number of specific implementations, these should not be construed as limitations on the scope of any invention or what may be claimed, but rather as descriptions of features that may be unique to particular embodiments of particular inventions. Certain features described herein in the context of separate embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented in multiple embodiments, either individually or in any suitable subcombination. Furthermore, although features may operate in a particular combination and may initially be described as being claimed as such, one or more features from a claimed combination may in some cases be excluded from that combination, and the claimed combination may be modified into a subcombination or variation of a subcombination.

Likewise, although operations are depicted in the drawings in a particular order, this should not be understood to imply that the operations must be performed in the particular order illustrated or in any sequential order to achieve a desirable result, or that all of the illustrated operations must be performed. In certain cases, multitasking and parallel processing may be advantageous. Furthermore, the separation of the various device components of the embodiments described above should not be understood to require such separation in all embodiments, and it should be understood that the program components and devices described may generally be integrated together in a single software product or packaged into multiple software products.

Meanwhile, the embodiments of the present invention disclosed in this specification and drawings are merely specific examples presented to help understanding, and are not intended to limit the scope of the present invention. It will be apparent to those skilled in the art to which the present invention pertains that other modified examples based on the technical idea of the present invention can be implemented in addition to the embodiments disclosed herein.

110: Audio encoding device
111: Main Codec Encoder
112: Main Codec Decoder
113: Feature Extractor
114: Feature Extractor
115: Additional Information Encoder
120: Audio Decoding Device
121: Main Codec Decoder
122: Feature Extractor
123: Additional Information Decoder
124: Post-processing processor

Claims (16)

A step of encoding the original signal and outputting the bitstream of the main codec;
A step of decrypting a bitstream of the above main codec;
A step of determining a residual error feature vector from the feature vector of the decoded signal and the feature vector of the original signal;
A step of encoding the above residual error feature vector to output a bitstream of additional information; and
A step of performing end-to-end deep learning by jointly training an additional information encoder that encodes the residual error feature vector, an additional information decoder that decodes the additional information bitstream, and a post-processing processor that estimates the feature vector of the original signal based on the bitstream of the main codec and the bitstream of the additional information.
An audio signal encoding method comprising: In the first paragraph
The step of outputting the above additional information bitstream is:
A step of mapping the above residual error feature vector to a latent space;
A step of encoding by assigning the residual error feature vector corresponding to the latent space as a code vector for vector quantization; and
A step of quantizing the encoded residual error feature vector to output a side information bitstream.
An audio signal encoding method comprising: In the first paragraph,
The additional information encoder that encodes the above residual error feature vector,
An audio signal encoding method, wherein the training is performed according to a loss function determined according to a loss due to encoding of the above-mentioned additional information encoder, a loss due to vector quantization of the above-mentioned additional information decoder that decodes the above-mentioned additional information bitstream, and a difference between a feature vector of the original signal and a feature vector of the original signal estimated from the bitstream of the main codec and the bitstream of the above-mentioned additional information. delete In the first paragraph,
The above training steps are:
An audio signal encoding method for training the additional information encoder, the additional information decoder, and the post-processing processor using a loss function based on a mean squared error (MSE) function and a loss function of a Vector Quantized Variational AutoEncoder (VQ-VAE). In the first paragraph,
A step of extracting a feature vector of the decoded signal from the acoustic features included in the decoded signal; and
A step of extracting a feature vector of the original signal from the acoustic features included in the original signal.
An audio signal encoding method further comprising: A step of receiving a bitstream of a main codec and a bitstream of additional information;
A step of decrypting a bitstream of the above main codec;
A step of extracting a feature vector of a decoded signal from acoustic features included in the decoded signal;
A step of decoding the bitstream of the above additional information to restore the residual error feature vector;
A step of estimating the feature vector of the original signal from the feature vector of the decoded signal and the residual error feature vector.
A step of performing end-to-end deep learning by jointly training an additional information decoder that decodes the additional information bitstream, a post-processing processor that estimates a feature vector of an original signal based on the bitstream of the main codec and the bitstream of the additional information, and an additional information encoder that encodes the residual error feature vector in an encoding device.
A method for decoding an audio signal including: In Article 7,
The step of estimating the feature vector of the above original signal is:
An audio signal decoding method for estimating a feature vector of the original signal by combining the feature vector of the decoded signal and the residual error feature vector. In Article 7,
A step of converting the feature vector of the estimated original signal into a time domain representation and outputting it.
A method for decoding an audio signal further comprising: delete A computer-readable recording medium having recorded thereon a program for executing the method of any one of claims 1 to 3 and claims 5 to 9. A main codec encoder that encodes the original signal and outputs the bitstream of the main codec;
A main codec decoder for decoding a bitstream of the above main codec; and
An additional information encoder that determines a residual error feature vector from the feature vector of the decoded signal and the feature vector of the original signal, and encodes the residual error feature vector to output a bitstream of additional information.
Including,
An audio signal encoding device that performs end-to-end deep learning by jointly training an additional information encoder that encodes the residual error feature vector, an additional information decoder that decodes the additional information bitstream, and a post-processing processor that estimates the feature vector of the original signal based on the bitstream of the main codec and the bitstream of the additional information. In Article 12
The above additional information encoder,
An audio signal encoding device that corresponds the above residual error feature vector to a latent space, encodes the residual error feature vector corresponding to the latent space by assigning it as a code vector for vector quantization, and outputs an additional information bitstream by quantizing the encoded residual error feature vector. In Article 12,
The above additional information encoder,
An audio signal encoding device trained according to a loss function determined according to a loss due to encoding of the above-mentioned additional information encoder, a loss due to vector quantization of the above-mentioned additional information decoder, and a difference between a feature vector of the original signal and a feature vector of the original signal estimated from the bitstream of the main codec and the bitstream of the above-mentioned additional information. delete In Article 12,
The above additional information encoder,
An audio signal encoding device trained together with the additional information decoder and the post-processing processor using a loss function based on a mean squared error (MSE) function and a loss function of a Vector Quantized Variational AutoEncoder (VQ-VAE).

Images