CN104702856A - System device and method for synthesizing MV with real-time Selfie special effects applied to accompaniment player - Google Patents

Abstract

A system device and method for synthesizing MV with special effect of autodyne in real time, which is applied to the karaoke, the device includes an image input interface, in order to produce an original image data; the special effect image library is used for storing at least one piece of special effect image data; the lyric caption library is used for storing lyric caption data of at least one song; the dynamic image synthesis processing unit is coupled with the image input interface and the special effect image library, and reads the at least one special effect image data from the special effect image library and carries out real-time synthesis processing on the original image data so as to generate synthetic image data; the core processing unit is coupled with the lyric caption library, is in communication connection with the dynamic image synthesis processing unit, and reads lyric caption data of the at least one song from the lyric caption library to carry out superposition processing and word coating processing; and the image output interface is coupled with the core processing unit and used for outputting the synthesized image data and the lyric caption data of the at least one song to a display device.

Description

System device and method for synthesizing MV with real-time Selfie special effects applied to accompaniment player

technical field

The present invention relates to a system device and method thereof, in particular to a system device and method for synthesizing MV with real-time Selfie special effects applied to an accompaniment player.

Background technique

Generally, when playing karaoke equipment, the built-in preset or externally stored scenery, plot, situational video or picture is used as the background, and the lyrics and subtitles are superimposed on it, or the music video (MV) of the original singer is played directly. , providing user accompaniment. However, most users want to be the main character in the music video (MV) when they are singing. Although the current more advanced karaoke equipment can allow users to connect external image input devices, such as video cameras, they can only capture live images. Monotonous original user images cannot produce dynamic visual effects similar to MV clips containing user images and various song scene changes in real time. However, if you want to use computer post-production to achieve this effect, not only lack of real-time performance, but also a huge cost .

Contents of the invention

One of the purposes of the present invention is to provide a system device and method for synthesizing MV (Music Video) with real-time self-portraits and special effects applied to a singing machine (Karaoke machine/Karaoke/Karaoke), which can display or record in real time the content of the user's collocation. Various situations and visual effects.

In order to achieve what is described above, the present invention provides a system device for synthesizing MV with real-time Selfie special effects applied to accompaniment players, including:

an image input interface for inputting an image output by an image output device to generate an original image data;

a special effect image database for storing at least one special effect image data;

A lyric and subtitle library for storing the lyric and subtitle data of at least one song;

A dynamic image synthesis processing unit, coupled to the image input interface and the special effect image library, for receiving and processing the original image data, and reading the at least one special effect image data and the special effect image data from the special effect image library performing real-time composite processing on the original image data to generate composite image data;

A core processing unit, coupled to the lyrics and subtitle library and communicated with the dynamic image synthesis processing unit, for receiving and processing the synthesized image data, and reading the at least one song from the lyrics and subtitle library Lyric subtitle data and superimposed processing and doodling processing with the synthetic image data; and

An image output interface, coupled to the core processing unit, for outputting the synthesized image data and the lyrics and subtitle data of the at least one song to a display device, so that the display device can display the synthesized image data and the Lyric subtitle data describing at least one song.

Wherein, the image output by the image output device is a real-time image of a user.

Wherein, the special effect image data is at least one of the following: snowflake special effect image data, star special effect image data, bubble special effect image data, scene special effect image data.

Wherein, the special effect image data includes at least one of the following: a dynamic object and a static object.

Wherein, the system device further includes an operation unit, the operation unit is coupled to the core processing unit, and the operation unit has an operation interface for the user to input instructions to make the core processing unit and the dynamic image The synthesis processing unit performs corresponding operations according to the instructions input from the operation interface.

Wherein, the system device further includes an audio processing unit, the audio processing unit is coupled to the core processing unit, and the audio processing unit is used to receive at least one user's voice and perform it with the background music of a corresponding song. The audio mixing process is superimposed and mixed to generate a mixed audio data.

Wherein, the system device further includes a recording unit, the recording unit is coupled to the core processing unit, and the recording unit is used to selectively record the synthesized image data, the audio mixing data, and the The lyrics and subtitle data are used to form an audio-video file.

Wherein, the system device further includes a storage medium, the storage medium includes an internal storage medium and an external storage medium, the internal storage medium is coupled to the dynamic image synthesis processing unit and the core processing unit, and the The internal storage medium is used to store the lyrics subtitle library and the special effect image library, the external storage medium is coupled to the recording unit and the core processing unit, and the external storage medium is used to store the video files .

Wherein, the system device also includes a network interface, the network interface is coupled to the core processing unit for linking to a network, so that the audio and video files are read by the core processing unit and uploaded to the network to share.

In order to achieve what is described above, the present invention provides a method for synthesizing an MV with real-time self-timer special effects applied to an accompaniment machine, which is used to synthesize at least one user's image in real time, and the method for synthesizing an MV with real-time self-timer special effects applied to an accompaniment machine includes:

receiving an original image data of the user;

Discriminating and selecting at least one special effect image data;

synthesizing the original image data and the special effect image data in real time to form a synthetic image data;

superimposing a lyric subtitle data; and

and displaying the synthesized image data and the lyrics and subtitle data.

Further, the method further includes performing dynamic image pre-processing on the original image data.

Further, the method further includes performing dynamic image post-processing on the synthetic image data.

Further, the method for discriminating and selecting at least one special effect image data is to randomly select the at least one special effect image data from a special effect image library according to a random number rule.

Further, the method for discriminating and selecting at least one special effect image data is to sequentially select the at least one special effect image data from a special effect image library according to a specific order.

Further, the method for discriminating and selecting at least one special effect image data is to select the at least one special effect image data from a special effect image library according to an instruction input by an operating unit.

Further, the method of synthesizing the original image data and the special effect image data in real time is to perform face recognition on the original image data of the user and capture the user's face as the foreground and the special effect image data. The image data is subjected to real-time de-background synthesis to form the synthesized image data.

Further, the method of synthesizing the original image data and the special effect image data in real time is to perform human body recognition on the original image data of the user and capture the whole body of the user as the foreground and the special effect image data performing real-time de-background synthesis to form the synthesized image data.

In order to further understand the features and technical content of the present invention, please refer to the following detailed description and accompanying drawings of the present invention. However, the accompanying drawings are provided for reference and illustration only, and are not intended to limit the present invention.

Description of drawings

FIG. 1 shows a schematic diagram of an architecture of an embodiment of the present invention;

FIG. 2 shows a functional block diagram of an embodiment of the present invention;

Fig. 3 represents the schematic diagram of an embodiment of the present invention;

Fig. 4 shows a flow chart of a method of the present invention.

Explanation of reference signs:

User A

Accompaniment machine M

Accompaniment machine real-time audio-visual special effects synthesis device 100

Image output device C

Display device D

Screen D1

Video input interface 11

Video output interface 12

Core Processing Unit 13

Dynamic image synthesis processing unit 14

VFX Library 15

Lyric subtitle library 16

Internal storage media 17

External storage media18

Operating unit 19

Operation interface 191

Audio Processing Unit 20

recording unit 21

network interface 22

Detailed ways

Please refer to FIG. 1 , the present invention mainly provides a system device for synthesizing MV with real-time Selfie special effects applied to a song player. As shown in FIG. 1 , the system device of the present invention for synthesizing MV with real-time self-timer special effects applied to a chorus player includes a real-time audio-visual special effects synthesis device 100 for a chorus player, an image output device C, and a display device D. The device 100 for synthesizing real-time audio-visual special effects with a vocal player can be coupled between an image output device C and a display device D. As shown in FIG. The accompaniment machine M referred to in the present invention is a device capable of processing multimedia data and reproducing multimedia data and having the function of accompaniment, and the accompaniment machine M can also be called a karaoke machine/karaoke/karaoke. In addition, it should be emphasized that the term "video and audio" in the present invention can essentially include at least one of video or audio, and is not limited to include both video and audio formats.

Please refer to Figure 2 and Figure 3. FIG. 2 is a functional block diagram of a preferred embodiment of the present invention. As shown in FIG. 2 , the device 100 for synthesizing real-time audio-visual effects for a singing player basically includes an image input interface 11 , an image output interface 12 , a core processing unit 13 , a dynamic image synthesis processing unit 14 , and a special effect image library 15 . Furthermore, the real-time audio-visual special effect synthesis device 100 of the accompaniment player can also include a lyrics subtitle library 16, an internal storage medium 17, an external storage medium 18, an operating unit 19, an audio processing unit 20, a recording unit 21, and A network interface 22 .

The image input interface 11 can be connected to the image output device C for inputting the image output by the image output device C. Wherein, the image output device C may include, but not limited to, a charge-coupled device (CCD), a camera, a computer or a network camera (PC-cam, Web-cam), or a camera and other devices capable of capturing images. For any Media player. The image output interface 12 can be connected to the display device D for outputting images to the display device D. Wherein, the display device D may include but not limited to a CRT, a liquid crystal display (LED, LCD) or a plasma display, a projector, and the like. Wherein, the image input interface 11 and the image output interface 12 may include but not limited to HDMI interface, CVBS interface, component terminal interface, AV terminal interface, VGA interface, DVI interface, wireless signal transmission interface and so on.

The core processing unit 13 can be a central processing unit (CPU) or a system-on-a-chip (SOC), and can be a main control component for controlling the accompaniment player M.

The dynamic image synthesis processing unit 14 is coupled to the image input interface 11 and communicated with the core processing unit 13 . The dynamic image synthesis processing unit 14 can be an independent digital signal processor (DSP), or can be integrated into the core processing unit 13 or installed in a set-top box (set-top box). In addition, an executable program can be written in the video synthesis processing unit 14 . The dynamic image synthesis processing unit 14 can receive an original image data generated by the image captured by the image output device C through the image input interface 11 . The image captured by the image output device C can be presented on the screen D1 of the display device D through the image output interface 12 . In this embodiment, the image output device C is a video camera, which can capture a continuous moving image. The continuous dynamic image captured by the image output device C is displayed on the screen D1 of the display device D in real time at least one user A (or can be called a singer) captured by the image output device C, that is, captured by the image output device C The continuous dynamic image taken is the real-time image of the user. It can also be said that the image output device C can capture the real-time image of the user A in real time and convert it into an original image data received by the dynamic image synthesis processing unit 14 after analog/digital conversion.

The special effect image library 15 can be built in the internal storage medium 17 or directly placed in the dynamic image synthesis processing unit 14 . The internal storage medium 17 may include but not limited to a flash memory, a temporary memory, and a random access memory. The special effect image library 15 pre-stores a plurality of special effect image data for selection and application. The special effect image database 15 includes at least the following: snowflakes, star special effect image data, bubble special effect image data, scene special effect image data, face special effect images, etc., which often appear in MV films and have various special effect images with situational effects and visual effects. data. The above-mentioned special effect image data may include a dynamic object and a static object.

After the dynamic image synthesis processing unit 14 receives the original image data generated by the image captured by the image output device C through the image input interface 11, it reads a special effect image data or more than one special effect image data from the special effect image library 15 according to the instruction and adds With the application, the original image data and the special effect image data are synthesized in real time to generate a composite image data which is output to the core processing unit 13 and displayed on the screen D1 of the display device D through the image output interface 12 in real time. In detail, the dynamic image synthesis processing unit 14 can automatically perform various predetermined dynamic image processing and predetermined dynamic synthesis processing on the original image data according to the program written in it, such as: brightness, hue, contrast, saturation, screen division , stack, rotation processing, etc. It can also perform face recognition and background synthesis processing, or face copying and background synthesis processing, or human body recognition and background synthesis processing, so that the original image data presents a picture similar to an MV film effect. Furthermore, the processed original image data is synthesized with the special effect image data in real time to generate composite image data.

The lyrics and subtitle library 16 can be built in the internal storage medium 17 and pre-store the lyrics and subtitle data of multiple songs. The core processing unit 13 is coupled to the lyrics and subtitle library 16, and after the core processing unit 13 receives the synthesized image data output by the dynamic image synthesis processing unit 14, the lyrics and subtitle data of the corresponding song can be read by the lyrics and subtitle library 16, and the corresponding The lyrics and subtitle data of the song and the synthesized image data are superimposed and painted. The word painting process refers to the color of the lyrics displayed in the lyrics and subtitle data from white to blue as the song progresses. Of course, it can also be filled from white to red. Or green, there is no limit to the color of the words here, and the progress of the current song can be clearly separated through different visual effects (for example: the dot effect presented according to the progress of the song), and then the synthesized image data and lyrics The subtitle data is presented in real time on the screen D1 of the display device D through the video output interface 12 , so as to present a screen similar to an MV movie effect and a screen containing lyrics graffiti effect in real time.

The operation unit 19 is coupled to the core processing unit 13 . The operation unit 19 may include but not limited to a remote controller, a tablet, a smart phone, a sensor and so on. The operation unit 19 has an operation interface 191 , which can be a plurality of functional keys, or a graphical user interface, a human motion sensing interface, a voice control interface, and the like. Therefore, instructions can be given by human body movements or voices, or by pressing or touching functional buttons or functional images on the operation interface 191 to allow users to input instructions, so that the core processing unit 13 and the dynamic image synthesis processing unit 14 can be processed according to the user's own The command input by the operation interface 191 is used to perform the corresponding operation. Therefore, when the screen D1 of the display device D shows the user A who is in the image output device C in real time, the user A can issue an instruction through the operation interface 191 to select a special effect image data or two special effect image data from the special effect image database 15. The above special effect image data and the original image data are synthesized in real time with special effects, and presented on the screen D1 of the display unit D in real time for the user A to watch in real time. For example, when the user A wants to present the scene of being in snow in the screen D1, he can issue an instruction through the operation interface 191 to select snowflake special effect image data from the special effect image library 15 for real-time synthesis, or he can combine it with changing the color tone. Real-time synthesis of special effect image data, you can feel like you are in a snowy and discolored situation. In addition, other special effect image data can also be selected from the special effect image database 15, for example, facial special effect image data can be synthesized in real time to present the visual effect of facial special effects. In addition, the user A can also issue instructions through the function settings of the operation interface 191 to add artistic effects, such as oil painting effects, watercolor effects, sketch effects, and other processing. To this end, user A issues an instruction through the operation interface 191 to select the preferred special effect image data and the original image data containing user A to perform real-time synthesis to present a variety of situational effects and visual effects in real time during singing. In the screen D1 of the display device D, the user is like an MV film director.

The audio processing unit 20 can be built in the core processing unit 13 or be a separate audio chip or sound card coupled to the core processing unit 13 . The audio processing unit 20 is used to receive multiple or one user A's vocals (or the user's singing voice) and select the background music of a corresponding song from the internal storage medium 17 to carry out the mixing process of superposition and mixing, so as to Generate a mix data.

The recording unit 21 can be built in the core processing unit 13 or be a separate recording chip coupled to the core processing unit 13 . The recording unit 21 is used for selectively recording composite image data, audio mixing data, and song subtitle data to form an audio-video file. For this reason, user A can set the recording function option through the operation interface 191 of the operation unit 19, and select only video recording or simultaneous video recording and audio recording, so as to record the real-time video or singing voice that contains user A's self-portrait and synthesis, and further store it. The medium 18 is stored externally. The external storage medium 18 may include a hard disk, a CD, a flash drive, and a memory card.

The network interface 22 can be an Ethernet network adapter or a wireless network adapter coupled to the core processing unit 13 . The network interface 22 is used for wired or wireless connection to a network. The aforementioned network may be a wireless network, a local area network or the Internet. To this end, user A can set the network function option through the operation interface 191 of the operation unit 19 to perform a network link, and then make the video and audio files stored in the external storage medium 18 be read by the core processing unit 13 and uploaded to the network for sharing. In addition, wireless transmission can also be directly performed through the network interface 22, so that the video and audio files can be transmitted unlimitedly to tablets or smart phones within the range of the wireless signal for sharing. Allow user A to take a selfie and synthesize the real-time video and audio to record the audio and video file, which can be immediately shared with his relatives and friends, specific people or the public for viewing.

Please refer to Figure 4, and refer to Figure 2 together. As shown in Figure 4, it is the main process steps of a method for synthesizing MV with real-time self-timer special effects applied to the accompaniment player of the present invention. This method is used to synthesize at least one user A's image in real time, and the user's image can be composed of any image The output device C captures in real time and undergoes encoding, decoding and compression processing to generate the user's original image data. First, the original image data of the above-mentioned user is received by the dynamic image synthesis processing unit 14 (S201). At this time, the original image data can be subjected to dynamic image pre-processing (for example: brightness, hue, contrast, saturation, screen division, etc.) Stacking, rotation processing, etc.) (S203), then it can be judged by the dynamic image synthesis processing unit 14 and select at least one special effect image data (S205) from the special effect image library 15 (for example: snowflake special effect image data, raining special effect image data, star special effect image data, scene special effect image data). Wherein, the method for judging and selecting the special effect image data can be to randomly select at least one special effect image data from a special effect image database 15 according to a random number rule, or sequentially select the special effect image data from a special effect image database 15 according to a specific order. At least one special effect image data, or select the at least one special effect image data from a special effect image library 15 according to the instruction input by the operation unit 19 according to the user's preference, or automatically select the corresponding special effect image data. Afterwards, the dynamic image synthesis processing unit 14 performs real-time dynamic synthesis of the processed original image data and special effect image data to form a composite image data ( S207 ). Among them, the method of real-time dynamic synthesis of processed original image data and special effect image data can use face recognition technology to capture the user's face as the foreground, and then perform real-time background removal and synthesis with special effect image data to form synthetic image data , or use human body recognition technology to capture the user's whole body as the foreground, and then perform real-time de-background synthesis with special effect image data to form synthetic image data. Afterwards, the synthetic image data can be subjected to dynamic image post-processing (S209) as required (for example: oil painting effect, watercolor effect, sketch effect and other art effect processing), in other words, the synthetic image data can be calculated according to the instruction to make the synthetic image data Presents a picture with artistic effects such as oil painting, watercolor or sketch. Finally, add lyric and subtitle data (including graffiti effect) (S211), and display the processed synthetic image data and lyric and subtitle data on the screen D1 of a display device D (S211), to present a similar MV in real time Movie effects and frames with doodle effects. In addition, it is also possible to receive a mixed sound data that includes the user's singing voice and the background music of a song, and choose to record only synthetic image data, lyrics and subtitle data, or mixed sound data to form a video file, and then store the video file or Send this video file for sharing.

The above descriptions are only preferred feasible embodiments of the present invention, and do not limit the patent scope of the present invention. Therefore, all equivalent technical changes made by using the description and illustrations of the present invention are included in the scope of the present invention.

Claims (17)

1. be applied to a system and device of the real-time auto heterodyne special efficacy synthesis MV of accompanying video, it is characterized in that, comprising:

One image input interface, in order to input the image that an image output device exports, to produce raw video data;

One special efficacy Image Database, in order to store at least one special efficacy image data;

One dynamic image synthesis processing unit, couple described image input interface and described special efficacy Image Database, in order to receive and to process described raw video data, and synthesize process in real time, to produce resultant image data by the described special efficacy Image Database described at least one special efficacy image data of reading and described raw video data;

One lyrics captions storehouse, in order to store the lyrics caption data of at least one song;

One core processing unit, couple described lyrics captions storehouse and synthesize processing unit communication with described dynamic image and be connected, in order to receive and to process described resultant image data, and read the lyrics caption data of at least one song described by described lyrics captions storehouse and carry out superposition process with described resultant image data and be coated with word processing; And

One image output interface, couple described core processing unit, in order to export lyrics caption data to display device of described resultant image data and at least one song described, described display device is made to show the lyrics caption data of described resultant image data and at least one song described.

2. the system and device being applied to the real-time auto heterodyne special efficacy synthesis MV of accompanying video according to claim 1, it is characterized in that, the image that described image output device exports is the real-time imaging of a user.

3. the system and device being applied to the real-time auto heterodyne special efficacy synthesis MV of accompanying video according to claim 1, it is characterized in that, described special efficacy image data is at least following wherein a kind of: snowflake special efficacy image data, star special efficacy image data, bubble special efficacy image data, scene special effect image data.

4. the system and device being applied to the real-time auto heterodyne special efficacy synthesis MV of accompanying video according to claim 1, it is characterized in that, described special efficacy image data at least comprises following wherein a kind of: a dynamic object, a static object.

5. the system and device being applied to the real-time auto heterodyne special efficacy synthesis MV of accompanying video according to claim 1, it is characterized in that, also comprise an operating unit, described operating unit couples described core processing unit, and described operating unit has an operation-interface, for user input instruction, described core processing unit and described dynamic image synthesis processing unit is made to carry out corresponding operation according to the instruction inputted from described operation-interface.

6. the system and device being applied to the real-time auto heterodyne special efficacy synthesis MV of accompanying video according to claim 1, it is characterized in that, also comprise an audio treatment unit, described audio treatment unit is coupled to described core processing unit, described audio treatment unit carries out the stereo process of superposition mixing, to produce audio mixing data in order to the background music of the voice song corresponding to that receive at least one user.

7. the system and device being applied to the real-time auto heterodyne special efficacy synthesis MV of accompanying video according to claim 6, it is characterized in that, also comprise a recording elements, described recording elements is coupled to described core processing unit, and described recording elements is used for formation one audio and video file in order to optionally to record described resultant image data, described audio mixing data and described lyrics caption data.

8. the system and device being applied to the real-time auto heterodyne special efficacy synthesis MV of accompanying video according to claim 7, it is characterized in that, also comprise a Storage Media, described Storage Media comprises internal reservoir media and an outside Storage Media, described internal reservoir media couple described dynamic image synthesis processing unit and described core processing unit, and described internal reservoir media are in order to deposit described lyrics captions storehouse and described special efficacy Image Database, described outside Storage Media couples described recording elements and described core processing unit, and described outside Storage Media is in order to deposit described audio and video file.

9. the system and device being applied to the real-time auto heterodyne special efficacy synthesis MV of accompanying video according to claim 8, it is characterized in that, also comprise a network interface, described network interface is coupled to described core processing unit, in order to link to a network, and then be uploaded to described network after described audio/video file is read by described core processing unit and shared.

10. be applied to a method of the real-time auto heterodyne special efficacy synthesis MV of accompanying video, be used for synthesizing in real time the image of at least one user, it is characterized in that, described in be applied to accompanying video the real-time auto heterodyne special efficacy method of synthesizing MV comprise:

Receive raw video data of described user;

Differentiate and select at least one special efficacy image data;

Real-time synthesis described raw video data and described special efficacy image data, be used for formation one resultant image data;

A lyrics caption data on superposition; And

Show described resultant image data and described lyrics caption data.

11. methods being applied to the real-time auto heterodyne special efficacy synthesis MV of accompanying video according to claim 10, is characterized in that, also comprise and carry out dynamic image pre-treatment to described raw video data.

12. methods being applied to the real-time auto heterodyne special efficacy synthesis MV of accompanying video according to claim 10, is characterized in that, also comprise and carry out dynamic image reprocessing to described resultant image data.

13. methods being applied to the real-time auto heterodyne special efficacy synthesis MV of accompanying video according to claim 10, it is characterized in that, described differentiation also selects the method for at least one special efficacy image data to be from a special efficacy Image Database, select described at least one special efficacy image data randomly according to a random number rule.

14. methods being applied to the real-time auto heterodyne special efficacy synthesis MV of accompanying video according to claim 10, it is characterized in that, described differentiation also selects the method for at least one special efficacy image data to be from a special efficacy Image Database, select described at least one special efficacy image data in order according to specific order.

15. methods being applied to the real-time auto heterodyne special efficacy synthesis MV of accompanying video according to claim 10, it is characterized in that, described differentiation also selects the method for at least one special efficacy image data to be that described at least one special efficacy image data is selected in the instruction inputted according to an operating unit from a special efficacy Image Database.

16. methods being applied to the real-time auto heterodyne special efficacy synthesis MV of accompanying video according to claim 10, it is characterized in that, the method of described real-time synthesis described raw video data and described special efficacy image data performs face recognition to the described raw video data of described user and the face capturing described user goes background to synthesize as prospect and described special efficacy image data, to form described resultant image data in real time.

17. methods being applied to the real-time auto heterodyne special efficacy synthesis MV of accompanying video according to claim 10, it is characterized in that, the method of described real-time synthesis described raw video data and described special efficacy image data performs human body identification to the described raw video data of described user and the whole body capturing described user goes background to synthesize as prospect and described special efficacy image data, to form described resultant image data in real time.