CN117596373A - Method and electronic device for information display based on dynamic digital human image - Google Patents

Abstract

The embodiment of the application discloses a method and electronic equipment for information display based on dynamic digital human images, wherein the method comprises the following steps: obtaining video contents of a plurality of real visual angles, wherein the video contents of the plurality of real visual angles are obtained by synchronously shooting a process of showing a real person on a target garment in a state of wearing the target garment from different visual angles through a plurality of camera devices; extracting character images contained in the video frames from a plurality of video frames contained in the video content respectively; generating character images at corresponding time points for a plurality of intermediate perspectives between adjacent real perspectives using an image-based rendering technique; and generating multi-view video according to the character images of the real view angles and the intermediate view angles at a plurality of time points so as to match the multi-view video into a preset virtual 3D space scene model at a client. By the embodiment of the application, more real visual experience can be provided for the user at lower cost.

Description

Method and electronic device for information display based on dynamic digital human image

Technical field

This application relates to the field of information processing technology, and in particular to methods and electronic devices for information display based on dynamic digital human images.

Background technique

With the rapid development of technology, especially the maturity of computer vision, augmented reality (AR) and virtual reality (VR) technologies, virtual and real fusion technology is attracting more and more attention. This technology provides users with an immersive experience by integrating real-world elements and computer-generated virtual elements, and it opens up new application areas and opportunities for various industries. Especially in industries such as entertainment, advertising, and e-commerce, the rise of these technologies not only brings new experiences to consumers, but also opens up new markets and marketing opportunities for enterprises. For example, in the e-commerce platform, for the apparel industry, functions such as "model catwalk" and "fitting room" can be provided to bring an immersive visual experience to consumers.

However, the process of realizing the above functions also poses new technical and design challenges. For example, in the "model catwalk" scene, including how to provide a more realistic virtual environment and "digital people", etc. Among them, regarding "digital human", in the existing technology, there are usually the following implementation methods:

Method 1, traditional 3D portrait modeling technology: use high-precision 3D scanning equipment to scan a real model, and then manually or semi-automatically perform texture mapping and detail carving to generate a human body model. However, this method usually requires expensive scanning equipment and highly skilled experts to process the scanned data. It takes a long time from scanning to the completed model, and it is mainly suitable for static modeling, dynamic expressions and motion capture needs. extra work.

Method 2, AI-based portrait synthesis: Use neural networks and a large amount of training data to directly synthesize or convert the portrait perspective and posture. However, this method requires a large amount of annotated data for training. In some complex scenes and angles, the generated results may not be realistic enough. In addition, real-time applications may require high-performance hardware support and high computing requirements.

Method three, stereoscopic image rendering scheme based on depth images: use color images and corresponding depth images to generate new viewpoint images. With depth information, the method can estimate the 3D structure of the scene and render the scene from a new viewpoint. However, the output quality of this method depends heavily on the quality of the depth image, and low-quality or inaccurate depth information may also lead to artifacts or distortions in the rendered image. In addition, the acquisition cost of the depth image is relatively high and requires some methods. hardware, lidar, etc., which can add cost and complexity.

Therefore, how to provide users with a more realistic visual experience at a lower cost in the process of information display based on digital people has become a technical problem that needs to be solved by those skilled in the art.

Contents of the invention

This application provides a method and electronic device for information display based on dynamic digital human images, which can provide users with a more realistic visual experience at a lower cost.

This application provides the following solutions:

A method of information display based on dynamic digital human images, including:

Obtain video content from multiple real perspectives. The video content from multiple real perspectives is obtained by synchronously shooting the process of real people displaying the target clothing while wearing the target clothing from different perspectives using multiple camera devices. of;

Extract the character images contained in multiple video frames contained in the video content respectively;

Using image-based rendering technology, based on the pixel offset relationship information between adjacent real-view person images at the same time point, generate character images at corresponding time points for multiple intermediate viewing angles between the adjacent real-view angles;

Generate multi-view videos based on the multiple real views and the multiple intermediate view character images at multiple time points, so as to match the multi-view videos to the preset virtual 3D space scene model on the client, To provide a dynamic digital human image to display the target clothing in the virtual 3D space scene, and to provide an interactive effect for simulating continuous perspective switching.

Wherein, the video content of multiple real perspectives is obtained by synchronously shooting the process of real people walking in the target space wearing the target clothing from different perspectives through multiple camera devices to display the target clothing. of.

Wherein, generating character images corresponding to time points from multiple intermediate perspectives between adjacent real perspectives includes:

Using image-based rendering technology, based on the pixel offset relationship information between adjacent real-view person images at the same time point, the pixel positions of multiple intermediate views between the adjacent real-view views at the corresponding time points are processed. Estimation to generate images of people at multiple time points from the intermediate perspective.

Wherein, the estimating the pixel positions of multiple intermediate views between the adjacent real views at corresponding time points includes:

Taking the character images corresponding to adjacent real-view angles at the same time point as input, a deep learning model is used to fit the dense optical flow field between the adjacent real-perspective character images, and the dense optical flow field is used to The pixel positions of the human image at corresponding time points from multiple intermediate viewing angles between the adjacent real viewing angles are estimated.

Among them, it also includes:

The multi-view video is compressed for transmission to the client.

Wherein, the compression processing of the multi-view video includes:

The video frames corresponding to multiple viewing angles are spliced in units of time points to obtain a frame sequence formed by multiple combined frames; wherein, the multiple video frames corresponding to multiple viewing angles at the time point are divided into multiple sets, Multiple video frames in each set are spliced into a combined frame, so that video frames from adjacent perspectives are located at the same position of adjacent combined frames;

A general video encoder is used to encode the frame sequence formed by the multiple combined frames, and inter-frame compression is performed on the multiple combined frames.

Among them, the resolution of each combined frame is lower than the maximum resolution supported by the end device.

Among them, it also includes:

After the frame sequence formed by multiple combined frames is encoded and inter-frame compressed, the frame sequence is also sliced so that the slices obtained after slicing are transmitted as units, and the receiving end is independently decoded in units of segments. Play.

Wherein, when encoding the frame sequence formed by multiple combined frames, it also includes:

The keyframe interval during inter-frame encoding is controlled based on the number of combined frames included in each slice in order to reduce the number of frames in the same slice that are encoded as keyframes.

Wherein, when encoding the frame sequence formed by multiple combined frames, it also includes:

For combined frames other than key frames, the number of frames encoded into bidirectional reference frames in the same slice is increased by lowering the judgment threshold for bidirectional reference frames.

A method of information display based on dynamic digital human images, including:

In response to a viewing request initiated by the user, a multi-view video is obtained. The multi-view video is generated through the process of displaying the target clothing on a real person wearing the target clothing from different perspectives through multiple camera devices. Synchronous shooting is performed to obtain video content from multiple real-view angles, and the character images contained in the multiple video frames contained in the video content from multiple real-view angles are extracted respectively, and image-based rendering technology is used to create adjacent real-view angle images. Generate character images from a plurality of intermediate perspectives, and generate the multi-view video based on the multiple real perspectives and the character images at multiple time points from the multiple intermediate perspectives;

decoding the multi-view video;

Match the multi-view video to a preset virtual 3D space scene model to provide content in which a dynamic digital human image displays the target clothing in the virtual 3D space scene;

In response to the interactive operation of continuous perspective switching, the interactive effect of simulating continuous perspective switching is provided by switching to character image video content from other perspectives.

A method for generating dynamic digital human images, including:

Obtain video content from multiple real viewing angles, which is obtained by synchronously shooting the process of real people performing target actions from different viewing angles through multiple camera devices;

Extract the character images contained in multiple video frames contained in the video content respectively;

Using image-based rendering technology, based on the pixel offset relationship information between adjacent real-view person images at the same time point, generate character images at corresponding time points for multiple intermediate viewing angles between the adjacent real-view angles;

Multi-view videos are generated based on the multiple real view angles and the multiple intermediate view character images at multiple time points, so that corresponding dynamic digital human images can be displayed through the multi-perspective videos.

A method for displaying clothing information based on dynamic digital human images, including:

In response to a request to display the target clothing through a dynamic digital human, a virtual 3D space scene model is obtained, as well as a dynamic digital human image expressed in the form of a multi-view video, wherein the multi-view video is used to display the target from multiple perspectives. Display the process of displaying the target clothing while the character is wearing the target clothing;

Match the multi-view video to the virtual 3D space scene model to provide content of a dynamic digital human image displaying the target clothing in the virtual 3D space scene, and provide a method for simulating continuous viewing angle switching. interactive effect.

A device for displaying information based on dynamic digital human images, including:

A video content acquisition unit is used to obtain video content from multiple real perspectives. The video content from multiple real perspectives is performed on real people wearing the target clothing from different perspectives through multiple camera devices. It is obtained by synchronously shooting the display process;

A character image extraction unit, configured to extract character images contained in multiple video frames contained in the video content;

A perspective synthesis unit, configured to use image-based rendering technology to generate correspondences for multiple intermediate perspectives between adjacent true perspectives based on pixel offset relationship information between adjacent real perspective images of people at the same time point. Images of people at points in time;

A multi-view video storage and unit is configured to generate a multi-view video based on the multiple real views and the multiple intermediate view character images at multiple time points, so as to match the multi-view video to the predetermined The virtual 3D space scene model is placed in the virtual 3D space scene model to provide the content of the dynamic digital human image displaying the target clothing in the virtual 3D space scene, and to provide an interactive effect for simulating continuous perspective switching.

A device for displaying information based on dynamic digital human images, including:

A multi-view video acquisition unit is configured to obtain a multi-view video in response to a viewing request initiated by a user. The multi-view video is generated in the following manner: using multiple camera devices to view real people wearing target clothing from different perspectives. The process of displaying the target clothing is synchronously shot to obtain video content from multiple real perspectives, and the character images contained in the multiple video frames contained in the video content from the multiple real perspectives are extracted respectively, and image-based Rendering technology to generate character images for multiple intermediate perspectives between adjacent real perspectives, and generate the multi-view video based on the multiple real perspective and the character images of the multiple intermediate perspectives at multiple time points;

A decoding unit, used to decode the multi-view video;

Adding a unit for matching the multi-view video to a preset virtual 3D space scene model to provide content in which a dynamic digital human image displays the target clothing in the virtual 3D space scene;

The perspective switching interaction unit is used to respond to the interactive operation of continuous perspective switching and provide an interactive effect that simulates continuous perspective switching by switching to character image video content from other perspectives.

A device for generating dynamic digital human images, including:

A video content acquisition unit is used to obtain video content from multiple real viewing angles. The video content from multiple real viewing angles is obtained by synchronously shooting the process of real people performing target actions from different viewing angles through multiple camera devices;

A character image extraction unit, configured to extract character images contained in multiple video frames contained in the video content;

A perspective synthesis unit, configured to use image-based rendering technology to generate correspondences for multiple intermediate perspectives between adjacent true perspectives based on pixel offset relationship information between adjacent real perspective images of people at the same time point. Images of people at points in time;

A dynamic digital human asset generation unit, configured to generate multi-view videos based on the multiple real views and the multiple intermediate view character images at multiple time points, so as to generate corresponding dynamic digital images through the multi-view videos. Display of human image.

A device for displaying clothing information based on dynamic digital human images, including:

A request receiving unit, configured to obtain a virtual 3D space scene model and a dynamic digital human image expressed in the form of a multi-view video in response to a request to display the target clothing through a dynamic digital human, wherein the multi-view video is used for Demonstrate the process of displaying the target clothing by the target person while wearing the target clothing from multiple perspectives;

A display unit configured to match the multi-view video to the virtual 3D space scene model to provide content for displaying the target clothing by a dynamic digital human image in the virtual 3D space scene, and to provide content for Simulate the interactive effect of continuous perspective switching.

A computer-readable storage medium on which a computer program is stored, which implements the steps of any of the foregoing methods when executed by a processor.

An electronic device including:

one or more processors; and

A memory associated with the one or more processors, the memory being used to store program instructions that, when read and executed by the one or more processors, perform any of the foregoing methods. A step of.

According to the specific embodiments provided in this application, this application discloses the following technical effects:

Through the embodiments of this application, when it is necessary to provide users with functions such as "model show", the process of displaying the target clothing by real people wearing the target clothing can be synchronized from different perspectives through multiple camera devices. Shoot to obtain video content from multiple real-view angles. Afterwards, the character images contained in the multiple video frames contained in the video content can be extracted respectively, and then image-based rendering technology is used to render the same video content based on adjacent real-view angles. The pixel offset relationship information between the person images at the time point is used to generate person images corresponding to the time point for multiple intermediate perspectives between the adjacent real viewing angles. Then, a multi-view video can be generated based on the multiple real viewing angles and the multiple intermediate viewing angles of character images at multiple time points, so that the character image video content from one of the viewing angles can be added to the preset virtual Rendering and display are performed in the 3D space scene model to provide the content of the dynamic digital person displaying the target clothing in the virtual 3D space scene, and to provide an interactive effect for simulating continuous perspective switching. In this way, there is no need to explicitly model the model character, and only the images taken from multiple perspectives can be used to achieve a 3D-like real-life digital human effect, avoiding complex modeling processes and high modeling costs, thus enabling Get low-cost, high-fidelity, dynamic real-time interactive digital effects. Moreover, since the format of the real-life digital human assets created in the embodiment of the present application is ordinary HEVC and other format videos, it can be parsed by most mobile devices and avoids complex rendering processes and large computing overhead.

In addition, for multiple video contents corresponding to multiple viewing angles, specifically during video encoding, multiple video frames corresponding to the multiple viewing angles can be spliced in units of time points to obtain a video frame formed by multiple combined frames. frame sequence, and divide the multiple video frames corresponding to the multiple perspectives at that time point into multiple sets, and splice the multiple video frames in each set into a combination, so that the video frames from adjacent perspectives are located adjacent to each other. The same position of the combined frame. After that, a general video encoder can be used to encode the frame sequence formed by the multiple combined frames, and by performing inter-frame compression processing on the multiple combined frames, the redundancy between video frames of adjacent perspectives can be eliminated or reduced. remaining information. In this way, since the video frames from multiple perspectives are grouped and spliced, the resolution of the spliced combined frame is not too high, which facilitates real-time decoding in most terminal devices; in addition, since the grouped splicing is performed, , the grouping method and arrangement method are controlled so that video frames from adjacent perspectives are located at the same position of adjacent combined frames, that is, video frames from adjacent perspectives are located in different but adjacent combined frames, and in The positions in different combined frames are the same, and video frames from adjacent perspectives have relatively high similarities. Therefore, adjacent combined frames spliced in this way have relatively high similarities, and then through universal The inter-frame compression algorithm can obtain a higher compression rate by eliminating or reducing redundant information between video frames from adjacent perspectives. In other words, in the embodiment of the present application, the ideal compression rate can be obtained through a universal video encoder. Correspondingly, the decoding can be completed by using a universal decoder at the decoding end, so that it can be supported on more terminal devices. .

Of course, implementing any product of this application does not necessarily require achieving all the above-mentioned advantages at the same time.

Description of drawings

In order to more clearly explain the embodiments of the present application or the technical solutions in the prior art, the drawings needed to be used in the embodiments will be briefly introduced below. Obviously, the drawings in the following description are only some of the drawings of the present application. Embodiments, for those of ordinary skill in the art, other drawings can also be obtained based on these drawings without exerting creative efforts.

Figure 1 is a schematic diagram of the system architecture provided by the embodiment of the present application;

Figure 2 is a flow chart of the first method provided by the embodiment of the present application;

Figure 3 is a schematic diagram of a frame rearrangement method provided by an embodiment of the present application;

Figure 4 is a flow chart of the second method provided by the embodiment of the present application;

Figure 5 is a flow chart of the third method provided by the embodiment of the present application;

Figure 6 is a flow chart of the fourth method provided by the embodiment of the present application;

Figure 7 is a schematic diagram of an electronic device provided by an embodiment of the present application.

Detailed ways

The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, rather than all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by those of ordinary skill in the art fall within the scope of protection of this application.

First of all, it should be noted that in the embodiments of this application, scenarios such as "model shows" can be provided to users in application systems such as product information service systems. The effect to be achieved in this scenario is usually: users can The client sees a "digital person" walking in a certain spatial scene while wearing certain clothing, or performing certain display actions, and it is usually necessary to provide users with an interactive effect of continuous perspective switching. Among them, Digital Human/Meta Human is a digital character created using digital technology that is close to a human image. That is to say, when the user is viewing this "model show" interface through a terminal device such as a mobile phone, he or she can switch to an "any" viewing angle by performing a sliding operation on the mobile phone screen, just like the user is in person. It is the same as the "catwalk" scene, and you can go to different locations to watch the models' catwalk process.

In the existing technology, there are also products related to "model shows" in some product information service systems. However, modeling is mainly used to generate "3D digital people", and then the 3D clothing models are matched to such products. "3D digital people", and simulate the effect of real people "walking on the catwalk". During this process, interactive effects such as multi-view viewing can be provided. However, since the "3D digital human" is generated through 3D modeling, it will look obviously "cartoon-like" and not realistic enough. Movements such as walking and posture will also be unnatural; in addition, it is difficult to fully restore the characteristics of clothing through 3D clothing models, etc.

In the embodiment of this application, in order to provide users with a more realistic visual experience in scenes such as "model show" and "fitting room" in the product information service system, so that users can see more realistic characters. (rather than a cartoon image that seems relatively lacking in realism), it provides an implementation plan for using "real digital people" to provide users with functions such as "model shows".

Specifically, multi-view videos can be used to replace 3D modeling in the existing technology. That is to say, videos from multiple views can be obtained through multi-camera shooting, etc., and then, users can be provided with "multi-view videos" The "model show" function makes the characters and clothing status in the "model" show more realistic. Moreover, since the assets actually produced exist in the form of multi-view videos rather than models, it can support rendering and display on more terminal devices.

In the process of realizing the "model show" through the above-mentioned multi-perspective videos, there are still the following problems: As mentioned above, in the "model show" function, it is usually necessary to provide users with the interactive effect of continuous perspective switching, for example, let Users can continuously switch perspectives by sliding the screen. However, in the embodiment of this application, since a specific "digital person" is represented through multi-view video, and the viewing angles are discrete, therefore, during the client display process, if the viewing angle needs to be switched, it can only be done at these fixed Switching between viewing angles, for example, switching from viewing angle 1 to viewing angle 2, if the distance between viewing angles is relatively large, the user may feel an obvious jump phenomenon. Of course, in theory, if the number of camera devices is large enough and the distribution is dense enough, so that video content from more viewing angles can be obtained, then during playback on the playback end, even when switching between discrete viewing angles, The interactive effect of continuous perspective switching can also be simulated to a certain extent. However, more camera equipment means higher costs.

Therefore, in order to be able to simulate continuous viewing angle switching at a lower cost, embodiments of the present application also adopt image-based rendering technology. Through this technology, images actually captured from two adjacent viewing angles can be used to estimate multiple The pixel position on the intermediate perspective and generates the image of the intermediate perspective. That is, there is no need to actually increase the number of camera devices. Instead, image-based rendering technology can be used to supplement images from multiple intermediate perspectives where camera devices are not actually deployed, thereby generating image content from more perspectives. , thereby achieving the purpose of better simulating continuous perspective switching.

Of course, during specific implementation, it may also be necessary to provide a variety of different scenes in the "model show" function, such as indoor scenes, outdoor scenes, technology scenes, etc. If these scenes are built directly in the real space, then The cost may still be relatively high, and the same model will need to re-complete the catwalk in multiple different scenes and re-shoot, etc. Therefore, the time cost will also be relatively high.

For this reason, in the embodiment of the present application, the generation of real digital people and the creation of scenes can be performed independently. After the multi-angle video content is captured through the camera device, the model characters in the video can be "cut out" and subsequently The image synthesis of the intermediate perspective can be performed based on the character image obtained after cutout. In addition, a variety of different 3D virtual space models can be provided. When displaying on the client, multi-view character image videos can be put into the 3D virtual space scene model for display. In this way, the same multi-view character image video can be displayed in different 3D virtual space scene models.

In addition, since the embodiments of this application involve multi-view videos, and in order to simulate continuous view switching, the number of views (including real views and intermediate views supplemented by algorithms) may be very large (perhaps every other A viewing angle of 3 degrees or 5 degrees means a total of 120 or 72 viewing angles, etc.). At this time, the compression and transmission of this multi-view video also need to be considered. To address this problem, the embodiments of this application also provide a corresponding solution, which will be introduced in detail later.

From a system architecture perspective, referring to Figure 1, embodiments of this application may involve a collection end, a server end, and a client end. Among them, on the collection side, multiple camera devices can be used to shoot the process of model tasks and catwalks, and obtain video content from multiple real perspectives. On the server side, processing such as portrait cutout, intermediate perspective image synthesis, and multi-view video compression can be performed. The compressed multi-view video can be transmitted to the client, and can also be sliced during transmission to shorten the client's waiting delay. On the client side, you can download the multi-view video, render a 3D spatial scene model, render a "digital human" (character image video content from a certain default perspective) based on the multi-view video, and then add the "digital human" to In the 3D space scene model, during this process, some processing such as light and shadow fusion can also be performed, for example, including achieving shadow consistency under different viewing angles, etc., to improve the display effect. During the display process, it can respond to the user's perspective switching operation and simulate the effect of continuous perspective switching by switching to video content from other adjacent perspectives. Additionally, you can respond to zoom operations performed by the user, and more.

The specific implementation solutions provided by the embodiments of this application are introduced in detail below.

Embodiment 1

First, from the perspective of the server, Embodiment 1 provides a method for displaying information based on dynamic digital human images. See Figure 2. The method may specifically include:

S201: Obtain video content from multiple real perspectives. The video content from multiple real perspectives is to synchronize the process of displaying the target clothing by real people wearing the target clothing from different perspectives through multiple camera devices. Photographed.

Among them, the so-called real perspective is the perspective from which camera equipment is actually deployed for shooting. When implemented specifically, for the "model show" function, multiple camera equipment can be deployed 180/360 degrees around real people (i.e. real model characters) in real space scenes such as studios to ensure that every perspective can be captured Clear shooting, using all camera equipment to capture the characters' movements and postures at the same time while the real model is wearing the specific clothing required to be displayed. In this way, a specific model character can walk in the above-mentioned space scene or perform certain display actions while wearing the specific clothing required for display. During this process, the above-mentioned multiple camera devices can perform synchronous shooting, thereby Get video content from multiple perspectives. In the embodiment of the present application, the number of specific camera devices does not need to be too large, for example, it may be on the order of more than a dozen, and so on.

S202: Extract character images contained in multiple video frames contained in the video content.

After obtaining video content from multiple perspectives, it is necessary to generate a "digital human" to facilitate fusion with multiple different scene models. Therefore, first, the video content contained in the video frames from multiple perspectives can be extracted separately. Character images. Specifically, since each perspective corresponds to a video content, each video content includes multiple video frames, and each video frame can include a character image, and the "cutout" technology can be used to extract the character image from it. In this way, multiple character images can be extracted from each perspective, corresponding to the same character. The "multiple character images" here refer to the character images extracted from the video frames at multiple different time points from each perspective. . Since the video frame has time information, the extracted character images will also have corresponding time point information. These character images can be arranged according to the time point, and the character image sequence can be obtained from each perspective. sequence.

S203: Using image-based rendering technology, based on the pixel offset relationship information between adjacent real-view character images at the same time point, generate characters at corresponding time points for multiple intermediate perspectives between adjacent real-view perspectives. image.

After obtaining the character image sequence corresponding to each perspective, in order to better simulate the effect of continuous perspective switching, interpolation can be used to synthesize character image sequences from more perspectives. Specifically, the perspective synthesis process can be performed between every two adjacent real perspectives. For example, assuming there are 10 real perspectives in total, and perspective 1 and perspective 2 are adjacent, then the perspective between perspective 1 and perspective 2 can be Synthesizing human images from multiple intermediate angles of view, that is, estimating what the captured image of a person would look like assuming a camera device is installed at the intermediate angles of view. In addition, image synthesis can also be performed on intermediate viewing angles between viewing angles 2 and 3, between viewing angles 3 and 4, and other adjacent viewing angles. In this way, images from multiple intermediate viewpoints can be estimated between every two adjacent real viewpoints.

Among them, the selection of the middle viewing angle can be determined according to actual needs. For example, after testing, if a viewing angle is set every 3 degrees, the user will not obviously feel the jump phenomenon when switching between viewing angles. , therefore, you can set the interval between the intermediate viewing angles to 3 degrees; of course, if you want to better simulate continuous viewing angle switching, you can also set the interval between the intermediate viewing angles smaller, or if you want to avoid excessive transmission bit rates. If it is high, you can also set the interval between the middle viewing angles to be larger, and so on.

After determining the interval between the intermediate perspectives, the pixel positions of each intermediate perspective can be estimated based on the pixel positions of the character images corresponding to the two adjacent perspectives and the pixel offset relationship information, and then the specific intermediate perspective can be synthesized. Character images. Among them, since there are multiple person images at each perspective, corresponding to different time points, therefore, when performing intermediate perspective image synthesis, it can also be performed in units of time points, that is, for specific intermediate perspective Generate character images corresponding to multiple time points, so that each intermediate perspective can correspond to a character image sequence.

In specific implementation, there are many ways to estimate the pixel position of the middle perspective based on the character images of two adjacent perspectives. For example, in one way, human images corresponding to adjacent real viewing angles at the same time point can be used as input, and the dense optical flow field between the character images corresponding to the adjacent real viewing angles can be fitted through a deep learning model, and then , this dense optical flow field can be used to estimate the pixel positions of multiple intermediate viewing angles at corresponding time points between the adjacent real viewing angles. Among them, optical flow (optical flow) is the instantaneous speed of pixel movement of a spatially moving object on the observation imaging plane. The optical flow method uses the changes in the time domain of pixels in the image sequence and the correlation between adjacent frames to A method to find the correspondence between the previous frame and the current frame to calculate the motion information of objects between adjacent frames. In space, motion can be described by a motion field, and on an image plane, the motion of an object is often reflected by the different grayscale distributions of different images in the image sequence. Therefore, the motion field in space is transferred to the image and is expressed as light. Flow field. The optical flow field is a two-dimensional vector field, which reflects the changing trend of the grayscale of each point on the image. It can be regarded as the instantaneous velocity field generated by the movement of pixels with grayscale on the image plane. The information it contains is the instantaneous motion velocity vector information of each image point. Among them, dense optical flow is an image registration method that performs point-by-point matching on an image or a specified area. It calculates the offset of all points on the image to form a dense optical flow field. Through this dense optical flow field, pixel-level image registration can be performed. Therefore, in the embodiment of the present application, this optical flow field information can be estimated first, and then, based on the optical flow field, the pixel position of the character image at the corresponding time point for each intermediate perspective can be estimated, and then the corresponding corresponding time point can be generated for the intermediate perspective. images of people.

S204: Generate multi-view videos based on the multiple real views and the multiple intermediate views of character images at multiple time points, so as to match the multi-view videos to the preset virtual 3D space scene model on the client. Rendering and display are performed in the virtual 3D space scene to provide the content of the dynamic digital human image displaying the target clothing in the virtual 3D space scene, and to provide an interactive effect for simulating continuous perspective switching.

After obtaining a sequence of person images from multiple intermediate perspectives, the sequences composed of multiple real perspectives and the multiple intermediate perspective person images at multiple time points can be organized into video formats, thereby obtaining a multi-view video . That is to say, in this embodiment of the present application, multi-view videos may include videos corresponding to multiple real views and videos corresponding to multiple intermediate views. This kind of multi-view video can be used to express a "digital person". This "digital person" is generated by shooting, cutting out, and intermediate perspective synthesis of real people, rather than by 3D modeling. , Therefore, while maintaining the 3D effect, it can also make the "digital people" look more real and reduce the cartoon feel, and can also achieve a more realistic and natural display effect for clothing and other products.

After the multi-view video is obtained through the above method, the multi-view video can also be compressed and encoded to provide it to the client for display. When displayed on the client, the specific multi-view video can be decoded. Since the cutout and other processing have been carried out before, this multi-view video can have characteristics such as a transparent background. Therefore, it can be placed on the pre-generated 3D in the spatial scene model, thereby presenting the visual effect of a specific "digital person" located in the 3D spatial scene model.

Specifically, when performing compression encoding processing, since multi-view videos are involved, special processing may also be performed on the compression encoding method in the embodiments of the present application. This is because, compared with ordinary videos, multi-view videos usually have the following characteristics: In terms of resolution: multi-view videos may include video data corresponding to dozens or even hundreds of views. If each view corresponds to high-definition video, then the overall video data will be huge. Assuming there are 120 viewing angles, the overall resolution will exceed 32K or even higher, which is a heavy load for most devices; in terms of video bit rate, high resolution means higher video bit rate. This will make real-time transmission and smooth playback more difficult. The bit rate of ordinary 720P video may be 2-5Mbps, but the bit rate of multi-view video will increase dozens or even hundreds of times. In terms of video volume, multi-view videos contain data from multiple perspectives, which causes the size of video files to grow rapidly. An hour of multi-view video may require tens or even hundreds of GB of storage space. These technical challenges make the storage, compression, and transmission of multi-view video particularly difficult.

In the existing technology, there are some solutions for compressing and transmitting multi-view videos. For example:

Method 1 can simply splice multi-view videos. Specifically, images from all views at the same time point can be put together into the same frame, and then compressed and transmitted. However, this will result in the spliced video having too high resolution, making it difficult to decode and play in real time, and difficult to transmit.

The second method is to transmit through streaming media. However, on the one hand, the compression efficiency is not ideal. On the other hand, in order to enable the client to switch the viewing angle, the multi-view video needs to be sliced and processed separately before streaming. When the user needs to switch from perspective A to perspective B at a certain moment, the data of perspective B in the corresponding time slice can be pulled for playback; however, the stream switching delay of slicing is relatively large, and whether smooth perspective switching is still possible Depends on the size of the slice, because you must wait for the previous slice to finish playing before switching to the next slice in the next perspective for playback.

Method three uses a codec designed specifically for multi-view videos for video encoding and decoding. This method provides better compression performance, but it increases the complexity of encoding and decoding and requires higher computing power. In addition, Since MV-HEVC is a relatively new standard and requires a dedicated decoder to complete the decoding, not all devices support this format, especially terminal devices such as ordinary mobile phones or computers usually cannot support it, even if It can be supported, but lags and other phenomena may also occur during the process of opening and playing videos.

In view of the above situation, in the embodiment of the present application, corresponding solutions are also provided for problems such as storage and compression of multi-view videos. Specifically, before encoding a specific multi-view video, the video content corresponding to multiple views can be spliced first, that is, the video content corresponding to different views at the same time point is spliced. However, it is not simply a matter of splicing. The video contents from all perspectives are all spliced into the same frame. Instead, multiple video frames corresponding to multiple perspectives at the same time point can be grouped to obtain multiple sets, and multiple video frames in the same set can be spliced into the same frame. (For the convenience of distinction, the frame obtained after this splicing can be called a "combined frame"), that is to say, the same point in time can correspond to multiple different combined frames. In this way, the resolution of the combined frame can be made Not too high.

Specifically, when grouping multi-view video frames, the number of frames that need to be divided into can be determined based on information such as the number of specific views, the resolution of a single video frame in each view, and the maximum resolution supported by the terminal device. groups, and how many views are included in each group such that the resolution of each combined frame is lower than the maximum resolution that the end device can support. For example, assume there are 72 viewing angles in total, and the resolution of the video frame of each viewing angle is 720P. Currently, most terminal devices on the market can usually support real-time decoding of 4K resolution. At this time, the above 72 viewing angles can be divided into 12 groups. , each combined frame will include video frames corresponding to 6 perspectives at the same point in time. The resolution of each combined frame is 720P×6=4320P, which is close to the resolution of general 4K images. Therefore, most terminal devices Real-time decoding of image frames of this resolution can be achieved.

After determining the specific number of groups, in order to achieve higher compression efficiency during the encoding process, the grouping method of different viewing angles and the arrangement method in the specific combined frame can also be determined. Among them, there can be many specific grouping and arrangement methods. For example, in the simplest way, the 1st to 6th perspectives can be a group, the 7th to 12th perspectives can be a group, etc., in the specific combination The frame can be divided into 3×2 (three rows and two columns) blocks, and the individual views can be arranged in numbered order within these blocks, and so on. However, considering that video frames captured at the same time point from different viewing angles tend to have relatively high similarities in content, especially from adjacent viewing angles, the similarity between the two will be higher. From an information encoding perspective, There will be a large amount of redundant information in the highly similar content between adjacent views, which is content that can be compressed during the encoding process. In other words, the existence of redundant information is conducive to improving the compression efficiency of encoding. Therefore, if this redundant information can be fully utilized during the encoding process, it will be of great help in improving compression efficiency.

In the video encoding process, specific information compression technology can be divided into two types: intra-frame compression and inter-frame compression. Intra-frame compression compresses in the spatial domain (space XY axis). In the compression process, the main reference is to the relationship between the data of this frame. Similarity; while inter-frame compression uses the inter-frame redundancy between different video frames in a video sequence, such as the similarity between previous and following frames, to reduce the amount of data through prediction methods. In general, inter-frame compression usually achieves higher compression ratios relative to intra-frame compression.

However, if the simple view grouping and arrangement method described in the previous example is followed, the video frames of adjacent views with the highest redundancy are in the same combined frame. Therefore, when compressing the combined frame, only This redundant information is used in the intra-frame compression process and cannot be fully utilized in the inter-frame coding process.

To this end, in the embodiment of the present application, a more optimal perspective grouping and arrangement method is also provided. Specifically, video frames from adjacent perspectives can be located at the same position of adjacent combined frames. That is to say, videos from adjacent perspectives can Frames will be grouped into different but adjacent groups, and will be located in the same position of adjacent combined frames.

For example, assume that there are 36 views in total (the number of views has been reduced for ease of description), divided into 6 groups, and each group has 6 video frames from the views, that is, every 6 video frames from the views form a Combined frames. As shown in Figure 3, assume that each combined frame includes 3×2 blocks, each block is used to place a video frame from one perspective, and the position number of each block is 0, 1, 2, 3, 4, 5 respectively. ; In addition, assume that the 36 viewing angles are represented by A1, A2, A3...A36 respectively. As shown in Figure 3, it can be seen that the viewing angles A1, A2, A3, A4, A5, and A6 are located at position 0 of the combined frames 1 to 6, and the viewing angles A7, A8, A9, A10, A11, and A12 are located at the combined frame 1 to position 1 of 6, and so on. In other words, A1, A7, A13, A19, A25, and A31 are the first group and are spliced into group frame 1; A2, A8, A14, A20, A26, and A32 are the second group, and are spliced into combined frame 2. , and so on. It can be seen that in each combined frame, the numbers of the viewing angles form an arithmetic sequence, and the difference between the viewing angle numbers is the number of groups, which is 6 in this example. In this way, the viewing angles at the same position between adjacent combined frames can be adjacent, which makes the image content at the same position between adjacent groups of frames have a high degree of similarity. , when performing inter-frame compression coding, the redundant information generated due to the high content similarity between adjacent views can be fully utilized, which is conducive to obtaining a higher compression rate.

Of course, the above example only shows the splicing of video frames from each perspective at one point in time. Video frames from each perspective at other time points can also be grouped and arranged in the above manner. In this way, 6 combined frames can be spliced at each time point. After each time point is spliced in the above manner, the resulting combined frame can be formed into a frame sequence. For example, if each combined frame is represented as "combined frame mn", where m represents the number of the time point and n represents the same time point. Corresponding numbers of each combined frame, the frame sequence formed can be: (combined frame 11, combined frame 12, combined frame 13, combined frame 14, combined frame 15, combined frame 16, combined frame 21, combined frame 22, combined Frame 23, combined frame 24, combined frame 25, combined frame 26, combined frame 31, combined frame 32...).

Through the above grouping and arrangement, the video frames of adjacent viewing angles are dispersed into different but adjacent combined frames, and are located at the same position of adjacent frames, and there are usually many gaps between video frames of adjacent viewing angles. High similarity, therefore, can make adjacent combined frames, at least between video frames at each position, have a high degree of similarity, that is, there is a large amount of redundant information, which is Objects that can be compressed and optimized during inter-frame encoding. Therefore, when encoding combined frames, the redundant information between video frames from adjacent perspectives can be fully utilized through inter-frame coding to obtain higher compression efficiency. Moreover, since this high compression rate is achieved through inter-frame coding technology, and general-purpose video encoders have the ability to inter-frame coding, encoding can be achieved using a general-purpose video encoder without relying on a dedicated According to the multi-view encoding encoder, a universal video decoder can be used for decoding on the playback end. Therefore, it further supports decoding and playback in more terminal devices.

That is to say, through the embodiments of the present application, the method of grouping and splicing video frames from multiple perspectives is adopted. Compared with the method of directly splicing all perspectives into the same frame, the resolution of each combined frame can be reduced, and the terminal The decoding pressure of the device; in addition, special processing has been carried out on the arrangement of different perspectives in different combined frames, so that the content similarity between different combined frames will be relatively high. In this way, the combined frames can be inter-frame Compression is used to eliminate or reduce redundant information between video frames from adjacent perspectives, thereby obtaining a higher compression rate, thus making it possible for general-purpose encoders (for example, HEVC (High Efficiency Video Coding), etc. ) to complete the encoding process. Correspondingly, a general decoder can also be used for decoding. In this way, specific videos can be decoded and played on most terminal devices, avoiding complex rendering processes, large computing overhead, and the need for dedicated codecs. rely.

After the frame sequence formed by multiple combined frames is encoded and inter-frame compressed, it can be used for transmission to the client for decoding and display on the client. Among them, in an optional implementation mode, the frame sequence can also be sliced before transmission, so that the slices obtained after slicing can be transmitted in units, and the receiving end can perform independent decoding in units of segments. Play. In this way, the receiving end only needs to receive the first segment to decode and play it, without waiting for all frame sequences to be transmitted. Therefore, the waiting delay can be shortened.

Among them, the specific slice duration can be determined according to actual needs. If the slice is smaller, the delay at the receiving end will be smaller. For example, the duration of each slice can be 1S, or it can be 0.5S, etc. Among them, in the embodiment of the present application, since video frames from multiple perspectives are grouped and spliced, after determining the slice duration, the number of combined frames that need to be included in each slice can be based on the playback frame rate of the playback end. Depends. For example, still taking 72 perspectives divided into 12 groups as an example, each time point corresponds to 12 combined frames. In addition, assuming that the playback frame rate of the playback end is 30 frames/S, the fragmentation when the combined frame is fragmented If the duration is 1S, each segment needs to contain 30×72/6=360 combined frames. That is to say, the combined frames included in each clip need to meet the number of frames that the player needs to play within 1S. Specifically, when playing, the player needs to decode the combined frames and select a certain perspective from them. The corresponding video frames are played and played. Moreover, the 30 frames played by the player within 1S are usually 30 video frames from the same perspective, and it is possible to play them from any specific perspective. Therefore, the combined frames are processed When slicing, if each segment is 1S, there need to be 30 video frames from each perspective in the same segment. If the number of views is 72, the number of video frames is 30×72. Since these video frames are grouped and spliced into combined frames, the number of combined frames is 30×72/6=360. Of course, if the above assumptions remain unchanged, if the fragmentation duration is changed to 0.5S, each fragment will include 180 combined frames, and so on.

In addition, if the above-mentioned fragmented transmission is performed, the compression rate can be further improved by controlling the number of key frames and bidirectional reference frames during inter-frame coding. Specifically, the encoder encodes multiple images and produces segments of GOP (Group of Pictures). During playback, the decoder reads the segments of GOP, decodes them, reads the pictures, and then renders and displays them. GOP is a group of continuous pictures, consisting of one I frame and several B/P frames. It is the basic unit of access for video image encoders and decoders. Its arrangement sequence will be repeated until the end of the image. Among them, I frame is an intra-coded frame (also called a key frame), P frame is a forward prediction frame (forward reference frame), and B frame is a bidirectional interpolation frame (bidirectional reference frame). Specifically, I frame is usually a complete picture, while P frame and B frame record changes relative to I frame. Among them, P frame and B frame do not have complete picture data, and P frame only has the same data as the previous frame. Picture difference data, B frame records the difference between this frame and the previous and next frames. Among them, the amount of information required to be recorded in B frames is relatively small, so it usually has a higher compression rate. If a GOP contains fewer I frames and more B frames, the overall compression rate will be higher.

In practical applications, which frames will be encoded as I frames, P frames, B frames, etc. are usually determined by the encoder based on the algorithm. In this embodiment of the present application, in order to further control the compression rate of the video, the The encoder can be intervened to reduce the number of I frames and increase the number of B frames. During specific implementation, the key frame interval in the inter-frame encoding process can be controlled according to the number of combined frames included in each slice, so as to reduce the number of frames encoded into key frames in the same slice. For example, assuming that each slice includes 360 combined frames, the key frame interval can be set to 360 frames or 180 frames, that is, so that only one or two frames in the same slice will be encoded as I frames. In addition, for combined frames other than key frames, you can also increase the number of frames in the same slice that are encoded into bidirectional reference frames by lowering the judgment threshold for bidirectional reference frames. That is to say, for B frames, the encoder usually determines whether the current frame can be encoded as a B frame by calculating the similarity between the current frame and the previous and subsequent frames and comparing it with a certain threshold. In this application In embodiments, the threshold can be lowered, and more frames can be encoded as B frames to improve the compression rate.

What needs to be explained here is that since the decoding of P and B frames depends on the I frame, and the decoding of the B frame depends on the previous frame and the next frame, theoretically, if the number of I frames is small, B The larger the number of frames, the better the compression rate will be, but the image quality may be affected during decoding. However, in the embodiment of the present application, since multi-view video frames are grouped and spliced, and video frames from adjacent views are located at the same position of adjacent combined frames, the gap between each two adjacent combined frames is All have the characteristics of relatively high similarity. Under this premise, even if the number of I frames and B frames is controlled through the above method, the image quality at the decoder will usually not be affected. After testing, compared with the solution provided by the embodiment of the present application for encoding after simple splicing, the resolution and code rate have been significantly reduced. PSNR (PeakSignal-to-Noise Ratio, peak signal-to-noise ratio, represents the maximum possible signal The ratio of the power to the destructive noise power that affects its representation accuracy, which is one of the indicators to measure image quality) has been improved instead, as shown in Table 1:

Table 1

Of course, in actual applications, if higher picture quality is required, the number of I frames can be appropriately increased and the number of B frames can be reduced. For example, each slice can include 2 or more I frames, etc. wait.

The compression and transmission of multi-view videos are introduced in detail above. In practical applications, assuming that a user initiates access to the "Model Show" through the client, the specific multi-view video can be transmitted to the client, and the client can download it and use a general video decoder to decode it. . Afterwards, the video frames from one of the perspectives (usually one of the perspectives can be set as the default perspective) can be added to the preset 3D space scene model. Among them, the 3D space scene model can be generated in advance through 3D modeling. That is, the client can render the 3D space scene model and multi-view videos respectively, and add the video content corresponding to one of the views to the 3D space. shown in the scene model.

During specific implementation, in order to achieve a better integration effect between the video content and the 3D space scene model, the scene perspective and the character perspective can also be rendered synchronously, and the perspective can also be switched synchronously. In addition, it can also be implemented The fusion of light and shadow between characters and scenes, etc. For example, lighting information can be added to the 3D space scene, and then the character's shadow position, light position, etc. can be calculated based on the lighting information, and shadow information, light information, etc. can be added to the 3D space scene. In this way, the character can be compared with the character. Better integration of scenes.

In short, through the embodiments of the present application, when it is necessary to provide users with functions such as "model show", the process of displaying the target clothing on real people wearing the target clothing can be performed from different perspectives through multiple camera devices. Perform synchronous shooting to obtain video content from multiple real-view angles. Afterwards, the character images contained in the multiple video frames contained in the video content can be extracted respectively, and then image-based rendering technology can be used to generate video content based on adjacent real-view angles. The pixel offset relationship information between the person images at the same time point is used to generate person images at corresponding time points for multiple intermediate viewing angles between the adjacent real viewing angles. Then, a multi-view video can be generated based on the multiple real viewing angles and the multiple intermediate viewing angles of character images at multiple time points, so that the character image video content from one of the viewing angles can be added to the preset virtual Rendering and display are performed in the 3D space scene model to provide the content of the dynamic digital person displaying the target clothing in the virtual 3D space scene, and to provide an interactive effect for simulating continuous perspective switching. In this way, there is no need to explicitly model the model character, and only the images taken from multiple perspectives can be used to achieve a 3D-like real-life digital human effect, avoiding complex modeling processes and high modeling costs, thus enabling Get low-cost, high-fidelity, dynamic real-time interactive digital effects. Moreover, since the format of the real-life digital human assets created in the embodiment of the present application is ordinary HEVC and other format videos, it can be parsed by most mobile devices and avoids complex rendering processes and large computing overhead.

In addition, for multiple video contents corresponding to multiple viewing angles, specifically during video encoding, multiple video frames corresponding to the multiple viewing angles can be spliced in units of time points to obtain a video frame formed by multiple combined frames. frame sequence, and divide the multiple video frames corresponding to the multiple perspectives at that time point into multiple sets, and splice the multiple video frames in each set into a combined frame, so that the video frames from adjacent perspectives are located in the same position. The same position of adjacent combined frames. After that, a general video encoder can be used to encode the frame sequence formed by the multiple combined frames, and by performing inter-frame compression processing on the multiple combined frames, the redundancy between video frames of adjacent perspectives can be eliminated or reduced. remaining information. In this way, since the video frames from multiple perspectives are grouped and spliced, the resolution of the spliced combined frame is not too high, which facilitates real-time decoding in most terminal devices; in addition, since the grouped splicing is performed, , the grouping method and arrangement method are controlled so that video frames from adjacent perspectives are located at the same position of adjacent combined frames, that is, video frames from adjacent perspectives are located in different but adjacent combined frames, and in The positions in different combined frames are the same, and video frames from adjacent perspectives have relatively high similarities. Therefore, adjacent combined frames spliced in this way have relatively high similarities, and then through universal The inter-frame compression algorithm can obtain a higher compression rate by eliminating or reducing redundant information between video frames from adjacent perspectives. In other words, in the embodiment of the present application, the ideal compression rate can be obtained through a universal video encoder. Correspondingly, the decoding can be completed by using a universal decoder at the decoding end, so that it can be supported on more terminal devices. .

Embodiment 2

The second embodiment corresponds to the first embodiment. From the perspective of the client, a method for displaying information based on a dynamic digital human image is provided. See Figure 4. The method may include:

S401: In response to a viewing request initiated by the user, obtain a multi-view video. The multi-view video is generated by displaying the target clothing on a real person wearing the target clothing from different perspectives through multiple camera devices. The process of synchronizing shooting to obtain video content from multiple real perspectives, extracting the character images contained in multiple video frames contained in the video content from multiple real perspectives, and using image-based rendering technology to create adjacent images. Generate character images from multiple intermediate perspectives between real perspectives, and generate the multi-view video based on the multiple real perspective and character images at multiple time points from the multiple intermediate perspectives;

S402: Decode the multi-view video;

S403: Match the multi-view video to a preset virtual 3D space scene model to provide content of a dynamic digital human image displaying the target clothing in the virtual 3D space scene;

S404: In response to the interactive operation of continuous perspective switching, provide an interactive effect simulating continuous perspective switching by switching to character image video content from other perspectives.

Embodiment 3

The above Embodiments 1 and 2 mainly take the need to provide users with functions such as "model show" in the product information service system as an example, and introduce the specific implementation plan. In practical applications, the method of producing dynamic real-life digital humans provided by the embodiments of the present application can also be used in other application scenarios. To this end, the third embodiment also provides a method for generating a dynamic digital human image. See Figure 5. The method may include:

S501: Obtain video content from multiple real viewing angles. The video content from multiple real viewing angles is obtained by synchronously shooting the process of real people performing target actions from different viewing angles through multiple camera devices.

The specific target action can be determined according to the needs of the actual scene, for example, it can be to perform a certain dance move, etc.

S502: Extract character images contained in multiple video frames contained in the video content.

S503: Using image-based rendering technology, based on the pixel offset relationship information between adjacent real-view character images at the same time point, generate characters at corresponding time points for multiple intermediate perspectives between adjacent real-view perspectives. image.

S504: Generate a multi-view video based on the multiple real view angles and the multiple intermediate view character images at multiple time points, so that the corresponding dynamic digital human image can be displayed through the multi-perspective video.

The data related to the dynamic digital human image produced through the above method can exist in the form of multi-view video. Multi-view video is composed of files in ordinary video format corresponding to multiple viewing angles. Therefore, this kind of real-life digital human has terminal It has side-friendly features and can be displayed in a variety of scenarios, including being displayed after being integrated with a certain 3D space scene model, thereby showing the perspective effect of a specific digital person making corresponding actions in a specific 3D space scene, and , users can perform continuous viewing angle switching or viewing angle zooming, etc.

Embodiment 4

In this fourth embodiment, the method of matching multi-view videos into virtual 3D space scene models is mainly protected. Specifically, this fourth embodiment provides a method for displaying clothing information based on dynamic digital human images, see Figure 6. The method may include:

S601: In response to a request to display the target clothing through a dynamic digital human, obtain a virtual 3D space scene model and a dynamic digital human image expressed in the form of a multi-view video, where the multi-view video is used to view the target clothing from multiple perspectives. Demonstrate the process of displaying the target clothing by the target person while wearing the target clothing;

S602: Match the multi-view video to the virtual 3D space scene model to provide content for the dynamic digital human image to display the target clothing in the virtual 3D space scene, and provide content for simulating continuous viewing angles Switching interactive effects.

In this fourth embodiment, the specific multi-view video can be generated by the method provided in the previous embodiment, or it can also be generated by other methods. For example, the multi-view video can be generated directly by densely deploying more cameras, etc. wait. Among them, when matching multi-view videos to virtual 3D space scene models, it can be achieved through a 3D rendering engine. Of course, during specific implementation, some function customization can be performed on the basis of a general 3D rendering engine, for example, Achieve shadow consistency across multiple different viewing angles, and more.

For the parts not described in detail in Embodiments 2 to 4, please refer to the records in Embodiment 1 and other parts of this specification, and will not be described again here.

It should be noted that the embodiments of this application may involve the use of user data. In actual applications, this can be done in compliance with the applicable laws and regulations of the country (for example, the user explicitly agrees, and the user is effectively notified, etc.), use user-specific personal data in the scenarios described herein to the extent permitted by applicable laws and regulations.

Corresponding to Embodiment 1, this embodiment of the present application also provides a device for displaying information based on a dynamic digital human image. The device may include:

A video content acquisition unit is used to obtain video content from multiple real perspectives. The video content from multiple real perspectives is performed on real people wearing the target clothing from different perspectives through multiple camera devices. It is obtained by synchronously shooting the display process;

A character image extraction unit, configured to extract character images contained in multiple video frames contained in the video content;

A perspective synthesis unit, configured to use image-based rendering technology to generate correspondences for multiple intermediate perspectives between adjacent true perspectives based on pixel offset relationship information between adjacent real perspective images of people at the same time point. Images of people at points in time;

A multi-view video storage and unit is configured to generate a multi-view video based on the multiple real views and the multiple intermediate view character images at multiple time points, so as to match the multi-view video to the predetermined The virtual 3D space scene model is placed in the virtual 3D space scene model to provide the content of the dynamic digital human image displaying the target clothing in the virtual 3D space scene, and to provide an interactive effect for simulating continuous perspective switching.

Wherein, the video content of multiple real perspectives is obtained by synchronously shooting the process of real people walking in the target space wearing the target clothing from different perspectives through multiple camera devices to display the target clothing. of.

Specifically, the perspective synthesis unit can be used for:

Using image-based rendering technology, based on the pixel offset relationship information between adjacent real-view person images at the same time point, the pixel positions of multiple intermediate views between the adjacent real-view views at the corresponding time points are processed. Estimation to generate images of people at multiple time points from the intermediate perspective.

More specifically, the perspective synthesis unit can be used for:

Taking the character images corresponding to adjacent real-view angles at the same time point as input, a deep learning model is used to fit the dense optical flow field between the adjacent real-perspective character images, and the dense optical flow field is used to The pixel positions of the human image at corresponding time points from multiple intermediate viewing angles between the adjacent real viewing angles are estimated.

Additionally, the device may include:

A video compression unit, configured to compress the multi-view video for transmission to the client.

Wherein, the video compression unit may specifically include:

The splicing processing subunit is used to splice video frames corresponding to multiple perspectives in units of time points to obtain a frame sequence formed by multiple combined frames; wherein, for the same time point, multiple perspectives at that time point are The corresponding multiple video frames are divided into multiple sets, and the multiple video frames in each set are spliced into a combined frame, so that video frames from adjacent perspectives are located at the same position of adjacent combined frames;

The inter-frame compression subunit is used to encode the frame sequence formed by the multiple combined frames using a general video encoder, and perform inter-frame compression processing on the multiple combined frames.

Among them, the resolution of each combined frame is lower than the maximum resolution supported by the end device.

Additionally, the device may include:

The slicing processing unit is used to perform encoding and inter-frame compression processing on the frame sequence formed by multiple combined frames, and also perform slicing processing on the frame sequence, so that the fragments obtained after slicing can be transmitted in units. At the receiving end, Fragments are decoded and played independently in units.

Furthermore, it can also include:

A key frame number control unit is used to control the key frame interval in the inter-frame encoding process according to the number of combined frames included in each slice, so as to reduce the number of frames encoded into key frames in the same slice.

The bidirectional reference frame number control unit is used to increase the number of frames encoded into bidirectional reference frames in the same slice by lowering the judgment threshold for bidirectional reference frames for combined frames other than key frames.

Corresponding to Embodiment 2, this embodiment of the present application also provides a device for displaying information based on a dynamic digital human image. The device may include:

A multi-view video acquisition unit is configured to obtain a multi-view video in response to a viewing request initiated by a user. The multi-view video is generated in the following manner: using multiple camera devices to view real people wearing target clothing from different perspectives. The process of displaying the target clothing is synchronously shot to obtain video content from multiple real perspectives, and the character images contained in the multiple video frames contained in the video content from the multiple real perspectives are extracted respectively, and image-based Rendering technology to generate character images for multiple intermediate perspectives between adjacent real perspectives, and generate the multi-view video based on the multiple real perspective and the character images of the multiple intermediate perspectives at multiple time points;

A decoding unit, used to decode the multi-view video;

Adding a unit for matching the multi-view video to a preset virtual 3D space scene model to provide content in which a dynamic digital human image displays the target clothing in the virtual 3D space scene;

The perspective switching interaction unit is used to respond to the interactive operation of continuous perspective switching and provide an interactive effect that simulates continuous perspective switching by switching to character image video content from other perspectives.

Corresponding to Embodiment 3, this embodiment of the present application also provides a device for generating a dynamic digital human image. The device may include:

A video content acquisition unit is used to obtain video content from multiple real viewing angles. The video content from multiple real viewing angles is obtained by synchronously shooting the process of real people performing target actions from different viewing angles through multiple camera devices;

A character image extraction unit, configured to extract character images contained in multiple video frames contained in the video content;

A perspective synthesis unit, configured to use image-based rendering technology to generate correspondences for multiple intermediate perspectives between adjacent true perspectives based on pixel offset relationship information between adjacent true perspective person images at the same time point. Images of people at points in time;

A dynamic digital human asset generation unit, configured to generate multi-view videos based on the multiple real views and the multiple intermediate view character images at multiple time points, so as to generate corresponding dynamic digital images through the multi-view videos. Display of human image.

Corresponding to Embodiment 4, this embodiment of the present application also provides a device for displaying clothing information based on dynamic digital human images. The device may include:

A request receiving unit, configured to obtain a virtual 3D space scene model and a dynamic digital human image expressed in the form of a multi-view video in response to a request to display the target clothing through a dynamic digital human, wherein the multi-view video is used for Demonstrate the process of displaying the target clothing by the target person while wearing the target clothing from multiple perspectives;

A display unit configured to match the multi-view video to the virtual 3D space scene model to provide content for displaying the target clothing by a dynamic digital human image in the virtual 3D space scene, and to provide content for Simulate the interactive effect of continuous perspective switching.

In addition, embodiments of the present application also provide a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the steps of the method described in any one of the foregoing method embodiments are implemented.

and an electronic device including:

one or more processors; and

A memory associated with the one or more processors, the memory being used to store program instructions that, when read and executed by the one or more processors, perform any one of the foregoing method embodiments steps of the method.

Among them, FIG. 7 exemplarily shows the architecture of the electronic device, which may specifically include a processor 710, a video display adapter 711, a disk drive 712, an input/output interface 713, a network interface 714, and a memory 720. The above-mentioned processor 710, video display adapter 711, disk drive 712, input/output interface 713, network interface 714, and the memory 720 can be connected through a communication bus 730.

Among them, the processor 710 can be implemented by a general CPU (Central Processing Unit, processor), a microprocessor, an application specific integrated circuit (Application Specific Integrated Circuit, ASIC), or one or more integrated circuits, for executing Related procedures to implement the technical solutions provided in this application.

The memory 720 can be implemented in the form of ROM (Read Only Memory), RAM (Random Access Memory), static storage device, dynamic storage device, etc. The memory 720 may store an operating system 721 for controlling the operation of the electronic device 700 and a basic input output system (BIOS) for controlling low-level operations of the electronic device 700 . In addition, a web browser 723, a data storage management system 724, an information display processing system 725, etc. can also be stored. The above-mentioned information display processing system 725 can be an application program that specifically implements the above-mentioned steps in the embodiment of the present application. In short, when the technical solution provided in this application is implemented through software or firmware, the relevant program code is stored in the memory 720 and called and executed by the processor 710 .

The input/output interface 713 is used to connect the input/output module to realize information input and output. The input/output/module can be configured in the device as a component (not shown in the figure), or can be externally connected to the device to provide corresponding functions. Input devices can include keyboards, mice, touch screens, microphones, various sensors, etc., and output devices can include monitors, speakers, vibrators, indicator lights, etc.

The network interface 714 is used to connect a communication module (not shown in the figure) to realize communication interaction between this device and other devices. The communication module can communicate through wired means (such as USB, network cable, etc.) or wirelessly (such as mobile network, WIFI, Bluetooth, etc.).

Bus 730 includes a path that carries information between various components of the device (eg, processor 710, video display adapter 711, disk drive 712, input/output interface 713, network interface 714, and memory 720).

It should be noted that although the above device only shows the processor 710, the video display adapter 711, the disk drive 712, the input/output interface 713, the network interface 714, the memory 720, the bus 730, etc., during the specific implementation process, the Equipment may also include other components necessary for proper operation. In addition, those skilled in the art can understand that the above-mentioned device may also include only the components necessary to implement the solution of the present application, and does not necessarily include all the components shown in the drawings.

From the above description of the embodiments, those skilled in the art can clearly understand that the present application can be implemented by means of software plus the necessary general hardware platform. Based on this understanding, the technical solution of the present application can be embodied in the form of a software product in essence or that contributes to the existing technology. The computer software product can be stored in a storage medium, such as ROM/RAM, disk , optical disk, etc., including a number of instructions to cause a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in various embodiments or certain parts of the embodiments of this application.

Each embodiment in this specification is described in a progressive manner. The same and similar parts between the various embodiments can be referred to each other. Each embodiment focuses on its differences from other embodiments. In particular, for the system or system embodiment, since it is basically similar to the method embodiment, the description is relatively simple. For relevant details, please refer to the partial description of the method embodiment. The system and system embodiments described above are only illustrative, in which the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, It can be located in one place, or it can be distributed over multiple network elements. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment. Persons of ordinary skill in the art can understand and implement the method without any creative effort.

The method and electronic equipment for information display based on dynamic digital human images provided by this application have been introduced in detail above. This article uses specific examples to illustrate the principles and implementation methods of this application. The description of the above embodiments is only for use. To help understand the method and its core idea of the present application; at the same time, for those of ordinary skill in the field, there will be changes in the specific implementation and application scope based on the ideas of the present application. In summary, the contents of this specification should not be construed as limiting this application.

Claims (13)

1. A method for information presentation based on dynamic digital human figures, comprising:

obtaining video contents of a plurality of real visual angles, wherein the video contents of the plurality of real visual angles are obtained by synchronously shooting a process of showing a real person on a target garment in a state of wearing the target garment from different visual angles through a plurality of camera devices;

extracting character images contained in the video frames from a plurality of video frames contained in the video content respectively;

generating character images at corresponding time points for a plurality of intermediate view angles between adjacent real view angles according to pixel offset relation information between the character images at the same time point of the adjacent real view angles by using an image-based rendering technology;

Generating multi-view video according to the person images of the real view angles and the intermediate view angles at a plurality of time points, so that the multi-view video is matched into a preset virtual 3D space scene model at a client side, the content of displaying the target clothes in the virtual 3D space scene by dynamic digital person images is provided, and the interactive effect for simulating continuous view angle switching is provided.

2. The method of claim 1, wherein the step of determining the position of the substrate comprises,

the video contents of the multiple real visual angles are obtained by synchronously shooting the process that the real person walks in the target space place in the state of wearing the target clothes from different visual angles through multiple camera devices so as to show the target clothes.

3. The method of claim 1, wherein the step of determining the position of the substrate comprises,

the generating a character image corresponding to a point in time for a plurality of intermediate perspectives between the adjacent real perspectives includes:

estimating pixel positions of a plurality of intermediate view angles between adjacent real view angles at corresponding time points according to pixel offset relation information between the character images of the adjacent real view angles at the same time points by using an image-based rendering technology so as to generate character images of the intermediate view angles at the plurality of time points.

4. The method of claim 3, wherein the step of,

the estimating pixel positions of a plurality of intermediate views between the adjacent real views at corresponding time points includes:

and fitting dense light flow fields between the character images of the adjacent real visual angles through a deep learning model by taking the character images respectively corresponding to the adjacent real visual angles at the same time point as input, and estimating the pixel positions of the character images of the plurality of intermediate visual angles between the adjacent real visual angles at the corresponding time points by utilizing the dense light flow fields.

5. The method as recited in claim 1, further comprising:

and compressing the multi-view video for transmission to a client.

6. The method of claim 5, wherein the step of determining the position of the probe is performed,

the compressing the multi-view video includes:

splicing the video frames corresponding to the multiple visual angles by taking the time point as a unit to obtain a frame sequence formed by multiple combined frames; dividing a plurality of video frames corresponding to a plurality of view angles of the time point into a plurality of sets, splicing the video frames in each set into a combined frame, and enabling the video frames of adjacent view angles to be positioned at the same position of the adjacent combined frames;

And encoding a frame sequence formed by the plurality of combined frames by using a universal video encoder, and carrying out inter-frame compression processing on the plurality of combined frames.

7. The method of claim 6, wherein the step of providing the first layer comprises,

the resolution of each combined frame is lower than the maximum resolution supportable by the terminal device.

8. The method as recited in claim 6, further comprising:

after the frame sequence formed by a plurality of combined frames is encoded and subjected to inter-frame compression, the frame sequence is further subjected to slicing processing so as to be transmitted in units of slices obtained after slicing, and independent decoding and playing are performed in units of slices at a receiving end.

9. A method for information presentation based on dynamic digital human figures, comprising:

responding to a view request initiated by a user, acquiring a multi-view video, wherein the multi-view video is generated by the following steps: synchronously shooting real people from different view angles in the process of displaying the target clothes in a state of wearing the target clothes through a plurality of camera devices to obtain video contents of a plurality of real view angles, respectively extracting person images contained in the video frames contained in the video contents of the plurality of real view angles, generating person images for a plurality of intermediate view angles between adjacent real view angles by utilizing an image-based rendering technology, and generating the multi-view video according to the person images of the plurality of real view angles and the plurality of intermediate view angles at a plurality of time points;

Decoding the multiview video;

matching the multi-view video into a preset virtual 3D space scene model to provide contents for displaying the target clothes in the virtual 3D space scene by dynamic digital human images;

in response to the interactive operation of the continuous view angle switching, an interactive effect simulating the continuous view angle switching is provided by switching to the character image video content of the other view angles.

10. A method of generating a dynamic digital human figure, comprising:

obtaining video contents of a plurality of real viewing angles, wherein the video contents of the plurality of real viewing angles are obtained by synchronously shooting a process of executing target actions on real characters from different viewing angles through a plurality of camera devices;

extracting character images contained in the video frames from a plurality of video frames contained in the video content respectively;

generating character images at corresponding time points for a plurality of intermediate view angles between adjacent real view angles according to pixel offset relation information between the character images at the same time point of the adjacent real view angles by using an image-based rendering technology;

and generating multi-view video according to the character images of the real view angles and the intermediate view angles at a plurality of time points, so as to display the corresponding dynamic digital character image through the multi-view video.

11. A method for displaying apparel information based on dynamic digital humanoid images, comprising:

responding to a request for showing target clothes by a dynamic digital person, obtaining a virtual 3D space scene model and a dynamic digital person image expressed in the form of multi-view video, wherein the multi-view video is used for showing the target clothes by the target person from multiple views in a state of wearing the target clothes;

and matching the multi-view video into the virtual 3D space scene model to provide contents for displaying the target clothes in the virtual 3D space scene by dynamic digital human images and provide an interactive effect for simulating continuous view switching.

12. A computer readable storage medium, on which a computer program is stored, characterized in that the program, when being executed by a processor, implements the steps of the method of any of claims 1 to 11.

13. An electronic device, comprising:

one or more processors; and

a memory associated with the one or more processors for storing program instructions that, when read for execution by the one or more processors, perform the steps of the method of any of claims 1 to 11.