CN107613239B - Video communication background display method and device - Google Patents

Abstract

The invention provides a video communication background display method and a video communication background display device, wherein the method comprises the following steps: acquiring a scene image of a current user; acquiring a depth image of a current user; processing the scene image and the depth image to extract a person region of the current user in the scene image to obtain a person region image; locking a preset first virtual background image to generate a second virtual background image, fusing the figure region image and the second virtual background image to obtain a combined image, and displaying the combined image to a target user performing video communication with the current user; determining familiarity between a current user and a target user; and acquiring corresponding image elements from the first virtual background image according to the familiarity, and displaying the image elements to the target user in the second virtual background image. Therefore, with the familiarity between the user and the target user, the virtual background image of the current user is gradually opened and displayed to the target user, the privacy of the user is protected, and the communication safety is realized.

Description

Video communication background display method and device

technical field

The present invention relates to the technical field of image processing, and in particular, to a video communication background display method and device.

Background technique

With the development of Internet technology, more and more communication functions have been developed and applied. Among them, the video communication function has been widely used because it can realize visual communication of remote users.

However, in the related art, when a user conducts a video chat, the current user may set a virtual background in advance according to personal preferences, etc., and these virtual backgrounds reflect the user's personal preferences, and may even reflect the real environment where the current user is located. Displaying the virtual background to the other party cannot effectively protect the privacy information of the current user.

SUMMARY OF THE INVENTION

The present invention provides a video communication background display method and device to solve the technical problem of exposing the user's personal privacy by directly displaying a virtual background image to a target user during a video chat in the prior art.

An embodiment of the present invention provides a video communication background display method for an electronic device, including: acquiring a scene image of a current user; acquiring a depth image of the current user; processing the scene image and the depth image to extract the the character area of the current user in the scene image to obtain a character area image; perform locking processing on the preset first virtual background image to generate a second virtual background image, and combine the character area image with the second virtual background image. The virtual background images are fused to obtain a combined image, which is displayed to the target user performing video communication with the current user; the familiarity between the current user and the target user is determined; Corresponding image elements are acquired from the background image, and the image elements are displayed to the target user in the second virtual background image.

Another embodiment of the present invention provides a video communication background display device for an electronic device, including: a visible light camera for acquiring a scene image of a current user; a depth image acquisition component for acquiring a depth image of the current user; a processor, configured to process the scene image and the depth image to extract the character area of the current user in the scene image to obtain a character area image; perform locking processing on the preset first virtual background image generating a second virtual background image, and merging the character area image with the second virtual background image to obtain a combined image, and displaying it to a target user who is in video communication with the current user; determining that the current user and the Familiarity between target users; acquiring corresponding image elements from the first virtual background image according to the familiarity, and displaying the image elements to the target users in the second virtual background image.

Yet another embodiment of the present invention provides an electronic device comprising: one or more processors; a memory; and one or more programs, wherein the one or more programs are stored in the memory and configured to Executed by the one or more processors, the program includes a method for performing the video communication background display method described in the above embodiments.

Another embodiment of the present invention provides a computer-readable storage medium, comprising a computer program used in conjunction with an electronic device capable of imaging, and the computer program can be executed by a processor to implement the video communication background display method described in the foregoing embodiment. .

The technical solutions provided by the embodiments of the present invention may include the following beneficial effects:

Obtain the scene image of the current user, obtain the depth image of the current user, process the scene image and the depth image to extract the character area of the current user in the scene image to obtain the character area image, and lock the preset first virtual background image The second virtual background image is generated by processing, and the image of the character area is fused with the second virtual background image to obtain a combined image, which is displayed to the target user performing video communication with the current user, and the familiarity between the current user and the target user is determined. The familiarity obtains corresponding image elements from the first virtual background image, and displays the image elements to the target user in the second virtual background image. Therefore, during video communication, as the user and the target user become familiar with each other, the virtual background image of the current user is gradually opened and displayed to the target user, which protects the user's privacy and realizes communication security.

Additional aspects and advantages of the present invention will be set forth, in part, from the following description, and in part will be apparent from the following description, or may be learned by practice of the invention.

Description of drawings

The above and/or additional aspects and advantages of the present invention will become apparent and readily understood from the following description of embodiments taken in conjunction with the accompanying drawings, wherein:

1 is a schematic flowchart of a video communication background display method according to some embodiments of the present invention;

2 is a schematic block diagram of a video communication background display device according to some embodiments of the present invention;

3 is a schematic structural diagram of an electronic device according to some embodiments of the present invention;

4 is a schematic flowchart of a video communication background display method according to some embodiments of the present invention;

5 is a schematic flowchart of a video communication background display method according to some embodiments of the present invention;

6(a) to 6(e) are schematic diagrams of scenes of structured light measurement according to an embodiment of the present invention;

Fig. 7(a) and Fig. 7(b) are schematic diagrams of scenes of structured light measurement according to an embodiment of the present invention;

8 is a schematic flowchart of a video communication background display method according to some embodiments of the present invention;

9 is a schematic flowchart of a video communication background display method according to some embodiments of the present invention;

10 is a schematic flowchart of a video communication background display method according to some embodiments of the present invention;

11 is a schematic flowchart of a video communication background display method according to some embodiments of the present invention;

12 is a schematic flowchart of a video communication background display method according to some embodiments of the present invention;

13 is a schematic block diagram of an electronic device according to some embodiments of the present invention; and

14 is a schematic block diagram of an electronic device according to some embodiments of the present invention.

Detailed ways

The following describes in detail the embodiments of the present invention, examples of which are illustrated in the accompanying drawings, wherein the same or similar reference numerals refer to the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary, and are intended to explain the present invention and should not be construed as limiting the present invention.

The method and apparatus for displaying a video communication background according to embodiments of the present invention are described below with reference to the accompanying drawings.

FIG. 1 is a flowchart of a method for displaying a video communication background according to an embodiment of the present invention. As shown in FIG. 1 , the method includes:

Step 101: Acquire a scene image of the current user.

Step 102, acquiring a depth image of the current user.

Step 103: Process the scene image and the depth image to extract the character area of the current user in the scene image to obtain the character area image.

2 and 3 , the video communication background display method according to the embodiment of the present invention can be implemented by the video communication background display apparatus 100 according to the embodiment of the present invention. The video communication background display 100 of the embodiment of the present invention is used in the electronic device 1000 . As shown in FIG. 3 , the video communication background display device 100 includes a visible light camera 11 , a depth image acquisition component 12 and a processor 20 . Step 101 may be implemented by the visible light camera 11 , step 102 may be implemented by the depth image acquisition component 12 , and steps 103 and 104 may be implemented by the processor 20 .

That is to say, the visible light camera 11 can be used to obtain the scene image of the current user; the depth image acquisition component 12 can be used to obtain the depth image of the current user; the processor 20 can be used to process the scene image and the depth image to extract the current user in the scene image. The person area image is obtained by the person area image, and the person area image is fused with the second virtual background image to obtain a combined image.

The scene image may be a grayscale image or a color image, and the depth image representation includes depth information of each person or object in the scene of the current user. The scene range of the scene image is consistent with the scene range of the depth image, and each pixel in the scene image can find the depth information corresponding to the pixel in the depth image.

The video communication background display device 100 of the embodiment of the present invention can be applied to the electronic device 1000 of the embodiment of the present invention. That is to say, the electronic device 1000 of the embodiment of the present invention includes the video communication background display device 100 of the embodiment of the present invention.

In some embodiments, the electronic device 1000 includes a mobile phone, a tablet computer, a laptop computer, a smart bracelet, a smart watch, a smart helmet, smart glasses, and the like.

The existing methods for segmenting people and backgrounds are mainly based on the similarity and discontinuity of adjacent pixels in terms of pixel values. However, this segmentation method is easily affected by environmental factors such as external lighting. The video communication background display method according to the embodiment of the present invention The video communication background display device 100 and the electronic device 1000 extract the character area in the scene image by acquiring the depth image of the current user. Since the acquisition of the depth image is not easily affected by factors such as illumination and color distribution in the scene, the character area extracted from the depth image is more accurate, especially the boundary of the character area can be accurately demarcated. Further, the effect of the merged image obtained by merging the more accurate person region image and the second virtual background is better.

Referring to FIG. 4, as a possible implementation manner, the step of acquiring the depth image of the current user in the above step 102 includes:

Step 201, projecting structured light to the current user.

Step 202, capturing a structured light image modulated by the current user.

Step 203: Demodulate phase information corresponding to each pixel of the structured light image to obtain a depth image.

In this example, continuing to refer to FIG. 3 , the depth image acquisition assembly 12 includes a structured light projector 121 and a structured light camera 122 . Step 201 may be implemented by the structured light projector 121 , and steps 202 and 203 may be implemented by the structured light camera 122 .

That is to say, the structured light projector 121 can be used to transmit structured light to the current user; the structured light camera 122 can be used to capture the structured light image modulated by the current user, and demodulate the phase information corresponding to each pixel of the structured light image to obtain depth image.

Specifically, after the structured light projector 121 projects a certain pattern of structured light onto the face and body of the current user, a structured light image modulated by the current user is formed on the surface of the current user's face and body. The structured light camera 122 captures the modulated structured light image, and then demodulates the structured light image to obtain a depth image. Among them, the mode of structured light can be laser stripes, Gray codes, sinusoidal stripes, non-uniform speckle, and the like.

Referring to FIG. 5, in some embodiments, the step of demodulating phase information corresponding to each pixel of the structured light image in step 203 to obtain a depth image includes:

Step 301: Demodulate phase information corresponding to each pixel in the structured light image.

Step 302, converting the phase information into depth information.

Step 303, generating a depth image according to the depth information.

Please continue to refer to FIG. 2 , in some embodiments, step 301 , step 302 and step 303 may all be implemented by the structured light camera 122 .

That is to say, the structured light camera 122 can be further used to demodulate the phase information corresponding to each pixel in the structured light image, convert the phase information into depth information, and generate a depth image according to the depth information.

Specifically, compared with the unmodulated structured light, the phase information of the modulated structured light has changed, and the structured light presented in the structured light image is the structured light after distortion is generated, wherein the changed phase information It can represent the depth information of the object. Therefore, the structured light camera 122 first demodulates the phase information corresponding to each pixel in the structured light image, and then calculates the depth information according to the phase information, thereby obtaining the final depth image.

In order to make those skilled in the art more clearly understand the process of collecting the depth image of the current user's face and body according to the structure, the following takes a widely used grating projection technology (fringe projection technology) as an example to illustrate its specific principle. Among them, the grating projection technology belongs to the surface structured light in a broad sense.

As shown in Fig. 6(a), when using surface structured light projection, first generate sinusoidal fringes through computer programming, project the sinusoidal fringes to the object to be measured through the structured light projector 121, and then use the structured light camera 122 to shoot the fringes The degree of curvature after modulation by the object, then demodulate the curved fringes to obtain the phase, and then convert the phase into depth information to obtain the depth image. In order to avoid the problem of errors or error coupling, the depth image acquisition component 12 needs to be calibrated before using structured light for depth information acquisition. The calibration includes geometric parameters (for example, the relative relationship between the structured light camera 122 and the structured light projector 121 ). position parameters, etc.), the internal parameters of the structured light camera 122 and the internal parameters of the structured light projector 121, and the like.

的条纹，然后结构光投射器121将该四幅条纹分时投射到被测物(图6(a)所示的面具)上，结构光摄像头122采集到如图6(b)左边的图，同时要读取如图6(b)右边所示的参考面的条纹。Specifically, in the first step, the computer was programmed to generate sinusoidal fringes. Since the distorted fringes need to be used to obtain the phase in the future, for example, the four-step phase-shift method is used to obtain the phase, so the four phase differences generated here are:

Then the structured light projector 121 time-divisionally projects the four fringes onto the object to be tested (the mask shown in Fig. 6(a) ), and the structured light camera 122 collects the picture on the left of Fig. 6(b), and at the same time To read the fringes of the reference plane as shown on the right in Figure 6(b).

The second step is to perform phase recovery. The structured light camera 122 calculates the modulated phase according to the collected four modulated fringe images (ie, structured light images), and the phase image obtained at this time is a truncated phase image. Because the result obtained by the four-step phase-shifting algorithm is calculated by the arc tangent function, the modulated phase of structured light is limited between [-π, π], that is, whenever the modulated phase exceeds [-π] ,π], it starts again. The final phase principal value is shown in Fig. 6(c).

Among them, in the process of phase recovery, it is necessary to perform de-jump processing, that is, to restore the truncated phase to a continuous phase. As shown in Fig. 6(d), the left side is the modulated continuous phase diagram, and the right side is the reference continuous phase diagram.

The third step is to subtract the modulated continuous phase and the reference continuous phase to obtain a phase difference (ie, phase information), which represents the depth information of the measured object relative to the reference plane, and then substitute the phase difference into the conversion of phase and depth Formula (parameters involved in the formula are calibrated), the three-dimensional model of the object to be measured as shown in Figure 6(e) can be obtained.

It should be understood that, in practical applications, according to different specific application scenarios, the structured light used in the embodiments of the present invention may also be any other pattern in addition to the above-mentioned grating.

As a possible implementation manner, the present invention can also use speckle structured light to collect depth information of the current user.

Specifically, the method for obtaining depth information from speckle structured light is to use a substantially flat diffraction element with a relief diffraction structure with a specific phase distribution and a stepped relief structure with two or more concavities and convexities in cross section. The thickness of the substrate in the diffractive element is approximately 1 micron, the heights of each step are not uniform, and the value range of the height may be 0.7 microns to 0.9 microns. The structure shown in FIG. 7( a ) is the local diffraction structure of the collimating beam splitting element of the present embodiment. Fig. 7(b) is a cross-sectional side view along the section A-A, and the units of the abscissa and the ordinate are both micrometers. The speckle pattern generated by speckle structured light is highly random and changes with distance. Therefore, before using the speckle structured light to obtain the depth information, the speckle pattern in the space needs to be calibrated first. After the calibration is completed, 400 speckle images are saved, and the smaller the calibration interval, the higher the accuracy of the acquired depth information. Then, the structured light projector 121 projects the speckle structured light onto the measured object (ie, the current user), and the height difference of the measured object surface changes the speckle pattern of the speckle structured light projected on the measured object. After the structured light camera 122 captures the speckle pattern (ie, structured light image) projected on the object to be measured, the speckle pattern is then cross-correlated with the 400 speckle images saved after the previous calibration one by one, and then 400 correlation images are obtained. degree image. The position of the measured object in the space will show a peak value on the correlation image, and the depth information of the measured object can be obtained by superimposing the above peaks and performing interpolation operation.

Since the ordinary diffractive element diffracts the light beam to obtain most of the diffracted light, but the light intensity of each diffracted light varies greatly, and the risk of damage to the human eye is also great. Even if the diffracted light is subjected to secondary diffraction, the uniformity of the resulting beam is low. Therefore, the effect of projecting the measured object with the light beam diffracted by the common diffractive element is poor. In this embodiment, a collimating beam splitting element is used, which not only has the function of collimating the non-collimated beam, but also has the function of splitting light, that is, the non-collimated light reflected by the mirror passes through the collimated beam splitting element Multiple collimated beams are emitted from different angles, and the cross-sectional areas of the emitted multiple collimated beams are approximately equal, and the energy fluxes are approximately equal, so that the projection effect of the diffracted scattered light of the beam is better. At the same time, the emitted laser light is dispersed to each beam, which further reduces the risk of harming the human eye. Compared with other structured lights with uniform distribution, the speckle structured light consumes the same amount of light when the same collection effect is achieved. lower power.

Referring to FIG. 8, as a possible implementation manner, step 103 processes the scene image and the depth image to extract the character area of the current user in the scene image to obtain the character area image, including:

Step 401: Identify the face area in the scene image.

Step 402: Acquire depth information corresponding to the face region from the depth image.

Step 403: Determine the depth range of the character area according to the depth information of the face area.

Step 404 , according to the depth range of the character area, determine a character area connected to the face area and falling within the depth range to obtain an image of the character area.

Referring to FIG. 2 again, in some embodiments, step 401 , step 402 , step 403 and step 404 may all be implemented by the processor 20 .

That is to say, the processor 20 may be further configured to identify the face area in the scene image, obtain depth information corresponding to the face area from the depth image, determine the depth range of the character area according to the depth information of the face area, and A person area connected to the face area and falling within the depth range is determined according to the depth range of the person area to obtain a person area image.

Specifically, the trained deep learning model can be used to identify the face region in the scene image, and then the depth information of the face region can be determined according to the corresponding relationship between the scene image and the depth image. Since the face region includes features such as nose, eyes, ears, lips, etc., the depth data corresponding to each feature in the face region in the depth image is different, for example, when the face is facing the depth image acquisition component 12 , in the depth image captured by the depth image acquisition component 12, the depth data corresponding to the nose may be smaller, while the depth data corresponding to the ear may be larger. Therefore, the above-mentioned depth information of the face region may be a value or a value range. Wherein, when the depth information of the face region is a numerical value, the numerical value can be obtained by taking the average value of the depth data of the face region; or, it can be obtained by taking the median value of the depth data of the face region.

Since the character area includes the face area, that is to say, the character area and the face area are within a certain depth range. Therefore, after the processor 20 determines the depth information of the face area, it can be based on the depth information of the face area. The depth range of the person area is set, and then the person area that falls within the depth range and is connected to the face area is extracted according to the depth range of the person area to obtain a person area image.

In this way, the human region image can be extracted from the scene image according to the depth information. Since the acquisition of depth information is not affected by factors such as illumination and color temperature in the environment, the extracted image of the human area is more accurate.

Referring to FIG. 9, in some embodiments, the video communication background display method further includes the following steps:

Step 501: Process the scene image to obtain a full-field edge image of the scene image.

Step 502, correcting the image of the person area according to the full-field edge image.

Referring to FIG. 2 again, in some embodiments, both step 501 and step 502 may be implemented by the processor 20 .

That is to say, the processor 20 can also be used to process the scene image to obtain the full-field edge image of the scene image, and to correct the person area image according to the full-field edge image.

The processor 20 first performs edge extraction on the scene image to obtain a full-field edge image, wherein the edge lines in the full-field edge image include the current user and the edge lines of background objects in the scene where the current user is located. Specifically, the edge of the scene image can be extracted by the Canny operator. The core of the algorithm for edge extraction by the Canny operator mainly includes the following steps: first, convolve the scene image with a 2D Gaussian filter template to eliminate noise; then, use the differential operator to obtain the gradient value of the gray level of each pixel, and Calculate the gradient direction of the grayscale of each pixel according to the gradient value, and the adjacent pixels of the corresponding pixel along the gradient direction can be found through the gradient direction; then, traverse each pixel, if the grayscale value of a pixel is the same as the two before and after the gradient direction. If the gray value of the adjacent pixel is not the largest, then the pixel is considered not to be an edge point. In this way, the pixel points at the edge position in the scene image can be determined, so as to obtain the full-field edge image after edge extraction.

After acquiring the full-field edge image, the processor 20 corrects the character area image according to the full-field edge image. It can be understood that the image of the person area is obtained by merging all the pixels in the scene image that are connected to the face area and fall within the set depth range. Objects within the depth range. Therefore, in order to make the extracted human region image more accurate, the full-field edge map can be used to correct the human region image.

Further, the processor 20 may further perform a secondary correction on the corrected person area image, for example, may perform expansion processing on the corrected person area image to expand the person area image to preserve the edge details of the person area image.

Step 104, performing locking processing on the preset first virtual background image to generate a second virtual background image, and merging the character area image with the second virtual background image to obtain a combined image, which is displayed to the target of video communication with the current user user.

Wherein, as an example, the preset first virtual background image may be generated by modeling based on the real scene information where the user is located, and the first virtual background image in this example reflects the real scene information where the user is located; as another Example: the preset first virtual background image may be constructed by the user according to personal preference, and the first virtual background image in this example reflects the user's preference. Regardless of what elements the first virtual background image is based on, the user's personal privacy may be exposed to a certain extent.

After obtaining the character area image, the processor 20 performs locking processing on the preset first virtual background image to generate a second virtual background image, and fuses the character area image with the second virtual background image to obtain a combined image. In some embodiments, the preset first virtual background image may be randomly selected by the processor 20 or selected by the current user. The merged combined image can be displayed on the display screen of the electronic device 1000 , and can also be printed by a printer connected to the electronic device 1000 .

In one embodiment of the present invention, the current user wishes to hide the first virtual background during the video process with others. In this case, the video communication background display method according to the embodiment of the present invention can be used to display the image of the character area corresponding to the current user with the preset image. The set first virtual background image is locked to generate a second virtual background image, and the person area image is fused with the second virtual background image to obtain a merged image, and the merged image is displayed to the target user. Since the current user is in a video call with the other party, the visible light camera 11 needs to capture the scene image of the current user in real time, and the depth image acquisition component 12 also needs to capture the depth image corresponding to the current user in real time, and the processor 20 timely captures the scene captured in real time. The images and depth images are processed so that the other party can see a smooth video image composed of multiple frames of merged images.

It should be noted that, according to different application scenarios, different implementations may be used to perform locking processing on the preset first virtual background image to generate the second virtual background image. Gaussian blurring is performed on a virtual background image to generate a second virtual background image. As another possible implementation manner, mosaic processing is performed on a preset first virtual background image to generate a second virtual background image.

Step 105: Determine the familiarity between the current user and the target user.

Step 106: Acquire corresponding image elements from the first virtual background image according to the familiarity, and display the image elements to the target user in the second virtual background image.

It is understood that in some application scenarios, the more familiar the user of video communication is, the more likely it is to show the first virtual background image exposing their private information to the other party. At this time, determine the familiarity between the current user and the target user. , and further, if it is determined that the learned familiarity is greater than the preset threshold, the merged image is switched to the scene image of the current user and displayed to the target user.

It should be noted that, according to different application scenarios, the familiarity between the current user and the target user can be determined in a variety of ways:

As a possible implementation manner, as shown in Figure 10, step 105 includes:

Step 601: Detect video interaction information between a current user and a target user according to a preset matching index.

Step 602 , if it is detected that the video interaction information satisfies the preset matching information, query the corresponding relationship between the preset matching information and the familiarity, and determine the familiarity between the current user and the target user.

In this example, it should be understood that the more familiar the users are, the more casual the topics they talk about or the words they use, or the greater the amount of information they talk about, the more familiar the users are. Content keywords of voice information and text information are detected according to a preset matching index, wherein the keyword is calibrated according to a large amount of experimental data, and can be a word that defines the familiarity between users, such as a term containing a relationship between users ( "Mom", "Dude", "Dad"), for example, colloquial words "Go to hell", "The devil believes in you", etc., and/or, the information volume of voice messages and text messages.

Further, it can be understood that the corresponding relationship between matching information and familiarity is preset, for example, the familiarity corresponding to "Mom" is relatively high, the familiarity corresponding to "Professor Li" is relatively low, and the familiarity corresponding to 10 pieces of voice information is relatively low. Therefore, if it is detected that the video interaction information meets the preset matching information, the corresponding relationship between the preset matching information and the familiarity is queried, and the relationship between the current user and the target user is determined. Familiarity.

As another possible implementation manner, as shown in FIG. 11 , step 105 includes:

Step 701: Send verification requests corresponding to different familiarity levels to the target user.

In step 702, verification is performed according to the request response fed back by the target user and preset standard information, and the familiarity between the current user and the target user is determined according to the verification result.

In this example, the user presets verification requests corresponding to different degrees of familiarity, and sets corresponding standard information for each verification request. The standard information set for the request may be "five mouths", the verification request set for low familiarity is "I am a man or a woman", and the standard information set for this request may be "woman", etc. Therefore, during video chat, Send verification requests corresponding to different familiarity levels to target users.

At this time, the target user feeds back the request response according to the verification request. At this time, only the more familiar target users can feed back the standard information corresponding to the verification request corresponding to the higher familiarity, and the relatively unfamiliar target users can only feedback the more familiar ones. The low familiarity corresponds to the standard information corresponding to the verification request. Therefore, verification is performed according to the request response fed back by the target user and the preset standard information, and the familiarity between the current user and the target user is determined according to the verification result.

As another possible implementation manner, as shown in Figure 12, step 105 includes:

In step 801, a user image of a target user is acquired, and facial feature information of the user image is extracted. Step 802: Query a preset image information database according to the facial feature information to obtain the identity information of the target user.

Step 803 , query the corresponding relationship between the preset identity information and the familiarity, and determine the familiarity between the current user and the target user.

In this example, the correspondence between the identity information of other users and the familiarity is established in advance, for example, the correspondence between the identity information of family members and the higher familiarity is established, the correspondence between the identity information of friends and the medium familiarity is established, and the stranger is established. Correspondence between people and lower familiarity, etc.

Further, the user image of the target user is obtained, and the facial feature information of the user image is extracted, and the user image can be extracted by intercepting the facial screenshot of the target user in the video call, and then, the preset image information database is queried according to the facial feature information to obtain the target user's facial feature information. Identity information, query the correspondence between preset identity information and familiarity, and determine the familiarity between the current user and the target user.

Further, the corresponding image elements are obtained from the first virtual background image according to the familiarity, and the image elements are displayed to the target user in the second virtual background image, wherein the image elements include the items, colors in the first virtual background image etc., where the higher the familiarity is, the more image elements are displayed to the target user in the second virtual background image. As a possible implementation, the number of items corresponding to the familiarity is obtained from the first virtual background image. For the corresponding number of image elements, as another possible implementation manner, image elements of corresponding types are obtained from the first virtual background image according to the item type (eg, office type, daily necessities type, etc.) corresponding to the familiarity.

To sum up, the video communication background display method according to the embodiment of the present invention is obtained by acquiring the scene image of the current user, acquiring the depth image of the current user, and processing the scene image and the depth image to extract the character area of the current user in the scene image. Character area image, lock the preset first virtual background image to generate a second virtual background image, and fuse the character area image with the second virtual background image to obtain a combined image, and display it to the video communication with the current user. The target user determines the familiarity between the current user and the target user, acquires corresponding image elements from the first virtual background image according to the familiarity, and displays the image elements to the target user in the second virtual background image. Therefore, during video communication, as the user and the target user become familiar with each other, the virtual background image of the current user is gradually opened and displayed to the target user, which protects the user's privacy and realizes communication security.

Please refer to FIG. 3 and FIG. 13 together. An embodiment of the present invention further provides an electronic device 1000 . The electronic device 1000 includes a video communication background display device 100 . VIDEO COMMUNICATIONS BACKGROUND Display apparatus 100 may be implemented using hardware and/or software. Video Communication Background The display apparatus 100 includes an imaging device 10 and a processor 20 .

The imaging device 10 includes a visible light camera 11 and a depth image acquisition component 12 .

Specifically, the visible light camera 11 includes an image sensor 111 and a lens 112. The visible light camera 11 can be used to capture color information of the current user to obtain a scene image, wherein the image sensor 111 includes a color filter array (such as a Bayer filter array), and the lens 112 can be one or more. When the visible light camera 11 acquires a scene image, each imaging pixel in the image sensor 111 senses the light intensity and wavelength information from the shooting scene to generate a set of raw image data; the image sensor 111 sends the set of raw image data to the processing In the processor 20, the processor 20 obtains a color scene image after performing operations such as denoising and interpolation on the original image data. The processor 20 may process each image pixel in the raw image data individually in a variety of formats, for example, each image pixel may have a bit depth of 8, 10, 12 or 14 bits, and the processor 20 may process the raw image data in the same or different formats. Bit depth is processed for each image pixel.

The depth image acquisition component 12 includes a structured light projector 121 and a structured light camera 122, and the depth image acquisition component 12 can be used to capture depth information of the current user to obtain a depth image. The structured light projector 121 is used for projecting structured light to the current user, wherein the structured light pattern may be a laser stripe, a Gray code, a sinusoidal stripe or a randomly arranged speckle pattern or the like. The structured light camera 122 includes an image sensor 1221 and a lens 1222, and the number of the lenses 1222 may be one or more. The image sensor 1221 is used to capture the structured light image projected by the structured light projector 121 onto the current user. The structured light image can be sent by the depth acquisition component 12 to the processor 20 for processing such as demodulation, phase recovery, phase information calculation, etc. to obtain the depth information of the current user.

In some embodiments, the functions of the visible light camera 11 and the structured light camera 122 can be realized by one camera, that is to say, the imaging device 10 only includes one camera and one structured light projector 121, and the above cameras can not only capture scene images, Structured light images can also be taken.

In addition to using structured light to acquire depth images, depth images of the current user can also be acquired through binocular vision methods and depth image acquisition methods based on time of flight (TOF).

The processor 20 is further configured to fuse the image of the character region extracted from the scene image and the depth image with the second virtual background image, display the merged image to the target user performing video communication with the current user, and determine the relationship between the current user and the target user. When it is determined that the familiarity is greater than the preset threshold, the merged image is switched to the scene image of the current user and displayed to the target user. When extracting the image of the person area, the processor 20 can extract a two-dimensional image of the person area from the scene image in combination with the depth information in the depth image, or can create a three-dimensional image of the person area according to the depth information in the depth image, and then combine the scene The color information in the image fills the three-dimensional character area with color to obtain a three-dimensional color image of the character area. Therefore, when fusing the image of the person area and the second virtual background image, the two-dimensional image of the person area and the second virtual background image may be fused to obtain a combined image, or the three-dimensional color image of the person area and the second virtual background image may be fused. The virtual background images are fused to obtain a merged image.

In addition, the video communication background display apparatus 100 further includes an image memory 30 . The image memory 30 may be embedded in the electronic device 1000, or may be a memory independent of the electronic device 1000, and may include a Direct Memory Access (DMA) feature. The original image data collected by the visible light camera 11 or the related data of the structured light image collected by the depth image collection component 12 can be sent to the image memory 30 for storage or buffering. The processor 20 can read raw image data from the image memory 30 for processing to obtain a scene image, and can also read data related to structured light images from the image memory 30 for processing to obtain a depth image. In addition, the scene image and the depth image can also be stored in the image memory 30 for the processor 20 to call for processing at any time. For example, the processor 20 calls the scene image and the depth image to extract the character area, and then retrieves the obtained character area. The image is fused with the second virtual background image to obtain a merged image. Wherein, the second virtual background image and the merged image may also be stored in the image memory 30 .

VIDEO COMMUNICATIONS BACKGROUND Display device 100 may also include display 50 . The display 50 may obtain the merged image directly from the processor 20 , and may also obtain the merged image from the image memory 30 . The display 50 displays the combined image for viewing by the target user, or further processing is performed by a graphics engine or a graphics processing unit (Graphics Processing Unit, GPU). Video communication background The display device 100 further includes an encoder/decoder 60, the encoder/decoder 60 can encode and decode image data such as scene images, depth images, and merged images, and the encoded image data can be stored in the image memory 30, And may be decompressed by the decoder for display before the image is displayed on the display 50 . The encoder/decoder 60 may be implemented by a central processing unit (Central Processing Unit, CPU), a GPU or a co-processor. In other words, the encoder/decoder 60 may be any one or more of a central processing unit (Central Processing Unit, CPU), a GPU, and a co-processor.

Video Communication Background Display device 100 also includes control logic 40 . When the imaging device 10 is imaging, the processor 20 analyzes the data obtained by the imaging device to determine image statistics of one or more control parameters (eg, exposure time, etc.) of the imaging device 10 . Processor 20 sends the image statistics to control logic 40, which controls imaging device 10 to determine good control parameters for imaging. Control logic 40 may include a processor and/or microcontroller executing one or more routines (eg, firmware). One or more routines may determine control parameters for imaging device 10 based on the received image statistics.

Referring to FIG. 14 , an electronic device 1000 according to an embodiment of the present invention includes one or more processors 200 , a memory 300 and one or more programs 310 . One or more of the programs 310 are stored in the memory 300 and configured to be executed by the one or more processors 200 . Program 310 includes instructions for performing the video communication background display method of any one of the above embodiments.

For example, program 310 includes instructions for performing the video communication background display method described in the following steps:

Step 01: Acquire a scene image of the current user.

Step 02: Obtain a depth image of the current user.

Step 03: Process the scene image and the depth image to extract the character area of the current user in the scene image to obtain the character area image.

Step 04, performing locking processing on the preset first virtual background image to generate a second virtual background image, and merging the character area image with the second virtual background image to obtain a combined image, which is displayed to the target of video communication with the current user. user.

Step 05: Determine the familiarity between the current user and the target user.

Step 06: Acquire corresponding image elements from the first virtual background image according to the familiarity, and display the image elements to the target user in the second virtual background image.

For another example, the program 310 also includes instructions for executing the video communication background display method described in the following steps:

0331: demodulate the phase information corresponding to each pixel in the structured light image;

0332: Convert phase information to depth information; and

0333: Generate a depth image based on the depth information.

The computer-readable storage medium of the embodiment of the present invention includes a computer program used in conjunction with the electronic device 1000 capable of imaging. The computer program can be executed by the processor 200 to implement the video communication background display method of any one of the above embodiments.

For example, a computer program can be executed by the processor 200 to perform the video communication background display method described in the following steps:

Step 01: Acquire a scene image of the current user.

Step 02: Obtain a depth image of the current user.

Step 03: Process the scene image and the depth image to extract the character area of the current user in the scene image to obtain the character area image.

Step 04, performing locking processing on the preset first virtual background image to generate a second virtual background image, and merging the character area image with the second virtual background image to obtain a combined image, which is displayed to the target of video communication with the current user. user.

Step 05: Determine the familiarity between the current user and the target user.

Step 06: Acquire corresponding image elements from the first virtual background image according to the familiarity, and display the image elements to the target user in the second virtual background image.

For another example, the computer program can also be executed by the processor 200 to complete the video communication background display method described in the following steps:

0331: demodulate the phase information corresponding to each pixel in the structured light image;

0332: Convert phase information to depth information; and

0333: Generate a depth image based on the depth information.

In the description of this specification, description with reference to the terms "one embodiment," "some embodiments," "example," "specific example," or "some examples", etc., mean specific features described in connection with the embodiment or example , structure, material or feature is included in at least one embodiment or example of the present invention. In this specification, schematic representations of the above terms are not necessarily directed to the same embodiment or example. Furthermore, the particular features, structures, materials or characteristics described may be combined in any suitable manner in any one or more embodiments or examples. Furthermore, those skilled in the art may combine and combine the different embodiments or examples described in this specification, as well as the features of the different embodiments or examples, without conflicting each other.

In addition, the terms "first" and "second" are only used for descriptive purposes, and should not be construed as indicating or implying relative importance or implying the number of indicated technical features. Thus, a feature delimited with "first", "second" may expressly or implicitly include at least one of that feature. In the description of the present invention, "plurality" means at least two, such as two, three, etc., unless otherwise expressly and specifically defined.

Any process or method description in the flowcharts or otherwise described herein may be understood to represent a module, segment or portion of code comprising one or more executable instructions for implementing custom logical functions or steps of the process , and the scope of the preferred embodiments of the invention includes alternative implementations in which the functions may be performed out of the order shown or discussed, including performing the functions substantially concurrently or in the reverse order depending upon the functions involved, which should It is understood by those skilled in the art to which the embodiments of the present invention belong.

The logic and/or steps represented in flowcharts or otherwise described herein, for example, may be considered an ordered listing of executable instructions for implementing the logical functions, may be embodied in any computer-readable medium, For use with, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a system including a processor, or other system that can fetch instructions from and execute instructions from an instruction execution system, apparatus, or apparatus) or equipment. For the purposes of this specification, a "computer-readable medium" can be any device that can contain, store, communicate, propagate, or transport the program for use by or in conjunction with an instruction execution system, apparatus, or apparatus. More specific examples (non-exhaustive list) of computer readable media include the following: electrical connections with one or more wiring (electronic devices), portable computer disk cartridges (magnetic devices), random access memory (RAM), Read Only Memory (ROM), Erasable Editable Read Only Memory (EPROM or Flash Memory), Fiber Optic Devices, and Portable Compact Disc Read Only Memory (CDROM). In addition, the computer readable medium may even be paper or other suitable medium on which the program may be printed, as the paper or other medium may be optically scanned, for example, followed by editing, interpretation, or other suitable medium as necessary process to obtain the program electronically and then store it in computer memory.

It should be understood that various parts of the present invention may be implemented in hardware, software, firmware or a combination thereof. In the above-described embodiments, various steps or methods may be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware as in another embodiment, it can be implemented by any one of the following techniques known in the art, or a combination thereof: discrete with logic gates for implementing logic functions on data signals Logic circuits, application specific integrated circuits with suitable combinational logic gates, Programmable Gate Arrays (PGA), Field Programmable Gate Arrays (FPGA), etc.

Those skilled in the art can understand that all or part of the steps carried by the methods of the above embodiments can be completed by instructing the relevant hardware through a program, and the program can be stored in a computer-readable storage medium, and the program can be stored in a computer-readable storage medium. When executed, one or a combination of the steps of the method embodiment is included.

In addition, each functional unit in each embodiment of the present invention may be integrated into one processing module, or each unit may exist physically alone, or two or more units may be integrated into one module. The above-mentioned integrated modules can be implemented in the form of hardware, and can also be implemented in the form of software function modules. If the integrated modules are implemented in the form of software functional modules and sold or used as independent products, they may also be stored in a computer-readable storage medium.

The above-mentioned storage medium may be a read-only memory, a magnetic disk or an optical disk, and the like. Although the embodiments of the present invention have been shown and described above, it should be understood that the above-mentioned embodiments are exemplary and should not be construed as limiting the present invention. Embodiments are subject to variations, modifications, substitutions and variations.

Claims (14)

1. A video communication background display method for an electronic device, comprising:

acquiring a scene image of a current user;

acquiring a depth image of the current user;

processing the scene image and the depth image to extract a person region of the current user in the scene image to obtain a person region image;

locking a preset first virtual background image to generate a second virtual background image, fusing the figure region image and the second virtual background image to obtain a combined image, and displaying the combined image to a target user performing video communication with the current user;

determining familiarity between the current user and the target user, the determining familiarity between the current user and the target user comprising: sending verification requests corresponding to different familiarity degrees to the target user; verifying according to the request response fed back by the target user and preset standard information, and determining the familiarity between the current user and the target user according to a verification result;

and acquiring corresponding image elements from the first virtual background image according to the familiarity, and displaying the image elements to the target user in the second virtual background image.

2. The method of claim 1, wherein the obtaining the depth image of the current user comprises:

projecting structured light towards the current user;

shooting a structured light image modulated by the current user; and

and demodulating phase information corresponding to each pixel of the structured light image to obtain the depth image.

3. The method of claim 2, wherein demodulating phase information corresponding to each pixel of the structured-light image to obtain the depth image comprises:

demodulating phase information corresponding to each pixel in the structured light image;

converting the phase information into depth information; and

and generating the depth image according to the depth information.

4. The method of claim 1, wherein the processing the scene image and the depth image to extract a human figure region of the current user in the scene image to obtain a human figure region image comprises:

identifying a face region in the scene image;

acquiring depth information corresponding to the face area from the depth image;

determining the depth range of the character region according to the depth information of the face region; and

and determining a person region which is connected with the face region and falls into the depth range according to the depth range of the person region to obtain the person region image.

5. The method of claim 4, further comprising:

processing the scene image to obtain a full-field edge image of the scene image; and

and correcting the image of the person region according to the full-field edge image.

6. The method according to claim 1, wherein the locking the preset first virtual background image to generate a second virtual background image comprises:

performing Gaussian blur processing on a preset first virtual background image to generate a second virtual background image;

or,

and performing mosaic processing on the preset first virtual background image to generate a second virtual background image.

7. The method of claim 1, wherein the determining familiarity between the current user and the target user further comprises:

detecting video interaction information of the current user and the target user according to a preset matching index;

and if the video interaction information is detected to meet the preset matching information, inquiring the preset corresponding relation between the matching information and the familiarity, and determining the familiarity between the current user and the target user.

8. The method according to claim 7, wherein the detecting video interaction information of the current user and the target user according to a preset matching index comprises:

and detecting content keywords of the voice information and the text information according to a preset matching index, and/or detecting the information content of the voice information and the text information.

9. The method of claim 1, wherein the determining familiarity between the current user and the target user further comprises:

acquiring a user image of the target user, and extracting facial feature information of the user image;

inquiring a preset image information base according to the facial feature information to acquire the identity information of the target user;

and inquiring the preset corresponding relation between the identity information and the familiarity, and determining the familiarity between the current user and the target user.

10. The method of claim 1, wherein the obtaining the corresponding image element from the first virtual background image according to the familiarity comprises:

acquiring a corresponding number of image elements from the first virtual background image according to the number of the articles corresponding to the familiarity; and/or the presence of a gas in the gas,

and acquiring corresponding types of image elements from the first virtual background image according to the types of the articles corresponding to the familiarity.

11. A video communication background display apparatus for an electronic apparatus, comprising:

the visible light camera is used for acquiring a scene image of a current user;

the depth image acquisition component is used for acquiring a depth image of the current user;

a processor, configured to process the scene image and the depth image to extract a person region of the current user in the scene image to obtain a person region image;

locking a preset first virtual background image to generate a second virtual background image, fusing the figure region image and the second virtual background image to obtain a combined image, and displaying the combined image to a target user performing video communication with the current user;

determining familiarity between the current user and the target user, the determining familiarity between the current user and the target user comprising: sending verification requests corresponding to different familiarity degrees to the target user; verifying according to the request response fed back by the target user and preset standard information, and determining the familiarity between the current user and the target user according to a verification result;

and acquiring corresponding image elements from the first virtual background image according to the familiarity, and displaying the image elements to the target user in the second virtual background image.

12. The apparatus of claim 11, wherein the depth image acquisition assembly comprises a structured light projector and a structured light camera, the structured light projector for projecting structured light to the current user;

the structured light camera is configured to:

shooting a structured light image modulated by the current user; and

and demodulating phase information corresponding to each pixel of the structured light image to obtain the depth image.

13. An electronic device, comprising:

one or more processors;

a memory; and

one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the programs comprising instructions for performing the video communication background display method of any of claims 1-10.

14. A computer-readable storage medium comprising a computer program for use in conjunction with an electronic device capable of capturing images, the computer program being executable by a processor to perform the video communication background display method of any of claims 1-10.

Images