CN114897913A - A portrait segmentation method, device, computer equipment and storage medium - Google Patents

Abstract

The invention discloses a portrait segmentation method, a portrait segmentation device, computer equipment and a storage medium. Inputting a picture to be segmented into an EfficientNet network for carrying out multiple dimension scaling treatment, and respectively outputting corresponding dimension reduction pictures; inputting the dimension reduction pictures output for the last two times into an attention fusion module for feature fusion to obtain fusion pictures, wherein the dimension reduction pictures output for the last two times are 7 multiplied by 192 pictures and 7 multiplied by 320 pictures; inputting the fused picture into a strip attention module for carrying out striping processing to obtain an output image; and segmenting the target object in the output image and outputting a target segmentation image. The invention has the advantage of improving the image segmentation accuracy by improving the network structure.

Description

A portrait segmentation method, device, computer equipment and storage medium

technical field

The present invention relates to the technical field of image processing, and in particular, to a portrait segmentation method, device, computer equipment and storage medium.

Background technique

With the maturity of image fusion and digital processing technology, higher requirements are also put forward for the replacement technology of objects such as faces and objects. In order to ensure the fidelity of the characters, the face replacement technology is required to have higher precision and finer texture details.

In response to the above problems, researchers use deep neural networks to segment target objects. For example, the color-based color digital image target segmentation technology mainly uses artificial neural network method to segment the interesting or non-interesting pixels of the face image by using the color space. Subsequently, a deep convolutional neural network is also proposed, that is, the multi-task joint learning algorithm of CNN is used to realize face alignment and segmentation. The algorithm realizes the collaborative fusion of cross-layer features through the designed residual module and improves the effect of segmentation.

However, the above algorithms as a whole rely on low-level visual information and artificial auxiliary information for shallow semantic segmentation, and lack key model training and deep semantic information, resulting in extremely poor robustness of the above algorithms to changes in the image background and unable to complete complex goals. object segmentation task.

SUMMARY OF THE INVENTION

The purpose of the present invention is to provide a portrait segmentation method, device, computer equipment and storage medium, which aims to solve the problem that the existing segmentation algorithms have extremely poor robustness to changes in the image background and cannot complete complex portrait segmentation tasks.

In a first aspect, an embodiment of the present invention provides a method for segmenting a portrait, which includes:

Input the image to be segmented into the EfficientNet network for multiple dimension scaling, and output the corresponding dimension-reduced images respectively;

The last two output dimensionality reduction pictures are input into the attention fusion module for feature fusion to obtain fusion pictures, of which the last two output dimensionality reduction pictures are 7×7×192 pictures and 7×7×320 pictures;

Inputting the fused image into a strip attention module for striping processing to obtain an output image;

Segment the target object in the output image, and output the target segmented image.

In a second aspect, an embodiment of the present invention provides a device for segmenting a portrait, which includes: .

The dimension scaling unit is used to input the image to be segmented into the EfficientNet network for multiple dimension scaling processing, and output the corresponding dimension-reduced images respectively;

The feature fusion unit is used to input the last two output dimensionality reduction pictures into the attention fusion module for feature fusion to obtain a fusion picture, where the last two output dimensionality reduction pictures are 7×7×192 pictures and 7×7×320 pictures ;

a striping processing unit, configured to input the fused image into a strip attention module for striping processing to obtain an output image;

The segmentation unit is configured to segment the target object in the output image, and output the target segmented image.

In a third aspect, an embodiment of the present invention further provides a computer device, which includes a memory, a processor, and a computer program stored on the memory and executable on the processor, where the processor executes the computer The program implements the portrait segmentation method described in the first aspect above.

In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when executed by a processor, the computer program causes the processor to execute the above-mentioned first step. The portrait segmentation method described on the one hand.

Embodiments of the present invention disclose a method, device, computer equipment and storage medium for portrait segmentation. The method includes inputting the image to be segmented into the EfficientNet network for multiple dimension scaling processing, and outputting corresponding dimension-reduced images respectively; inputting the last two output dimension-reduced images into an attention fusion module for feature fusion to obtain a fusion image, wherein the last two output images are fused. The output dimensionality reduction pictures are 7 × 7 × 192 pictures and 7 × 7 × 320 pictures; the fusion picture is input into the strip attention module for striping processing to obtain the output image; the target object in the output image is segmented, and the output target Split the image. The embodiment of the present invention has the advantage of improving the accuracy of portrait segmentation by improving the network structure.

Description of drawings

In order to explain the technical solutions of the embodiments of the present invention more clearly, the following briefly introduces the accompanying drawings used in the description of the embodiments. Obviously, the drawings in the following description are some embodiments of the present invention, which are of great significance to the art For those of ordinary skill, other drawings can also be obtained from these drawings without any creative effort.

1 is a schematic flowchart of a portrait segmentation method provided by an embodiment of the present invention;

2 is a schematic sub-flow diagram of a portrait segmentation method provided by an embodiment of the present invention;

3 is a schematic diagram of another sub-flow of the portrait segmentation method provided by an embodiment of the present invention;

4 is a schematic diagram of another sub-flow of the portrait segmentation method provided by an embodiment of the present invention;

FIG. 5 is a schematic block diagram of a portrait segmentation apparatus provided by an embodiment of the present invention;

FIG. 6 is a schematic block diagram of a computer device according to an embodiment of the present invention.

Detailed ways

The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, but not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

It is to be understood that, when used in this specification and the appended claims, the terms "comprising" and "comprising" indicate the presence of the described features, integers, steps, operations, elements and/or components, but do not exclude one or The presence or addition of a number of other features, integers, steps, operations, elements, components, and/or sets thereof.

It is also to be understood that the terminology used in this specification of the present invention is for the purpose of describing particular embodiments only and is not intended to limit the present invention. As used in this specification and the appended claims, the singular forms "a," "an," and "the" are intended to include the plural unless the context clearly dictates otherwise.

It should further be understood that, as used in this specification and the appended claims, the term "and/or" refers to and including any and all possible combinations of one or more of the associated listed items .

Please refer to FIG. 1, which is a schematic flowchart of a portrait segmentation method provided by an embodiment of the present invention;

As shown in FIG. 1 , the method includes steps S101 to S104.

S101. Input the image to be segmented into the EfficientNet network to perform multiple dimension scaling processing, and output the corresponding dimension-reduced images respectively;

In this step, the EfficientNet network contains 7 convolution modules,

S102, inputting the last two output dimensionality reduction pictures into an attention fusion module for feature fusion to obtain a fusion picture, wherein the last two output dimensionality reduction pictures are 7×7×192 pictures and 7×7×320 pictures;

S103, inputting the fused image into a strip attention module for striping processing to obtain an output image;

S104: Segment the target object in the output image, and output the target segmented image.

This embodiment adopts the AttaNet network framework and uses the EfficientNet network as the core Backbone, which can balance the three dimensions of width, depth and resolution, thereby improving the accuracy and speed of the AttaNet network.

The EfficientNet network is used to perform dimension scaling of seven convolution modules in turn on the images to be segmented, and output the corresponding dimension-reduced images respectively, and then the last two dimension-reduced images are fused through the attention fusion module (AFM) to obtain the fusion image. Then, the fused image is striped through the strip attention module (SAM) to obtain the output image. Different color features are used in the output image to distinguish different objects, and the target is segmented according to the characteristics of the target object (ie, portrait features), that is, The target segmentation image can be output.

In one embodiment, as shown in FIG. 2 , step S101 includes:

S201. Input the 224×224×32 picture into the first convolution module for convolution processing, and output the 112×112×32 picture;

Among them, the number of convolution layers of the first convolution module is 1, and the convolution kernel is 3 × 3; that is, 224 × 224 × 32 pictures are convolutional processed once in the first convolution module and output 112 × 112 × 32 picture;

S202, input the 112×112×32 pictures into the second convolution module for convolution processing, and output 56×56×24 pictures;

Among them, the number of convolution layers of the second convolution module is 3, and the convolution kernel is 3 × 3; that is, the 112 × 112 × 32 image is subjected to 3 convolution processing in the second convolution module, and the final output is 56 × 56 × 24 pictures;

S203, input the 56×56×24 picture into the third convolution module for convolution processing, and output the 28×28×40 picture;

Among them, the number of convolution layers of the third convolution module is 2, and the convolution kernel is 5 × 5; that is, the 56 × 56 × 24 image is subjected to 2 convolution processing in the third convolution module, and the final output is 28 × 28 × 40 pictures;

S204, input the 28×28×40 picture into the fourth convolution module for convolution processing, and output the 28×28×80 picture;

Among them, the number of convolution layers in the fourth convolution module is 3, and the convolution kernel is 3 × 3; that is, the 28 × 28 × 40 image is subjected to 3 convolution processing in the fourth convolution module, and the final output is 28 × 28 × 80 pictures;

S205, input the 28×28×80 picture into the fifth convolution module for convolution processing, and output the 7×7×192 picture;

Among them, the number of convolution layers of the fifth convolution module is 7, and the convolution kernel is 5 × 5; that is, the 28 × 28 × 80 image is subjected to 7 convolution processing in the fifth convolution module, and the final output is 7 × 7 × 192 pictures;

S206: Input the 7×7×192 picture into the sixth convolution module for convolution processing, and output the 7×7×320 picture.

Among them, the number of convolution layers of the sixth convolution module is 1, and the convolution kernel is 3 × 3, that is, 7 × 7 × 192 pictures are convolutional processed once in the sixth convolution module, and finally output 7 × 7 × 192 pictures.

Through the dimensionality reduction processing of the seven convolution modules in steps S201-S206, the overall performance of the network is maximized.

In one embodiment, as shown in FIG. 3 , step S102 includes:

S301. Perform convolution processing on a 7×7×192 picture to obtain a first picture;

In this step, a convolution layer with a convolution kernel of 3×3 is used for convolution processing;

S302, perform upsampling processing on the 7×7×192 picture, then perform convolution, pooling and activation processing, and perform weight assignment to obtain a second picture;

In this step, after the upsampling processing, the Concate function processing, the convolution processing of the 1×1 convolution kernel, the Relu activation function processing, the global pool pooling processing, the 1×1 convolution processing and the Sigmoid activation function processing are sequentially performed, and then the After the weight α, the second picture is obtained;

S303, performing a product operation on the first picture and the second picture and then performing upsampling processing to obtain a third picture;

In this step, adopt image multiplication operation Multply to carry out product operation;

S304, performing convolution processing on the 7×7×320 picture to obtain a fourth picture;

In this step, a convolution layer with a convolution kernel of 3×3 is used for convolution processing;

S305, performing convolution, pooling and activation processing on the fourth picture, and performing weight assignment to obtain the fifth picture;

In this step, Concate function processing, 1×1 convolution kernel convolution processing, Relu activation function processing, Global pool pooling processing, 1×1 convolution processing, and Sigmoid activation function processing are sequentially performed on the fourth picture, and then assigned After the weight is 1-α, the fifth picture is obtained;

S306, performing the product operation on the fourth picture and the fifth picture and then performing upsampling processing to obtain the sixth picture;

S307. Perform feature fusion of the third picture and the sixth picture to obtain a fusion picture.

In this embodiment, the attention fusion module performs feature fusion on the 7×7×192 picture and the 7×7×320 picture according to the process of steps S301-S307, so as to further improve the speed under the condition of ensuring the accuracy. It should be noted that, in this embodiment, the global attention mechanism is introduced into the attention fusion module to assign weights, and the weights are combined to perform feature fusion, which greatly improves the feature accuracy of the fused image.

In one embodiment, as shown in FIG. 4 , step S103 includes:

S401. Perform three convolution operations on the fused image respectively to obtain a first convolution image, a second convolution image and a third convolution image respectively;

In this step, a convolution layer with a convolution kernel of 3×3 can be used to perform three convolution operations on the fused image respectively, to obtain a first convolution image, a second convolution image and a third convolution image;

S402, performing dimension transposition processing on the first convolutional image to obtain a transposed image;

S403, performing striping processing and dimension arrangement processing on the second convolution image, and then batch processing with the transposed image to obtain an attention matrix;

In this step, after the second convolutional image is processed by Striping and Reshape dimension arrangement, and then Affinity batch processing with the transposed image, an N×W attention matrix is obtained, the H direction is removed, and the vertical direction is greatly reduced. The complexity of global context encoding;

S404 , performing striping processing and dimension transposing processing on the third convolutional image, multiplying the third convolutional image with the attention matrix, and performing arrangement processing, and then adding the image with the fusion image to obtain an output image.

In this embodiment, the fused image is striped by the strip attention module, and the image is output. The output image accurately divides the regions of each object, so as to facilitate subsequent accurate segmentation.

In one embodiment, step S104 includes:

Obtain the regional edge information of each object in the output image, and use the semantic segmentation network to segment the target object according to the regional edge information of the target object to obtain the target segmentation image.

In this embodiment, a semantic segmentation network is used to perform segmentation processing on the target object based on the accurately divided regions of each object in the output image and the edge information of the target object region to be segmented, so that a highly accurate target segmentation image can be obtained.

In one embodiment, the portrait segmentation method further includes:

Optimize the portrait segmentation network model according to the following loss function:

Loss=Length+λ·Region;

Among them, Length represents the distance function, λ represents the hyperparameter, and Region represents the region function.

In this embodiment, when the small target is segmented, due to the obvious imbalance between the background and the foreground, the background is often much larger than the foreground, so that the loss function is dominated by the background, and the segmentation effect is not good. Therefore, this embodiment introduces cross entropy loss. function to improve the segmentation effect of small objects.

An embodiment of the present invention further provides a portrait segmentation apparatus, which is used for performing any of the foregoing embodiments of the portrait segmentation method. Specifically, please refer to FIG. 5 , which is a schematic block diagram of an apparatus for segmenting a human portrait provided by an embodiment of the present invention.

As shown in FIG. 5 , the portrait segmentation apparatus 500 includes: a dimension scaling unit 501 , a feature fusion unit 502 , a striping processing unit 503 and a segmentation unit 504 .

The dimension scaling unit 501 is used to input the picture to be divided into the EfficientNet network for multiple dimension scaling processing, and output the corresponding dimension reduction picture respectively;

The feature fusion unit 502 is used for inputting the last two output dimensionality reduction pictures into the attention fusion module for feature fusion to obtain a fusion picture, wherein the last two output dimensionality reduction pictures are 7×7×192 pictures and 7×7×320 picture;

The striping processing unit 503 is used to input the fused image into the strip attention module for striping processing to obtain an output image;

The segmentation unit 504 is configured to segment the target object in the output image, and output the target segmented image.

The device adopts the AttaNet network framework, takes the EfficientNet network as the core Backbone, and balances the three dimensions of width, depth and resolution, thereby improving the accuracy and speed of the AttaNet network; by improving the network structure, it has the ability to improve portrait segmentation. The advantage of accuracy.

Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the above-described devices and units may refer to the corresponding processes in the foregoing method embodiments, which will not be repeated here.

The above-mentioned portrait segmentation apparatus can be implemented in the form of a computer program, and the computer program can be executed on a computer device as shown in FIG. 6 .

Please refer to FIG. 6, which is a schematic block diagram of a computer device provided by an embodiment of the present invention. The computer device 600 is a server, and the server may be an independent server or a server cluster composed of multiple servers.

Referring to FIG. 6 , the computer device 600 includes a processor 602 , a memory and a network interface 605 connected through a system bus 601 , wherein the memory may include a non-volatile storage medium 603 and an internal memory 604 .

The nonvolatile storage medium 603 can store an operating system 6031 and a computer program 6032 . When executed, the computer program 6032 can cause the processor 602 to execute the portrait segmentation method.

The processor 602 is used to provide computing and control capabilities to support the operation of the entire computer device 600 .

The internal memory 604 provides an environment for running the computer program 6032 in the non-volatile storage medium 603. When the computer program 6032 is executed by the processor 602, the processor 602 can execute the portrait segmentation method.

The network interface 605 is used for network communication, such as providing transmission of data information. Those skilled in the art can understand that the structure shown in FIG. 6 is only a block diagram of a partial structure related to the solution of the present invention, and does not constitute a limitation on the computer device 600 to which the solution of the present invention is applied. The specific computer device 600 may include more or fewer components than shown, or combine certain components, or have a different arrangement of components.

Those skilled in the art can understand that the embodiment of the computer device shown in FIG. 6 does not constitute a limitation on the specific structure of the computer device. Either some components are combined, or different component arrangements. For example, in some embodiments, the computer device may only include a memory and a processor. In such an embodiment, the structures and functions of the memory and the processor are the same as those of the embodiment shown in FIG. 6 , and details are not repeated here.

It should be understood that, in this embodiment of the present invention, the processor 602 may be a central processing unit (Central Processing Unit, CPU), and the processor 602 may also be other general-purpose processors, digital signal processors (Digital Signal Processor, DSP), dedicated integrated Circuit (Application Specific Integrated Circuit, ASIC), off-the-shelf Programmable Gate Array (Field-Programmable Gate Array, FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, and the like. Wherein, the general-purpose processor can be a microprocessor or the processor can also be any conventional processor or the like.

In another embodiment of the present invention, a computer-readable storage medium is provided. The computer-readable storage medium may be a non-volatile computer-readable storage medium. The computer-readable storage medium stores a computer program, wherein when the computer program is executed by the processor, the method for segmenting a portrait of the embodiment of the present invention is implemented.

The storage medium is a physical, non-transitory storage medium, such as a U disk, a removable hard disk, a read-only memory (Read-Only Memory, ROM), a magnetic disk or an optical disk, and other physical storage mediums that can store program codes.

Those skilled in the art can clearly understand that, for the convenience and brevity of description, for the specific working process of the above-described devices, devices and units, reference may be made to the corresponding processes in the foregoing method embodiments, which will not be repeated here.

The above are only specific embodiments of the present invention, but the protection scope of the present invention is not limited to this. Any person skilled in the art can easily think of various equivalents within the technical scope disclosed by the present invention. Modifications or substitutions should be included within the protection scope of the present invention. Therefore, the protection scope of the present invention should be subject to the protection scope of the claims.

Claims (10)

1. a portrait segmentation method, is characterized in that, comprises: Input the image to be segmented into the EfficientNet network for multiple dimension scaling, and output the corresponding dimension-reduced images respectively; The last two output dimensionality reduction pictures are input into the attention fusion module for feature fusion to obtain fusion pictures, of which the last two output dimensionality reduction pictures are 7×7×192 pictures and 7×7×320 pictures; Inputting the fused image into a strip attention module for striping processing to obtain an output image; Segment the target object in the output image, and output the target segmented image. 2. portrait segmentation method according to claim 1, it is characterised in that the last two output dimensionality reduction pictures are input into the attention fusion module to carry out feature fusion to obtain fusion pictures, comprising: Perform convolution processing on the 7×7×192 picture to obtain the first picture; Perform upsampling processing on the 7×7×192 picture, then perform convolution, pooling and activation processing, and perform weight assignment to obtain a second picture; Carry out product operation on the first picture and the second picture and then perform upsampling processing to obtain a third picture; Perform convolution processing on the 7×7×320 picture to obtain a fourth picture; Performing convolution, pooling and activation processing on the fourth picture, and performing weight assignment to obtain the fifth picture; The fourth picture and the fifth picture are multiplied and then subjected to up-sampling processing to obtain the sixth picture; Feature fusion is performed on the third picture and the sixth picture to obtain a fusion picture. 3. The portrait segmentation method according to claim 1, wherein the inputting the fused image into the strip attention module is striped to obtain an output image, comprising: Performing three convolution operations on the fused image respectively to obtain a first convolution image, a second convolution image and a third convolution image; performing dimension transposition processing on the first convolution image to obtain a transposed image; Perform striping processing and dimension arrangement processing on the second convolution image, and then perform batch processing with the transposed image to obtain an attention matrix; Striping processing and dimension transposing processing are performed on the third convolutional image, multiplied by the attention matrix, and then arranging processing, and then added with the fusion image to obtain an output image. 4. portrait segmentation method according to claim 1, is characterized in that: described to be divided into picture input EfficientNet network to carry out multiple dimension scaling processing, output corresponding dimensionality reduction picture respectively, comprising: Input the 224×224×32 pictures into the first convolution module for convolution processing, and output 112×112×32 pictures; Input the 112×112×32 picture into the second convolution module for convolution processing, and output a 56×56×24 picture; Input the 56×56×24 picture into the third convolution module for convolution processing, and output a 28×28×40 picture; Input the 28×28×40 picture into the fourth convolution module for convolution processing, and output a 28×28×80 picture; Input the 28×28×80 pictures into the fifth convolution module for convolution processing, and output 7×7×192 pictures; The 7×7×192 picture is input into the sixth convolution module for convolution processing, and a 7×7×320 picture is output. 5. portrait segmentation method according to claim 4, is characterized in that: The number of convolution layers of the first convolution module is 1, and the convolution kernel is 3×3; The number of convolution layers of the second convolution module is 3, and the convolution kernel is 3×3; The number of convolution layers of the third convolution module is 2, and the convolution kernel is 5×5; The number of convolution layers of the fourth convolution module is 3, and the convolution kernel is 3×3; The number of convolution layers of the fifth convolution module is 7, and the convolution kernel is 5×5; The number of convolution layers of the sixth convolution module is 1, and the convolution kernel is 3×3. 6. The portrait segmentation method according to claim 1, wherein the segmenting the target object in the output image and outputting the target segmented image comprises: Obtain the regional edge information of each object in the output image, and use a semantic segmentation network to perform segmentation processing on the target object according to the regional edge information of the target object to obtain a target segmented image. 7. portrait segmentation method according to claim 1, is characterized in that, also comprises: Optimize the portrait segmentation network model according to the following loss function: Loss=Length+λ·Region; Among them, Length represents the distance function, λ represents the hyperparameter, and Region represents the region function. 8. A portrait segmentation device, characterized in that, comprising: The dimension scaling unit is used to input the image to be segmented into the EfficientNet network for multiple dimension scaling processing, and output the corresponding dimension-reduced images respectively; The feature fusion unit is used to input the last two output dimensionality reduction pictures into the attention fusion module for feature fusion to obtain a fusion picture, where the last two output dimensionality reduction pictures are 7×7×192 pictures and 7×7×320 pictures ; a striping processing unit, configured to input the fused image into a strip attention module for striping processing to obtain an output image; The segmentation unit is configured to segment the target object in the output image, and output the target segmented image. 9. A computer device comprising a memory, a processor and a computer program stored on the memory and running on the processor, wherein the processor implements the computer program as claimed in the claims The portrait segmentation method according to any one of 1 to 7. 10. A computer-readable storage medium, characterized in that, the computer-readable storage medium stores a computer program, the computer program, when executed by a processor, causes the processor to execute any one of claims 1 to 7 The portrait segmentation method described in item.

Images