CN114724218A - Video detection method, device, equipment and medium - Google Patents

Abstract

The present disclosure relates to a video detection method, apparatus, device, and medium. The video detection method comprises the following steps: acquiring an image sequence to be detected, wherein the image sequence comprises at least two video frames in the same video; carrying out nonlinear transformation processing on the facial features of the images aiming at each image in the image sequence to obtain the attention features of a plurality of regions of the face corresponding to the images; constructing a time sequence relation characteristic among a plurality of regions of the face corresponding to the image sequence based on the attention characteristics of the plurality of regions of the face corresponding to each image; and calculating the probability that the video is the video with the forged face based on the time sequence relation characteristics. According to the embodiment of the disclosure, the accuracy of the probability calculation result is higher, the generalization capability is stronger, and the accuracy of the fake face video detection is further improved.

Description

Video detection method, device, equipment and medium

technical field

The present disclosure relates to the technical field of video processing, and in particular, to a video detection method, apparatus, device, and medium.

Background technique

Fake face videos refer to videos in which faces such as human faces and animal faces in the video content have been tampered with by deepfake algorithms.

Therefore, how to accurately detect fake face videos is a technical problem that needs to be solved urgently.

SUMMARY OF THE INVENTION

In order to solve the above technical problems, the present disclosure provides a video detection method, apparatus, device and medium.

In a first aspect, the present disclosure provides a video detection method, including:

Obtain an image sequence to be detected, the image sequence includes at least two video frames in the same video;

For each image in the image sequence, non-linear transformation processing is performed on the facial features of the image to obtain the attention features of multiple regions of the face corresponding to the image;

Based on the attention features of the multiple face regions corresponding to each image, construct the temporal relationship features between the multiple face regions corresponding to the image sequence;

Based on the temporal relationship feature, the probability that the video is a fake face video is calculated.

In a second aspect, the present disclosure provides a video detection device, comprising:

an image acquisition module for acquiring an image sequence to be detected, the image sequence including at least two video frames in the same video;

The nonlinear change module is used to perform nonlinear transformation processing on the facial features of the image for each image in the image sequence, so as to obtain the attention features of multiple areas of the face corresponding to the image;

The feature building module is used to construct the temporal relationship feature between the multiple face regions corresponding to the image sequence based on the attention features of the multiple face regions corresponding to each image;

The probability calculation module is used to calculate the probability that the video is a video of a fake face based on the time series relationship feature.

In a third aspect, the present disclosure provides a video detection device, including:

processor;

memory for storing executable instructions;

The processor is configured to read executable instructions from the memory and execute the executable instructions to implement the video detection method of the first aspect.

In a fourth aspect, the present disclosure provides a computer-readable storage medium storing a computer program, which, when executed by a processor, causes the processor to implement the video detection method of the first aspect.

Compared with the prior art, the technical solutions provided by the embodiments of the present disclosure have the following advantages:

The video detection method, device, device, and medium of the embodiments of the present disclosure can, after acquiring an image sequence to be detected that includes at least two video frames in the same video, perform facial features of each image in the image sequence. Through nonlinear transformation processing, the attention features of multiple face regions corresponding to each image are obtained, and based on the attention features of multiple face regions corresponding to each image, the relationship between multiple face regions corresponding to the image sequence is constructed. time sequence relationship feature, and then based on the time sequence relationship feature, calculate the probability that the video is a video of a fake face, and this probability can be used to determine whether the video is a video of a fake face, because in the embodiment of the present disclosure, it can be based on the corresponding image sequence in the image sequence. To calculate the probability, the timing relationship between multiple face regions can be calculated by introducing the timing relationship between multiple face regions, and then to detect the timing inconsistency of the face in the video, which makes the calculation of the probability The results are more accurate and generalize better, which in turn improves the accuracy of fake face video detection.

Description of drawings

The above and other features, advantages and aspects of various embodiments of the present disclosure will become more apparent when taken in conjunction with the accompanying drawings and with reference to the following detailed description. Throughout the drawings, the same or similar reference numbers refer to the same or similar elements. It should be understood that the drawings are schematic and that the originals and elements are not necessarily drawn to scale.

1 is a schematic flowchart of a video detection method according to an embodiment of the present disclosure;

FIG. 2 is a schematic flowchart of a method for acquiring an image sequence according to an embodiment of the present disclosure;

3 is a schematic flowchart of a method for acquiring an attention feature according to an embodiment of the present disclosure;

FIG. 4 is a schematic flowchart of another attention feature acquisition method provided by an embodiment of the present disclosure;

FIG. 5 is a schematic flowchart of a method for obtaining a timing relationship feature according to an embodiment of the present disclosure;

FIG. 6 is a schematic flowchart of another method for obtaining a timing relationship feature according to an embodiment of the present disclosure;

7 is a schematic diagram of a video detection model provided by an embodiment of the present disclosure;

8 is a schematic diagram of a three-dimensional attention neural network model provided by an embodiment of the present disclosure;

9 is a schematic diagram of a sequence diagram neural network model provided by an embodiment of the present disclosure;

10 is a schematic structural diagram of a video detection apparatus according to an embodiment of the present disclosure;

FIG. 11 is a schematic structural diagram of a video detection device according to an embodiment of the present disclosure.

Detailed ways

Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. While certain embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be embodied in various forms and should not be construed as limited to the embodiments set forth herein, but rather are provided for the purpose of A more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes, and are not intended to limit the protection scope of the present disclosure.

It should be understood that the various steps described in the method embodiments of the present disclosure may be performed in different orders and/or in parallel. Furthermore, method embodiments may include additional steps and/or omit performing the illustrated steps. The scope of the present disclosure is not limited in this regard.

As used herein, the term "including" and variations thereof are open-ended inclusions, ie, "including but not limited to". The term "based on" is "based at least in part on." The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". Relevant definitions of other terms will be given in the description below.

It should be noted that concepts such as "first" and "second" mentioned in the present disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order of functions performed by these devices, modules or units or interdependence.

It should be noted that the modifications of "a" and "a plurality" mentioned in the present disclosure are illustrative rather than restrictive, and those skilled in the art should understand that unless the context clearly indicates otherwise, they should be understood as "one or a plurality of". multiple".

The names of messages or information exchanged between multiple devices in the embodiments of the present disclosure are only for illustrative purposes, and are not intended to limit the scope of these messages or information.

At present, the detection methods of fake face videos are mainly divided into two categories:

(1) Image-based detection method. Among them, image-based detection methods may include traditional image forensics methods, deep learning methods, image internal pattern analysis methods, and forged image fingerprint methods.

Specifically, the image-based detection method is a method of detecting fake face videos with video frames as the granularity. Usually, artifacts generated by deep forgery algorithms, such as image noise, uneven texture, etc., are used as important clues for fake face detection. Implement detection of fake face videos. Although image-based detection methods have achieved good detection performance with the help of advanced convolutional neural network architectures, due to the continuous improvement of deepfake algorithms, image-based detection methods have the disadvantage of weak generalization performance. The detection method of the novel deepfake algorithm has poor accuracy in detecting fake face videos produced by the new deepfake algorithm.

(2) Video-based detection methods. Among them, the video-based detection method may include a method for detecting by using the temporal consistency of the video as a clue.

Specifically, the video-based detection method is a method of detecting fake face videos with video as the granularity. Since deep forgery technology generally tampers with face videos by processing video frames one by one, this processing method will lead to the existence of existence between video frames. Therefore, video-based detection methods usually use the temporal consistency generated by deepfake algorithms as an important clue for fake face detection to detect fake face videos.

Among them, the consistency can include multi-modal consistency and video frame consistency.

Multi-modal consistency can be used to determine whether the facial and lip motion modes of the video are consistent with the voice and audio modes, so as to detect fake face videos, although this kind of fake face video detection is based on multi-modal consistency. The method can improve the generalization performance of fake face video detection, but this method cannot be used to detect video without audio or audio is covered, that is, the accuracy of this method when detecting video without audio or audio is covered lower.

Video frame consistency can be used to detect the timing consistency between video frames, that is, by detecting whether the changes of video frames are continuous to detect whether the face video has been forged, although this method of forgery face video detection based on video consistency The generalization performance of fake face video detection is improved, and the limitations of the method for fake face video detection based on multimodal consistency can be avoided, but when using this method for fake face video detection based on video consistency During the process, the applicant found that it still has the following problems: (1) the prior art uses a two-dimensional attention mechanism to obtain local area features from the face image; (2) the prior art directly uses the local area features for classification after obtaining the local area features , or only model the inter-regional relationship of the image after obtaining local region features, and use the inter-regional relationship for classification. Therefore, this method for fake face video detection based on video consistency still has the problem of low accuracy.

To sum up, the above two types of detection methods for fake face videos have the problem of low accuracy in detecting fake face videos. Therefore, how to accurately detect fake face videos is a technical problem that needs to be solved urgently.

In view of the above problems, embodiments of the present disclosure provide a video detection method, apparatus, device, and medium. The video detection method is first introduced below with reference to specific embodiments.

In an embodiment of the present disclosure, the video detection method may be performed by a computing device. The computing devices may include electronic devices, servers, etc., which are not limited herein. Electronic devices may include mobile phones, tablet computers, desktop computers, notebook computers, vehicle-mounted terminals, wearable electronic devices, all-in-one computers, smart home devices, and other devices with computing functions, and may also be devices simulated by virtual machines or simulators. The server can be an independent server or a cluster of multiple servers, including a server built locally and a server built in the cloud.

FIG. 1 is a schematic flowchart of a video detection method provided by an embodiment of the present disclosure.

As shown in FIG. 1 , the video detection method may include the following steps.

S110. Acquire an image sequence to be detected, where the image sequence includes at least two video frames in the same video.

In an embodiment of the present disclosure, the computing device may acquire an image sequence to be detected that includes at least two video frames in the same video.

The image sequence includes at least two video frames in the same video, and the at least two video frames may be two adjacent video frames in the video, or may be two video frames that are not adjacent in the video. The sequence order of the image sequence, that is, the arrangement sequence of at least two video frames in the image sequence, may be determined in a first-come-last order according to the playback time of each video frame in the video.

Further, the video to which the image sequence belongs may be a video that needs to be detected whether the face in the video content has been tampered with.

In some embodiments, the at least two video frames in the image sequence may be video frames obtained by performing overall frame extraction on the video.

In other embodiments, the at least two video frames in the image sequence may be video frames obtained by segmenting the video.

S120. For each image in the image sequence, perform nonlinear transformation processing on the facial features of the image to obtain attention features of multiple face regions corresponding to the image.

In this embodiment of the present disclosure, after acquiring the image sequence to be detected, the computing device may perform nonlinear transformation processing on the facial features of each image in the image sequence, to obtain the facial features of multiple face regions corresponding to each image. Attention features.

Specifically, after acquiring the image sequence to be detected, the computing device may first extract the facial features of the images for each image in the image sequence, and then perform fusion computation based on the attention mechanism on the facial features of the images, so as to Realize the nonlinear transformation processing of the facial features of the image, and obtain the attention features of multiple areas of the face corresponding to the image.

The face in the image may be the face of any object, for example, the face of a human being, the face of an animal, or the face of an avatar, which is not limited herein.

In some embodiments, the facial features of an image may be understood as features used to characterize the features of the faces in the images, for example, the facial features may include features of specific parts of the face, features of the contours of the face, and features of the face The features of facial expressions, etc., are not limited here.

Optionally, the facial feature may be a two-dimensional facial feature or a three-dimensional facial feature, which is not limited herein.

In some embodiments, the face of each object can be pre-divided into multiple regions as required. Among them, the attention features of multiple face regions corresponding to the image can be understood as features used to represent the degree of correlation between each region in the face in the image and the face.

Optionally, the attention feature may be a two-dimensional attention feature or a three-dimensional attention feature, which is not limited herein.

Exemplarily, after acquiring the image sequence to be detected, the computing device may first extract the two-dimensional facial features of the image for each image in the image sequence, such as extracting the two-dimensional face of the image by using a two-dimensional facial feature extractor. Then, the fusion calculation based on the two-dimensional attention mechanism, such as the two-dimensional convolution operation, is performed on the two-dimensional facial features of the image, so as to realize the nonlinear transformation processing of the two-dimensional facial features of the image, and obtain the face corresponding to the image. 2D attention features in multiple regions.

For example, after acquiring the image sequence to be detected, the computing device may first extract the three-dimensional facial features of the images for each image in the image sequence, such as extracting the three-dimensional facial features of the images by using a three-dimensional facial feature extractor extraction model. feature, and then perform fusion calculation based on the 3D attention mechanism on the 3D facial features of the image, such as 3D convolution operation, for example, based on the 3D attention neural network model to realize the 3D convolution operation, so as to realize the 3D facial features of the image. Non-linear transformation processing to obtain three-dimensional attention features of multiple face regions corresponding to the image.

S130. Based on the attention features of the multiple face regions corresponding to each image, construct a time sequence relationship feature between the multiple face regions corresponding to the image sequence.

In the embodiment of the present disclosure, after obtaining the attention features of the multiple face regions in each image, the computing device may perform a time sequence relationship construction process based on the entire image sequence on the attention features of the multiple face regions in each image, The temporal relationship features between multiple face regions corresponding to the image sequence are obtained.

Among them, the time-series relationship feature between multiple face regions corresponding to the image sequence can be understood as a feature used to represent the variation of the attention feature of each region in the face in the time dimension.

For example, if the face in the image can be pre-divided into four areas, including the area where the eyebrows are located, the area where the eyes are located, the area where the mouth is located, and the area where the nose is located, the temporal relationship features between multiple areas of the face corresponding to the image sequence can be: Changes of the images in the area where the eyebrows are located, the area where the eyes are located, the area where the mouth is located, and the area where the nose is located at the times when the different video frames corresponding to the image sequence belong.

In some embodiments, the computing device may construct graph structure data of the multiple face regions corresponding to each image according to the attention features of the multiple face regions corresponding to the respective images, and then use the graph convolution operation to analyze the graph structure data for each face region. The graph structure data of the multiple face regions corresponding to the image are subjected to time sequence relationship construction processing to obtain the time sequence relationship features between the multiple face regions corresponding to the image sequence.

In other embodiments, the computing device may construct the vector data of the multiple face regions corresponding to each image according to the attention features of the multiple face regions corresponding to the respective images. The feature vectors corresponding to each face region corresponding to the image are spliced to obtain vector data of multiple face regions corresponding to each image. After the computing device obtains the vector data, the time-domain convolutional neural network can be used to construct the time series relationship between the vector data of the multiple face regions corresponding to each image, and obtain the relationship between the multiple face regions corresponding to the image sequence. temporal relationship characteristics.

S140. Calculate the probability that the video is a video of a fake face based on the time sequence relationship feature.

In the embodiment of the present disclosure, the computing device may classify and detect the time sequence relationship features between the multiple face regions corresponding to the image sequence after the time sequence relationship features between the multiple face regions corresponding to the image sequence, and obtain the Probability of detecting a video as a fake face video.

Optionally, the computing device may input the temporal relationship features between multiple regions of the face corresponding to the image sequence into a pre-trained classifier for detecting whether the video to which the temporal relationship feature belongs is a video of a fake face, to obtain: The probability that the video output by the classifier is a fake face video.

Further, the computing device may compare the probability with a preset probability threshold, and if the probability is greater than or equal to the preset probability threshold, determine that the video to be detected to which the image sequence belongs is a video of a fake face; If it is less than the preset probability threshold, it is determined that the video to be detected to which the image sequence belongs is not a video of a fake face.

It should be noted that the preset probability threshold may be a probability value preset as required and used to characterize the video as a fake face video. For example, the preset probability threshold may be 0.5 or 0.8, which is not made here. limit.

In the embodiment of the present disclosure, after acquiring an image sequence to be detected that includes at least two video frames in the same video, nonlinear transformation processing can be performed on the facial features of each image in the image sequence to obtain each image Corresponding attention features of multiple face regions, and based on the attention features of multiple face regions corresponding to each image, construct the temporal relationship features between multiple face regions corresponding to the image sequence, and then based on the temporal relationship features , calculate the probability that the video is a fake face video, this probability can be used to determine whether the video is a fake face video, and the processing of the time series relationship feature can detect the time series inconsistency of the face in the video, so that the calculation result of the probability It has higher accuracy and stronger generalization ability, thereby improving the accuracy of fake face video detection.

The method for acquiring the image sequence to be detected by the computing device will be described in detail below.

In some embodiments of the present disclosure, at least two video frames in the image sequence may be video frames obtained by segmenting the video.

In these embodiments, the computing device may segment the video to obtain a sequence of images. Specifically, a detailed description will be made with reference to FIG. 2 .

FIG. 2 is a schematic flowchart of a method for acquiring an image sequence according to an embodiment of the present disclosure. As shown in FIG. 2 , the image sequence acquisition method may include the following steps.

S210. Divide the video into multiple video segments.

In an embodiment of the present disclosure, the computing device may perform segmentation processing on the video to be detected, so as to divide the video into a plurality of video segments.

Specifically, the computing device may segment the video according to a preset segmentation manner to obtain multiple video segments.

The preset segmentation mode may be a preset mode for segmenting the video according to user requirements. For example, the preset segmentation method may be to divide the video into a certain number of video segments of equal length, and the preset segmentation method may also be to divide the video into video segments of a certain length, and the preset segmentation method It is also possible to randomly divide the video into multiple video segments, which is not limited here.

In some embodiments, the computing device may use an Open Source Computer Vision Library (OpenCV) tool to extract individual video frames in the video, and then employ a Multi-task convolutional neural network (MTCNN) face The partial detection model intercepts the facial images in each video frame, and stores the intercepted facial images in the form of pictures, and then divides the video into a certain number of video segments with equal length. The facial images in the consecutive video frames are segmented, and then the segmented groups of facial images are regarded as multiple video segments.

In other embodiments, the computing device can use the OpenCV tool to extract each video frame in the video, and store each extracted video frame in the form of a picture, and then divide the video into a certain number of video segments according to the same length. In the segmentation method, the stored continuous video frames are segmented, and then the segmented groups of video frames are regarded as multiple video segments.

S220. Extract key video frames of each video segment according to a preset frame extraction method.

In the embodiment of the present disclosure, after obtaining multiple video segments, the computing device may perform frame extraction processing on pictures in each video segment, and use the extracted pictures as key video frames of the video segment.

The key video frame can be understood as a video frame used to characterize the characteristics of the video segment to which it belongs.

Specifically, the preset frame extraction method may be a method preset as required for extracting frames from a video according to user requirements. For example, the preset frame extraction method may be to extract key video frames according to a certain interval, and the preset frame extraction method may also be to randomly extract a certain number of key video frames, which is not limited herein.

S230. Sort each key video frame according to the play time sequence to obtain an image sequence.

In the embodiment of the present disclosure, after obtaining the key video frames of each video segment, the computing device may sort the key video frames in the order of playback time to obtain an image sequence.

Therefore, in the embodiment of the present disclosure, the computing device can obtain an image sequence by segmenting frames, so that each video frame in the image sequence can cover each time period of the video, and then the detection of the fake face video can cover the entire video. , to further improve the accuracy of fake face video detection.

In other embodiments of the present disclosure, the at least two video frames in the image sequence may be video frames obtained by performing overall frame extraction on the video.

In some embodiments, the computing device may use an Open Source Computer Vision Library (OpenCV) tool to extract individual video frames in the video, and then employ a Multi-task convolutional neural network (MTCNN) face The facial image in each video frame is intercepted by the partial detection model, and each intercepted facial image is stored in the form of a picture to obtain a plurality of pictures corresponding to the video.

In other embodiments, the computing device may use the OpenCV tool to extract each video frame in the video, and store each extracted video frame in the form of a picture to obtain multiple pictures corresponding to the video.

In these embodiments, the computing device may perform overall frame extraction on the video to obtain the image sequence. Specifically, the computing device may perform an overall frame extraction process on multiple pictures corresponding to the video according to a preset frame extraction method to obtain multiple key video frames. After obtaining a plurality of key video frames, the computing device may sort each key video frame in the order of playback time to obtain an image sequence.

Wherein, the preset frame extraction method may be a method preset according to needs for extracting frames from a video according to user requirements. For example, the preset frame extraction method may be to extract key video frames according to a certain interval, and the preset frame extraction method may also be to randomly extract a certain number of key video frames, which is not limited herein.

Therefore, in some other embodiments of the present disclosure, the computing device can obtain an image sequence by extracting frames as a whole, so that each video frame in the image sequence can be flexibly selected from various positions in the entire video, thereby enabling the fake face Video detection can select the coverage range according to the actual situation, and further improve the accuracy of fake face video detection.

In still other embodiments of the present disclosure, in order to make it easier to extract the facial features of each frame of images in the key video frames, after the key video frames are extracted by the computing device, it is necessary to perform image enhancement processing on each key video frame first. , in order to enlarge the difference between different facial features in the image, obtain each processed key video frame, and then sort each processed key video frame according to the playback time sequence to obtain an image sequence.

By way of example, the data enhancement processing on the image may include horizontally flipping the image, translating a certain distance, scaling a certain ratio, rotating a certain angle, adjusting a certain hue value, adjusting a certain contrast, adjusting a certain saturation, adjusting a certain brightness, and adding a certain Gaussian noise. , at least one of processing methods such as performing certain motion blur filtering, performing certain Gaussian blur filtering, performing certain JPEG compression, and performing grayscale processing.

Optionally, the computing device may select at least one of the above data enhancement processing methods according to a certain probability to perform data enhancement processing on the image.

The specific processing method and selection probability of the image enhancement processing are described in detail below with a specific example, as shown in Table 1.

Table 1

Therefore, in the embodiment of the present disclosure, after the computing device extracts the key video frames, it first performs image enhancement processing on each key video frame to enlarge the difference between different facial features in the image, and then uses the processed key video frames. Frames constitute an image sequence, which makes the attention features and time sequence relationship features obtained by using each video frame in the image sequence more prominent, thereby making the calculated probability more accurate, and further improving the accuracy of fake face video detection.

The specific manner in which the computing device obtains the three-dimensional attention features of multiple face regions corresponding to the image will be described in detail below.

In some embodiments of the present disclosure, the computing device may obtain three-dimensional attention features of multiple regions of the face corresponding to the image through a three-dimensional convolution operation.

FIG. 3 is a schematic flowchart of a method for acquiring an attention feature according to an embodiment of the present disclosure. As shown in FIG. 3 , the attention feature acquisition method may include the following steps.

S310. Perform three-dimensional facial feature extraction processing on the image to obtain three-dimensional facial features corresponding to the image.

In the embodiment of the present disclosure, after acquiring any image in the image sequence, the computing device may perform three-dimensional facial feature extraction processing on the image to obtain the three-dimensional facial feature corresponding to the image.

Optionally, after acquiring any image, the computing device may input the image data corresponding to the image into a three-dimensional facial feature extractor to obtain a feature map output by the three-dimensional facial feature extractor, and the feature map can be used as the corresponding image data of the image. 3D facial features.

For example, the three-dimensional facial feature extractor may be a three-dimensional residual network (Residual Network), such as a mixed convolutional neural network (Mixed Convolution Networks, MC3). Layers, the third to fifth layers are two-dimensional convolution layers, the feature map output by the MC3 can have four dimensions, and the four dimensions can be the number of channels, length, height and width, for example, the size of the feature map can be It is 256×20×14×14 (number of channels×length×height×width).

S320. Perform a three-dimensional convolution operation on the three-dimensional facial features corresponding to the image to obtain attention weight matrices of multiple face regions corresponding to the image.

In this embodiment of the present disclosure, after acquiring the three-dimensional facial features corresponding to the image, the computing device may perform a three-dimensional convolution operation on the three-dimensional facial features of the image to obtain the attention of multiple regions of the face corresponding to the image. weight matrix.

Optionally, S320 may specifically include: based on the three-dimensional attention neural network model, performing a three-dimensional convolution operation on the three-dimensional facial features corresponding to the image to obtain the attention weight matrices of multiple areas of the face corresponding to the image, wherein the three-dimensional attention The force neural network model is trained based on the temporal continuity loss function and the sparse attention contrast loss function.

Specifically, after acquiring the 3D facial features corresponding to the image, the computing device may input the 3D facial features into the 3D attention neural network model to obtain the attention weight matrix output by the 3D attention neural network model.

Exemplarily, the three-dimensional attention neural network model may include three attention mechanism modules to implement three-dimensional convolution operations on three-dimensional facial features. Among them, each attention mechanism module includes a set of 3D convolution layers, 3D batch normalization layers and activation layers. The 3D convolution layer is the sliding window operation of the convolution kernel in the 3D space of the input image, and the 3D batch normalization layer is used for small The batch normalization operation is performed on the five-dimensional input composed of batch four-bit data, and the three-dimensional activation layer is the activation function to perform nonlinear transformation operations on the features in the three-dimensional space. Therefore, a three-dimensional attention mechanism can be realized through a multi-layer neural network to solve the problem that the two-dimensional attention mechanism ignores timing information.

The following is a detailed description of the three-dimensional convolution operation network with a specific example, as shown in Table 2.

Table 2

Among them, the three numbers in parentheses can be understood as parameters representing the dimensions of length, height and width respectively, Conv3D (Convolution 3D) can be understood as a three-dimensional convolution layer, and BN3D (Batch Normalization 3D) can be understood as a three-dimensional batch normalization operation layer.

Leaky ReLU can be understood as an activation function applied to the three-dimensional activation layer, and its formula is:

Softmax can be understood as another activation function applied to the three-dimensional activation layer, and its formula is:

Further, after the three-dimensional face feature corresponding to the image is processed by the three-dimensional convolution operation network shown in Table 2, each channel of the obtained attention weight stores an attention matrix with a size of 14×14.

Therefore, in the embodiment of the present disclosure, the computing device may process the three-dimensional facial features of the image based on the three-dimensional convolution operation to obtain a three-dimensional representation of the degree of correlation between each region in the face and the face in the image. Attention features, so that attention features do not ignore timing information, thereby improving the accuracy of fake face video detection.

In the embodiment of the present disclosure, the 3D attention neural network model can be trained through a loss function using the back propagation method. The loss function can include a time series continuity loss function and a sparse attention contrast loss function. The parameters of the model are adjusted until the loss values corresponding to the time series continuity loss function and the sparse attention contrast loss function are respectively smaller than the corresponding loss value thresholds.

的表达式为：Among them, the time series continuity loss function

The expression is:

表示图像序列中第i张图像第j个脸部区域的注意力权重矩阵。where T represents the number of images in the image sequence, M represents the total number of face regions that the 3D attention neural network model pays attention to,

Represents the attention weight matrix of the jth face region of the ith image in the image sequence.

的表达式为：Sparse Attention Contrastive Loss Function

The expression is:

[·,·]表示向量合并运算， 1_W×1表示尺寸为W×1的全1向量，1_1×H表示尺寸为1×H的全1向量，

为指示函数，表示当i≠j时函数值为1，否则函数值为0，

表示信息熵函数，

表示

的信息熵阈值。where the vector

[·,·] represents the vector merging operation, 1 _W×1 represents an all-ones vector of size W×1, 1 _1×H represents an all-ones vector of size 1×H,

is an indicator function, indicating that the function value is 1 when i≠j, otherwise the function value is 0,

represents the information entropy function,

express

The information entropy threshold of .

Therefore, in the embodiment of the present disclosure, the face region concerned by the 3D attention neural network model can be always stable in the time dimension through the time series continuity loss function, and the 3D attention neural network can be made stable through the sparse attention contrast loss function. The network model focuses on multiple different face regions to maintain diversity in the spatial dimension, thereby improving the accuracy of the attention weight matrix output by the three-dimensional attention neural network model, thereby improving the accuracy of fake face video detection.

S330 , based on the three-dimensional facial feature corresponding to the image and the attention weight matrix of the multiple face regions corresponding to the image, generate the attention features of the multiple face regions corresponding to the image.

In the embodiment of the present disclosure, after acquiring the attention weight matrices of the multiple face regions corresponding to the image, the computing device may fuse the three-dimensional facial features corresponding to the image with the attention weight matrices of the multiple face regions , to obtain the attention features of multiple regions of the face corresponding to the image, which are described below with reference to FIG. 4 .

FIG. 4 is a schematic flowchart of another attention feature acquisition method provided by an embodiment of the present disclosure. As shown in FIG. 4 , the attention feature acquisition method may include the following steps.

S410. Use the product of the three-dimensional facial features corresponding to the image and the attention weight matrices of the multiple face regions corresponding to the image as the attention feature matrices of the multiple face regions corresponding to the image.

In the embodiment of the present disclosure, after obtaining the three-dimensional facial feature corresponding to any image in the image sequence and the attention weight matrix of the multiple face regions corresponding to the image, the computing device may calculate the three-dimensional facial feature corresponding to the image and the product of the attention weight matrix to obtain the attention feature matrix of multiple regions of the face corresponding to the image.

The product of the three-dimensional face feature and the attention weight matrix specifically refers to the outer product of the three-dimensional face feature and the attention weight matrix.

Exemplarily, the parameter dimension of the attention feature matrix may include color value, vector dimension, length, height and width. For example, the dimension of the attention feature matrix can be 256×8×20×14×14 (color value×vector dimension×length×height×width).

S420 , summing the height dimension and the width dimension of the attention feature matrices of the multiple face regions corresponding to the image, respectively, to obtain the summed attention feature matrix.

In this embodiment of the present disclosure, after obtaining the attention feature matrices of the multiple face regions corresponding to the image, the computing device may perform a summation process on the height dimension and the width dimension of the attention feature matrix, respectively, to obtain the summation. The attention feature matrix of .

S430. Perform global average pooling on the length dimension of the summed attention feature matrix to obtain attention features of multiple regions of the face corresponding to the image.

In the embodiment of the present disclosure, after obtaining the summed attention feature matrix corresponding to the image, the computing device may perform global average pooling on the length dimension of the summed attention feature matrix to obtain the face corresponding to the image. attention features in multiple regions.

Among them, the global average pooling can be understood as adopting the average method to reduce the dimension of the length dimension, thereby speeding up the operation speed.

Specifically, the computing device can calculate the average value in the length dimension of all regions of the obtained feature map of each channel, and obtain the attention features of multiple face regions corresponding to the image after global average pooling in the length dimension. .

Illustratively, the dimension parameter of the attention feature may include color value and vector dimension. For example, the size of the attention feature can be 256×8 (color value×vector dimension).

Therefore, in this embodiment of the present disclosure, after the computing device obtains the three-dimensional facial features corresponding to each image in the image sequence and the attention weight matrix of multiple face regions of the corresponding images, the three-dimensional facial features corresponding to the images can be calculated by the computing device. It is fused with the attention weight matrix of multiple areas of the face to obtain more accurate attention features, which can be used to characterize the relationship between each area in the face and the face in the image. The correlation degree of the face, and then characterize the role of each area in the face in the video detection, and then get more accurate video detection results.

In other embodiments of the present disclosure, the computing device may further acquire attention features of multiple regions of the face corresponding to the image through the ResNet18 network model.

Among them, the ResNet18 network model contains 17 convolutional layers and 1 fully connected layer.

Specifically, after obtaining the three-dimensional facial features corresponding to the image, the computing device can input the facial features into the ResNet18 network model, perform 17 convolution operations on the facial features, and add the operation results, The attention features of multiple face regions corresponding to the image are obtained.

The following will describe in detail a method for a computing device to obtain time-series relationship features between multiple face regions corresponding to an image sequence by using a time-series graph convolutional neural network.

FIG. 5 is a schematic flowchart of a method for acquiring a timing relationship feature according to an embodiment of the present disclosure. As shown in FIG. 5 , the method for obtaining a time sequence relationship feature may include the following steps.

S510. For each image, construct graph structure data of multiple face regions corresponding to the image according to the attention features of the multiple face regions corresponding to the image.

In the embodiment of the present disclosure, after obtaining the attention features of the multiple face regions corresponding to each image in the image sequence, the computing device may construct a graph structure of the multiple face regions corresponding to each image according to the attention features data.

Specifically, the attention feature can be in the form of a matrix, the attention feature is divided into multiple attention feature vectors along the channel dimension of the matrix, the attention feature vector is used as the node of the graph structure data, and the adjacency matrix is defined as the attention feature vector. The initial relationship feature between the force feature vectors, the relationship feature is used as the edge of the graph structure data, and the nodes and edges corresponding to each image are combined to obtain the graph structure data of multiple areas of the face corresponding to the obtained image.

S520. Perform a time sequence relationship construction process on the graph structure data of the multiple face regions corresponding to each image, to obtain the time sequence relationship feature between the multiple face regions corresponding to the image sequence.

In the embodiment of the present disclosure, after obtaining the graph structure data of the multiple face regions corresponding to each image, the computing device may perform a graph convolution operation on the graph structure data to obtain the graph structure data between the multiple face regions corresponding to the image sequence. temporal relationship characteristics.

Specifically, the computing device can input the graph structure data of multiple face regions corresponding to each image into the time sequence diagram neural network model, update the relational features of the graph structure data according to the input graph structure data, and obtain the corresponding image sequence data. Temporal relationship features between multiple regions of the face.

Among them, the neural network model of the sequence diagram can be understood as a neural network model that is presented in the form of a graph and can represent dynamic properties.

Thus, in this embodiment of the present disclosure, the computing device may perform a time sequence relationship feature construction process based on the attention features of multiple face regions corresponding to the image, and obtain the time sequence relationship feature between multiple face regions corresponding to the image sequence, The time sequence inconsistency of the face in the image sequence is detected by the time sequence relationship feature, and the time sequence inconsistency of the face in the video is detected, thereby further improving the detection accuracy of the fake face video.

Further, in the embodiment of the present disclosure, S520 may perform a time sequence relationship construction process based on a time sequence diagram neural network model on the graph structure data of the multiple face regions corresponding to each image, and the last hidden image sequence corresponding to the obtained image sequence may be processed. The state diagram is used as the feature of the time series relationship between multiple face regions corresponding to the image sequence, which will be described below with reference to FIG. 6 .

FIG. 6 is a schematic flowchart of another method for obtaining a timing relationship feature according to an embodiment of the present disclosure. As shown in FIG. 6 , the method for obtaining a time sequence relationship feature may include the following steps.

S610. Based on the time sequence diagram neural network model, perform time sequence relationship construction processing on the graph structure data of multiple face regions corresponding to each image, to obtain the last hidden state diagram corresponding to the image sequence.

In the embodiment of the present disclosure, after obtaining the graph structure data of multiple face regions corresponding to each image in the image sequence, the computing device inputs the graph structure data corresponding to the first image into the sequence diagram neural network according to the order in the image sequence In the network model, an implicit state diagram that can characterize the time series relationship characteristics of the image is obtained, and then the implicit state diagram corresponding to the first image and the graph structure data corresponding to the second image are jointly input into the neural network model of the time series diagram. , obtain a hidden state diagram that can characterize the timing relationship between the second image and the first image, and so on, input all the images in the image sequence into the timing diagram neural network model in turn, until the image sequence The graph structure data corresponding to the last image in and the hidden state graph corresponding to the previous image are jointly input into the neural network model of the sequence diagram, and the last hidden state graph corresponding to the image sequence is obtained.

In this embodiment of the present disclosure, a backpropagation method can be used to train a time sequence graph neural network model through a loss function, and the loss function can be a cross-entropy loss function until the loss value corresponding to the cross-entropy loss function is less than the corresponding loss value threshold.

In the process of training the time series graph neural network model, the back propagation method can be used to adjust the parameters of the time series graph neural network model through the cross entropy loss function until the loss value corresponding to the cross entropy loss function is less than the corresponding loss value threshold.

的表达式为：Among them, the cross entropy loss function

The expression is:

Among them, φ represents the classifier, y represents the authenticity label of the video to be detected, and H ⁽⁾ represents the hidden state diagram obtained after inputting the last key video frame.

S620. Use the last hidden state diagram as a time series relationship feature between multiple face regions corresponding to the image sequence.

In the embodiment of the present disclosure, the last hidden state diagram obtained after the computing device sequentially inputs the graph structure data corresponding to each image in the image sequence into the sequence diagram neural network model can represent the same face in all images in the image sequence Therefore, the computing device can use the last hidden state diagram output by the neural network model of the time sequence diagram as a time sequence relationship feature between multiple regions of the face corresponding to the image sequence.

Therefore, in the embodiment of the present disclosure, after obtaining the graph structure data of the multiple face regions corresponding to each image in the image sequence, the computing device can sequentially input the graph structure data of each image into the sequence according to the order in the sequence In the graph neural network model, the last hidden state diagram corresponding to the image sequence is obtained, and then the temporal relationship of the same face region between different images in the image sequence is represented by the hidden state diagram. The timing inconsistency of the face is detected, and the timing inconsistency of the face in the video is detected, which further improves the detection accuracy of the fake face video.

In one embodiment of the present disclosure, the above-mentioned 3D facial feature extractor, 3D attention neural network model, fusion module, graph structure building module, sequence diagram neural network model and classifier can form a video detection model, which is implemented in the present disclosure. The video detection method provided in the example can be implemented based on a video detection model. The following describes the structure and principle of the video detection model for detecting face videos with reference to FIG. 7 to FIG. 9 .

FIG. 7 is a schematic diagram of a video detection model provided by an embodiment of the present disclosure. As shown in FIG. 7 , after extracting the face image sequence 710 from the face video, the computing device can input the face image sequence 710 into the video detection model, and the three-dimensional face feature extractor in the video detection model can The individual face images are respectively subjected to 3D face feature extraction processing to obtain 3D face features 720 corresponding to each face image, and then the 3D attention neural network model in the video detection model can calculate and obtain a plurality of faces corresponding to each face image. The attention weight matrix 730 of the region, and then the fusion module in the video detection model can respectively fuse the three-dimensional face feature 720 corresponding to each face image with the attention weight matrix 730 to obtain the attention feature 740 corresponding to each face image. , and then the graph structure building module (not shown in the figure) can construct graph structure data 750 corresponding to each attention feature 740 , and the time series graph convolutional neural network model can generate the time series relationship feature of the image sequence based on each graph structure data 750 760. Finally, the classifier can obtain the detection result of whether the video to be detected is a fake face video based on the time sequence relationship feature 760, that is, the probability of whether the video to be detected is a fake face video.

Further, the specific structure and principle of the three-dimensional attention neural network model in FIG. 7 can be described with reference to FIG. 8 .

FIG. 8 is a schematic diagram of a three-dimensional attention neural network model provided by an embodiment of the present disclosure. As shown in FIG. 8 , the 3D attention neural network model 800 may include three 3D convolution processing modules 820, and each 3D convolution processing module 820 may respectively include a 3D convolution layer 821, a 3D batch normalization layer 822, and a 3D batch normalization layer 822. 3D activation layer 823.

The first three-dimensional convolution processing module 820 can process the three-dimensional face feature 810 corresponding to any face image to obtain the first feature matrix 830, and the second three-dimensional convolution processing module 820 can process the first feature matrix 830 to obtain the first feature matrix 830. The second feature matrix 840, the third three-dimensional convolution processing module 820 can process the second feature matrix 840 to obtain the attention weight matrix 850.

Further, the specific structure and principle of the graph structure building module and the time sequence graph convolutional neural network model in FIG. 7 can be described with reference to FIG. 9 .

In the embodiment of the present disclosure, after the fusion module obtains the attention feature corresponding to any face image, the graph structure building module can divide the attention feature into 8 attention feature vectors along the second dimension such as the channel dimension, wherein , and the dimension of each attention feature vector is 1 × 256. Next, the graph structure building module can define an adjacency matrix as the initial relationship feature between the attention feature vectors, where the size of the adjacency matrix is 8×8. Finally, the graph structure building block can use the graph convolution operation to construct the graph G=<V,E> with the attention feature vector as the node V and the relation feature as the edge E.

Specifically, the process of constructing a graph by a graph structure building module is as follows:

The graph structure building module first defines a graph convolution parameter matrix W _g with a size of 256×384, inputs the graph convolution parameter matrix W _g 910 into the first graph convolution operation unit 920, and makes the first graph convolution operation unit 920 Perform a graph convolution operation GConv(G)=EVW _g on the graph convolution parameter matrix W _g 910 to obtain a result matrix with a size of 8×384, and input the result matrix into the first vector segmentation unit 930 to make the first vector slice The dividing unit 930 divides the result matrix along the second dimension, such as the channel dimension, to obtain three hidden vectors G _r , G _z , and G _h , each of which has a size of 8×128. Next, the graph structure building module can define an initial hidden state with a size of 8×128 and a graph convolution parameter matrix W _h of a hidden state with a size of 128×384, where the initial value of the hidden state is 0. Next, the graph structure building module can input the data to be processed 940 formed by the implicit state and the above-mentioned shared relational features into the second graph convolution operation unit 950, so that the second graph convolution operation unit 950 can make the data 940 to be processed. The graph convolution operation GConv(H)=EhW _h is performed to obtain the hidden state graph 960 . Finally, the graph structure building module can input the implicit state graph 960 into the second vector segmentation unit 970, so that the second vector segmentation unit 970 can segment the implicit state graph 960 along the second dimension such as the channel dimension to obtain three Hidden vectors H _r , H _z , H _h , each of which has a size of 8×128.

的运算，得到的

可以理解为候选隐含状态9913，将候选隐含状态9913和结果9911输入第三乘积运算子单元9914中，得到第三乘积结果9915，将第二乘积结果999和第三乘积结果9915输入加法运算子单元9916中，执行

的运算，得到的H可以理解为更新后的隐含状态图9917，将该更新后的隐含状态图9917输入隐含状态图替换单元9100中，通过隐含状态图替换单元9100替换第二图卷积运算单元950输出的隐含状态图960，并将该更新后的隐含状态图9917输入到偏置参数调整子单元991中，对偏置参数进行调整，得到能提升时序图神经网络模型提取图像序列的时序关系特征的效果的偏置参数。The sequence diagram convolutional neural network model can set the initial bias parameter 980, and input the hidden state diagram 960, hidden vectors _{Gr, Gz, Gh, Hr, Hz, Hh , and initial bias parameter} 980 to gate in the arithmetic unit 990. The initial bias parameter 980 is input into the bias parameter adjustment subunit 991, and the initial bias parameter 980 is _vector -segmented to obtain bias parameter vectors br , b _z , and b _h . Input the hidden vectors H _r , Gr , _br into the reset gate operation sub-unit 992, and perform the r=sigmoid(G _r +H _r + _br ) operation, and the obtained _r can be understood as the reset result 993. The set result 993 and the hidden vector H _h are input into the first product operation subunit 994 to obtain the first product result 995 . Input the hidden vectors H _z , G _z , b _z into the update gate operation sub-unit 996, execute the z=sigmoid(G _z +H _z +b _z ) operation, and the obtained z can be understood as the update result 997, and the update result 997 and the implicit state diagram 960 are input into the second product operation subunit 998, the second product result 999 is obtained, and the update result 997 is input into the operation unit 9910, the operation of 1-z is performed, and the calculation result 9911 is obtained. A product result 995, vectors G _h , b _h are input into the candidate hidden state operation sub-unit 9912, and the execution

operation, resulting in

It can be understood as the candidate hidden state 9913, input the candidate hidden state 9913 and the result 9911 into the third product operation subunit 9914, obtain the third product result 9915, and input the second product result 999 and the third product result 9915 into the addition operation Subunit 9916, executes

operation, the obtained H can be understood as the updated implicit state diagram 9917, the updated implicit state diagram 9917 is input into the hidden state diagram replacement unit 9100, and the second diagram is replaced by the implicit state diagram replacement unit 9100 The implicit state diagram 960 output by the convolution operation unit 950, and the updated implicit state diagram 9917 is input into the bias parameter adjustment sub-unit 991, and the bias parameters are adjusted to obtain a neural network model that can improve the timing diagram. Bias parameter for the effect of extracting temporal relationship features of image sequences.

Next, the graph structure building module can receive the attention feature corresponding to the next face image, and divide the attention feature into 8 attention feature vectors along the second dimension, such as the channel dimension, to obtain the corresponding attention feature of the next face image. The attention feature vector, that is, the next node V, and then complete the next update of the hidden state diagram based on this node, until the attention features of each image in the image sequence are input into the neural network model of the sequence diagram After that, the obtained hidden state diagram can represent the temporal relationship characteristics of the whole image sequence.

Further, in order to verify the accuracy of the video detection model shown in Figures 7 to 9 in detecting fake face videos, FaceForensics++ (FF++), Celeb-DF v2 and Deepfake Detection Challenge (DFDC) datasets can be used to test forgery detection. The performance of face video.

Among them, the FF++ dataset can tamper with 1,000 real videos through four deep forgery algorithms, Deepfakes, Face2Face, FaceSwap and NeuralTextures. Each deep-forgery algorithm generates 1,000 forged videos and 4,000 forged videos, of which , the FF++ dataset can contain a High Quality (HQ) version of the dataset and a Low Quality (LQ) version of the dataset.

The Celeb-DF v2 dataset can use the improved Deepfake deepfake algorithm to tamper with 590 real celebrity videos and get 5639 fake videos.

The DFDC dataset is a dataset produced by Facebook in 2019. It processed 1131 real videos for deep forgery and obtained 4119 forged videos.

In this test, through the area under the curve (AUC) model evaluation index, the video detection model is tested for model accuracy and generalization ability respectively. The model accuracy test is used to evaluate the detection accuracy of the video detection model. The same data set is used as the training set and test set for testing, and the generalization ability test is used to evaluate the adaptability of the video detection model to fresh samples, and different data sets are used as the training set and test set for testing.

In the model accuracy test, the FF++ dataset can be used for testing, and the test results are shown in Table 3.

table 3

Among them, w/o(without) means to cancel the comparison method of a specific model, which is used to verify the validity of the x-related model. For example, the video detection model w/o 3D attention neural network model means to cancel the 3D attention in the video detection model The model after the neural network model. FF++(HQ) and FF++(LQ) represent the test results of the video detection model in the high-definition (HQ) version of the FF++ dataset and the low-definition (LQ) version of the FF++ dataset, respectively, and the percentage values represent the test results and the real results. The same probability as the detection accuracy of the corresponding detection method in the corresponding dataset. In the model accuracy test, the video detection model can achieve the highest detection accuracy, and the detection accuracy of the video detection model in the FF++(LQ) dataset is 18.19% higher than that of the basic MC3 method. Cancellation of the 3D attention neural network model or the sequence diagram neural network model in the video detection model will reduce the detection accuracy of the video detection model.

In the generalization ability test, the FF++ (HQ) data set can be used to train the corresponding video detection method, and the FF++ (HQ) data set, Celeb-DF data set and DFDC data set are used as test sets. The test results are shown in Table 4. shown.

Table 4

Among them, Table 4 has the same meaning as Table 3, and Celeb-DF v2 and DFDC represent the test results of the video detection model in the Celeb-DF v2 dataset and the DFDC dataset, respectively. In this generalization ability test, the video detection model can achieve the highest generalization performance, and the detection accuracy of the video detection model in the Celeb-DF dataset is 11.92% higher than that of the basic MC3 method. Cancellation of the 3D attention neural network model or the sequence diagram neural network model in the video detection model will reduce the detection accuracy of fake face video detection.

To sum up, after the accuracy test and the generalization ability test are performed on the detection accuracy of the video detection model in the embodiment of the present disclosure, it can be known from the test results that the video detection model provided by the embodiment of the present disclosure and the corresponding video detection method implemented The highest detection accuracy can be obtained among various video detection models for multiple data sets and their corresponding video detection methods. Therefore, this test result indicates that the video detection method provided by the embodiment of the present disclosure can improve the performance of fake face detection methods. Accuracy and generalization ability of video detection.

FIG. 10 is a schematic structural diagram of a video detection apparatus provided by an embodiment of the present disclosure.

In the embodiment of the present disclosure, the video detection apparatus may be provided in a computing device. The computing devices may include electronic devices, servers, etc., which are not limited herein. The device may include a mobile phone, a tablet computer, a desktop computer, a notebook computer, a vehicle terminal, a wearable electronic device, an all-in-one computer, a smart home device, or other device with computing functions, or a device simulated by a virtual machine or a simulator. The server can be an independent server or a cluster of multiple servers, including a server built locally and a server built in the cloud.

As shown in FIG. 10 , the video detection apparatus 1000 includes: an image acquisition module 1010 , a linear change module 1020 , a feature construction module 1030 and a probability calculation module 1040 .

The image acquisition module 1010 can be used to acquire an image sequence to be detected, where the image sequence includes at least two video frames in the same video.

The nonlinear change module 1020 can be configured to perform nonlinear transformation processing on the facial features of the images for each image in the image sequence, so as to obtain the attention features of multiple face regions corresponding to the images.

The feature construction module 1030 may be configured to construct a time series relationship feature between the multiple face regions corresponding to the image sequence based on the attention features of the multiple face regions corresponding to each image.

The probability calculation module 1040 may be configured to calculate the probability that the video is a video of a fake face based on the time sequence relationship feature.

In the embodiment of the present disclosure, after acquiring an image sequence to be detected that includes at least two video frames in the same video, nonlinear transformation processing can be performed on the facial features of each image in the image sequence to obtain each The attention features of multiple face regions corresponding to the image, and based on the attention features of multiple face regions corresponding to each image, the temporal relationship features between the multiple face regions corresponding to the image sequence are constructed, and then based on the temporal relationship feature, to calculate the probability that the video is a video of a fake face, and the probability can be used to determine whether the video is a video of a fake face, because in the embodiment of the present disclosure, it can be based on the corresponding face regions in the image sequence. When calculating the probability, the timing relationship between multiple areas of the face can be introduced when calculating the probability, and then the timing inconsistency of the face in the video can be detected, which makes the calculation result of the probability more accurate and more general. Therefore, the detection ability of fake face video is improved, and the detection accuracy of fake face video is improved.

In some embodiments of the present disclosure, the image acquisition module 1010 may include a video segmentation unit, a key frame extraction unit, and a key frame sorting unit.

The video segmentation unit may be used to divide the video into a plurality of video segments.

The key frame extraction unit can be used to extract key video frames of each video segment according to a preset frame extraction method.

The key frame sorting unit can be used to sort each key video frame according to the play time sequence to obtain an image sequence.

In some embodiments of the present disclosure, the nonlinear change module 1020 may include a feature extraction unit, a three-dimensional convolution operation unit, and an attention feature construction unit.

The feature extraction unit can be used to perform three-dimensional facial feature extraction processing on the image to obtain three-dimensional facial features corresponding to the image.

The three-dimensional convolution operation unit can be used to perform a three-dimensional convolution operation on the three-dimensional facial features corresponding to the image to obtain attention weight matrices of multiple face regions corresponding to the image.

The attention feature construction unit may be configured to generate attention features of multiple face regions corresponding to the image based on the three-dimensional face feature corresponding to the image and the attention weight matrix of the multiple face regions corresponding to the image.

In some embodiments of the present disclosure, the attention feature construction unit may include a feature matrix construction subunit, a summation subunit, and a pooling subunit.

The feature matrix construction subunit can be used to multiply the product of the three-dimensional facial features corresponding to the image and the attention weight matrices of the multiple face regions corresponding to the image as the attention feature matrices of the multiple face regions corresponding to the image.

The summation subunit may be used to sum the height dimension and the width dimension of the attention feature matrices of multiple face regions corresponding to the image, respectively, to obtain the summed attention feature matrix.

The pooling subunit can be used to perform global average pooling on the length dimension of the summed attention feature matrix to obtain the attention features of multiple face regions corresponding to the image.

In some embodiments of the present disclosure, the attention feature construction unit may include a weight matrix construction sub-unit.

The weight matrix construction sub-unit can be used to perform a three-dimensional convolution operation on the three-dimensional facial features corresponding to the image based on the three-dimensional attention neural network model, so as to obtain the attention weight matrices of multiple areas of the face corresponding to the image; The force neural network model includes three attention mechanism modules. Each attention mechanism module includes a 3D convolution layer, a 3D batch normalization layer and an activation layer. The loss function of the 3D attention neural network model includes time series continuity loss function and sparseness. Attention Contrastive Loss Function. The 3D attention neural network model is trained based on the temporal continuity loss function and the sparse attention contrast loss function.

In some embodiments of the present disclosure, the feature building module may include a graph-structured data building unit and a temporal relationship feature building unit.

The graph structure data construction unit can be configured to, for each image, construct graph structure data of multiple face regions corresponding to the image according to the attention features of the multiple face regions corresponding to the image.

The temporal relationship feature construction unit can be used to perform temporal relationship building processing on the graph structure data of multiple face regions corresponding to each image, so as to obtain temporal relationship features between multiple face regions corresponding to the image sequence.

In some embodiments of the present disclosure, the time sequence relationship feature construction unit may include a hidden state diagram construction subunit and a time sequence relationship feature acquisition subunit.

The hidden state diagram construction subunit can be used to construct a time sequence relationship on the graph structure data of multiple face regions corresponding to each image based on the time sequence diagram neural network model, and obtain the last hidden state diagram corresponding to the image sequence.

The time sequence relationship feature acquisition subunit can be used to use the last hidden state diagram as the time sequence relationship feature between multiple face regions corresponding to the image sequence.

It should be noted that, the video detection apparatus 1000 shown in FIG. 10 may perform various steps in the method embodiments shown in FIG. 1 to FIG. 5 , and implement each process and The effect will not be repeated here.

Embodiments of the present disclosure also provide a video detection device, the video detection device may include a processor and a memory, and the memory may be used to store executable instructions. The processor may be configured to read executable instructions from the memory and execute the executable instructions to implement the video detection method in the above-mentioned embodiments.

FIG. 11 shows a schematic structural diagram of a video detection device provided by an embodiment of the present disclosure. Referring specifically to FIG. 11 below, it shows a schematic structural diagram of a video detection device 1100 suitable for implementing the embodiments of the present disclosure.

In an embodiment of the present disclosure, the video detection device 1100 may be a computing device. The computing devices may include electronic devices, servers, etc., which are not limited herein. Electronic devices may include mobile phones, tablet computers, desktop computers, notebook computers, vehicle-mounted terminals, wearable electronic devices, all-in-one computers, smart home devices, and other devices with computing functions, and may also be devices simulated by virtual machines or simulators. The server can be an independent server or a cluster of multiple servers, including a server built locally and a server built in the cloud.

FIG. 11 is a schematic structural diagram of a video detection device according to an embodiment of the present disclosure. The computer device provided by the embodiment of the present disclosure can use the processing flow of the above method embodiment. As shown in FIG. 11 , the computer device 1100 includes: a memory 1110 , a processor 1120 , a computer program, and a communication interface 1130 ; wherein the computer program is stored in the memory 1110 and is configured to perform the video detection method as described above by the processor 1120.

It should be noted that the video detection device 1100 shown in FIG. 11 is only an example, and should not impose any limitations on the functions and scope of use of the embodiments of the present disclosure.

In addition, an embodiment of the present disclosure further provides a computer-readable storage medium on which a computer program is stored, and the computer program is executed by a processor to implement the video detection method described in the foregoing embodiments.

It should be noted that, in this document, relational terms such as "first" and "second" etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply these There is no such actual relationship or sequence between entities or operations. Moreover, the terms "comprising", "comprising" or any other variation thereof are intended to encompass a non-exclusive inclusion such that a process, method, article or device that includes a list of elements includes not only those elements, but also includes not explicitly listed or other elements inherent to such a process, method, article or apparatus. Without further limitation, an element qualified by the phrase "comprising a..." does not preclude the presence of additional identical elements in a process, method, article or apparatus that includes the element.

The above descriptions are only specific embodiments of the present disclosure, so that those skilled in the art can understand or implement the present disclosure. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the generic principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present disclosure. Therefore, the present disclosure is not intended to be limited to the embodiments described herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims (10)

1. a video detection method, is characterized in that, comprises: obtaining a sequence of images to be detected, the sequence of images comprising at least two video frames in the same video; For each image in the image sequence, non-linear transformation processing is performed on the facial features of the image to obtain the attention features of multiple regions of the face corresponding to the image; Based on the attention features of the multiple areas of the face corresponding to each of the images, construct the time series relationship features between the multiple areas of the face corresponding to the image sequence; Based on the time sequence relationship feature, a probability that the video is a video of a fake face is calculated. 2. The method according to claim 1, wherein the acquiring the image sequence to be detected comprises: dividing the video into a plurality of video segments; According to the preset frame extraction method, extract the key video frame of each described video segment; Sort each of the key video frames according to the play time sequence to obtain the image sequence. 3. The method according to claim 1, wherein the non-linear transformation processing is performed on the facial features of the image to obtain the attention features of multiple regions of the face corresponding to the image, comprising: performing three-dimensional facial feature extraction processing on the image to obtain three-dimensional facial features corresponding to the image; Performing a three-dimensional convolution operation on the three-dimensional facial features corresponding to the image to obtain the attention weight matrices of multiple areas of the face corresponding to the image; Based on the three-dimensional face feature corresponding to the image and the attention weight matrix of the multiple face regions corresponding to the image, the attention features of the multiple face regions corresponding to the image are generated. 4 . The method according to claim 3 , wherein, generating the corresponding three-dimensional facial features of the image and the attention weight matrix of multiple face regions corresponding to the image based on the three-dimensional facial features corresponding to the image. 5 . Attention features in multiple regions of the face, including: Taking the product of the three-dimensional facial features corresponding to the image and the attention weight matrices of the multiple face regions corresponding to the image as the attention feature matrices of the multiple face regions corresponding to the image; The height dimension and the width dimension of the attention feature matrices of the multiple regions of the face corresponding to the image are respectively summed to obtain the summed attention feature matrix; A global average pooling is performed on the length dimension of the summed attention feature matrix to obtain the attention features of multiple face regions corresponding to the image. 5. The method according to claim 3, wherein the three-dimensional convolution operation is performed on the three-dimensional facial features corresponding to the image to obtain attention weight matrices of multiple areas of the face corresponding to the image, include: Based on the three-dimensional attention neural network model, a three-dimensional convolution operation is performed on the three-dimensional facial features corresponding to the image to obtain attention weight matrices of multiple face regions corresponding to the image; wherein, the three-dimensional attention neural network The model includes three attention mechanism modules, each of which includes a three-dimensional convolution layer, a three-dimensional batch normalization layer and an activation layer, and the loss function of the three-dimensional attention neural network model includes a loss function based on time series continuity and Sparse attention contrast loss function. 6 . The method according to claim 1 , wherein the time sequence between the multiple face regions corresponding to the image sequence is constructed based on the attention features of the multiple face regions corresponding to each of the images. 7 . Relationship characteristics, including: For each of the images, according to the attention features of the multiple areas of the face corresponding to the image, construct the graph structure data of the multiple areas of the face corresponding to the image; The time sequence relationship construction process is performed on the graph structure data of the multiple face regions corresponding to each of the images, so as to obtain the time sequence relationship features between the multiple face regions corresponding to the image sequence. 7 . The method according to claim 6 , wherein the process of constructing a time series relationship is performed on the graph structure data of multiple face regions corresponding to each of the images, so as to obtain multiple faces corresponding to the image sequence. 8 . Timing relationship characteristics between regions, including: Based on the time sequence diagram neural network model, the time sequence relationship construction process is performed on the graph structure data of the multiple regions of the face corresponding to each of the images, and the last hidden state diagram corresponding to the image sequence is obtained; The last hidden state diagram is used as the time-series relationship feature among multiple face regions corresponding to the image sequence. 8. A video detection device, comprising: an image acquisition module for acquiring an image sequence to be detected, the image sequence comprising at least two video frames in the same video; a nonlinear change module, for performing nonlinear transformation processing on the facial features of the image for each image in the image sequence, to obtain the attention features of multiple regions of the face corresponding to the image; a feature building module, configured to construct the time series relationship feature between the multiple face regions corresponding to the image sequence based on the attention features of the multiple face regions corresponding to each of the images; A probability calculation module, configured to calculate the probability that the video is a video of a fake face based on the time sequence relationship feature. 9. A video detection device is characterized in that, comprising: processor; memory for storing executable instructions; Wherein, the processor is configured to read the executable instructions from the memory, and execute the executable instructions to implement the video detection method according to any one of the above claims 1-7. 10. A computer-readable storage medium, characterized in that the storage medium stores a computer program that, when the computer program is executed by a processor, causes the processor to implement the method described in any one of the preceding claims 1-7. The video detection method described above.

Images