CN112149538A - A Pedestrian Re-identification Method Based on Multi-task Learning - Google Patents

Abstract

The invention proposes a pedestrian re-identification method based on multi-task learning, including: acquiring cross-camera pedestrian images, constructing a pedestrian re-recognition training data set; constructing a multi-task learning network, which can combine attribute tasks and identity recognition The combination of tasks can jointly optimize all parameters and losses, so as to achieve the purpose of improving the accuracy of pedestrian re-identification; the attribute task and the recognition task respectively include verification and verification steps, classification and verification loss to optimize the distance of the sample, and the recognition loss is used for Constructing a large class space while validating the loss, the present invention optimizes the space by minimizing similar images and maximizing the distance between different images to improve recognition accuracy.

Description

A Pedestrian Re-identification Method Based on Multi-task Learning

technical field

The invention relates to a pedestrian re-identification method, in particular to a pedestrian re-identification method based on multi-task learning, and belongs to the technical field of deep learning.

Background technique

Pedestrian re-identification recognizes pedestrian images captured by multiple cameras without overlapping. The target pedestrian captured by one camera is used as the retrieval object to accurately identify pedestrians from images captured by other cameras. The robust representation features are extracted and an effective measurement model is used. The realization of recognition consists of two processes. Most of the research work focuses on two aspects of pedestrian appearance image representation and similarity measurement model. The main task of pedestrian re-identification is to find an effective representation of pedestrian images.

Before the advent of deep learning techniques, early research on person re-identification mainly focused on how to manually design better visual features and how to learn better similarity measures. In recent years, with the development of deep learning, deep learning technology has been widely used in pedestrian re-identification tasks. Different from traditional methods, deep learning methods can automatically extract better pedestrian image features and learn better similarity measures at the same time.

Due to the influence of various conditions, the recognition accuracy of person re-identification is still not high, and new technologies and algorithms with better robustness need to be proposed. The current research mainly focuses on image feature learning and similarity measurement. Single and single-level image representation features have different performances in different scenarios. The subspace technology with strong discriminative ability to perform projection transformation for image global or local structure information cannot identify images from different cameras well.

The specific reasons are analyzed as follows:

1) The representation features of different levels and parts represent different semantics, and the images are described by visual color features, structural features or depth features of different views, different levels, and different parts, reflecting different semantics. These description data, which generate Different methods carry different information, different sensitivities to various environmental changes, and have strong complementarity. The research results show that using a single type to represent features and a single level to represent features, the recognition effect is not as good as multiple features and multi-level features. The fusion effect is good. However, if a variety of features are used without distinction, the robustness is not good. It is necessary to study the feature fusion and utilization technology with strong environmental adaptability. The role of the scene is more robust, and the role of the features with poor robustness in the scene is reduced. For this reason, it is necessary to study and analyze the representation features suitable for different changes.

2) Deep convolutional neural networks show excellent performance in image classification. Using deep convolutional neural networks to learn pedestrian representation depth features has good discriminative ability and good robustness to noise. However, network training requires A large amount of labeled data is time-consuming and labor-intensive. It is necessary to study how to increase the training data of deep networks and improve the robustness of deep features.

3) Intra- and inter-class differences between samples: For datasets of person re-identification, intra-class differences and inter-class confusion are common problems. Intra-class difference refers to the diversity of agreeing pedestrians under different cameras, such as pose, angle, appearance, etc., and inter-class confusion refers to the similar appearance of different pedestrians under different cameras. The reason for these sample differences is the complexity introduced by the cross-camera, and it is difficult for the model to learn and match these features.

SUMMARY OF THE INVENTION

The purpose of the present invention is to provide a pedestrian re-identification method based on multi-task learning, which can only identify the overall outline of the pedestrian in the recognition task without considering the local information, and attribute verification problems that are not considered in the attribute task, Improve the accuracy of recognition.

The object of the present invention is achieved in this way: a pedestrian re-identification method based on multi-task learning, comprising the following steps:

Step 1) obtaining cross-camera pedestrian images, and constructing a pedestrian re-identification training data set, which contains a preset number of pedestrian images;

Step 2) Using ResNet-50 as the backbone network, the pre-training model based on ImageNet is pre-trained on the pedestrian re-identification data, and the pedestrian re-identification pre-training network model is obtained; the identity label and the attribute label are respectively used to re-identify the pedestrian. Perform fine-tuning to generate identity recognition network and attribute recognition network;

Step 3) Divide the identity recognition network into an identity classification task and an identity verification task, and divide the attribute recognition network into an attribute classification task and an attribute verification task. In the identity classification task, a loss function is used for identity prediction, and the feature vector output from the sample image is calculated. classification loss;

Step 4) In the identity verification task, given two pictures, input them into the pedestrian re-identification pre-training network model, obtain the global feature vectors of the two pictures, and calculate the loss function of the image pairs; select N image pairs, and calculate the total verification. loss;

Step 5) In the attribute classification task, use M loss functions for attribute prediction, and then calculate the loss of m attributes of M networks;

Step 6) The attribute verification task is to use the contrast loss function to calculate the total loss of the attribute verification task in order to judge whether the extracted attribute features conform to the pedestrian characteristics;

Step 7) The total loss generated by the four tasks is adjusted with balancing parameters α and β, where α balances the contributions between identity and attributes, and β balances the contributions between classification and verification.

As a further improvement of the present invention, step 1) specifically includes:

1.1) Use Gaussian mixture model to detect the foreground of pedestrian images;

1.2) According to the detection result of step 1.1, if there is a video frame with a moving foreground, use a pre-trained pedestrian detector to detect pedestrians, accurately locate the pedestrian position, and intercept the corresponding area image from the video frame as a pedestrian image;

1.3) According to the detection result of step 1.1, if the Gaussian mixture model does not detect the motion foreground, the pedestrian detector does not execute;

1.4) The same pedestrian extracted from different cameras is marked as the same class by manual labeling, and a number is assigned, and each type of pedestrian image is not less than the preset number of samples; different types of pedestrian images use different numbers. Representation; the attribute labels of each pedestrian are manually marked; the above sample collection process is iterated, and data collection can be stopped when the size of the training data set contains a preset number of pedestrian images.

As a further improvement of the present invention, step 2) specifically includes:

2.1) Obtain the deep convolutional network model pre-trained on the ImageNet dataset, and train it on the pedestrian re-identification data;

2.2) When pre-training the deep convolutional neural network model on the person re-identification data, the sample identity information and attribute information were used to fine-tune the deep convolutional network model respectively, and the ResNet50 network model pre-trained on the ImageNet dataset was added to the average. Pooling layer, insert a dropout layer after the average pooling layer.

As a further improvement of the present invention, step 3) specifically includes:

3.1) Obtain an image group from the dataset, and the image group contains identity tags and attribute tags;

3.2) Use the loss function to predict the identity, and obtain the predicted probability of the pedestrian global image according to the identity recognition classification network;

3.3) Define the loss function in the identity classification network according to the predicted probability.

As a further improvement of the present invention, step 4) specifically includes:

4.1) Determine two images I, J, and extract the corresponding high-level features f _a , f _b of the pictures through the identity recognition network;

4.2) According to the features extracted in step 4.1, use the contrast loss function to perform similarity estimation;

4.3) Calculate the validation loss for N image pairs in each batch according to the loss function of a set of images.

As a further improvement of the present invention, step 5) specifically includes:

5.1) Given a sample image I, it is assumed that there are m categories of attributes;

5.2) Use the loss function for attribute prediction, and obtain the predicted probability of pedestrian image attributes according to the attribute classification network;

5.3) Define the loss function in the attribute classification network according to the predicted probability.

As a further improvement of the present invention, step 7) specifically includes:

7.1 Combine the four losses generated in the above steps 3), 4), 5), and 6) to generate the final multi-task loss;

7.2 Using the parameter α to balance the loss between identity and attributes;

7.3 The loss between classification and validation is balanced using the parameter β.

Compared with the prior art, the present invention adopts the above technical solution, and has the following technical effects: improving the existing end-to-end learning framework of deep learning, combining attribute tasks with identity recognition tasks, and proposing a multi-task learning network, which enables pedestrians to learn more efficiently. Recognition is a task of finding the queried person from non-overlapping cameras, while the goal is to find the queried person; attribute recognition aims to predict the presence of a set of attributes from an image; attributes describe the details of a person , including gender, accessories, clothes color, etc.; the purpose of the recognition task is to judge whether it is the person we want to query; using attribute labels, the model can learn to classify pedestrians by explicitly focusing on some local semantic descriptions, which greatly simplifies The training of the model uses complementary cues in attribute labels to improve the performance of large-scale person re-identification; combines the pedestrian attribute task with the pedestrian recognition task, and proposes supplementary information from different perspectives by integrating multi-context information; attributes focus on A person's local information, while identity pays more attention to the overall outline and appearance, the recognition loss is used to construct a large class space, and the verification loss is used to optimize the space so that the distance between similar pictures is smaller, and the distance between different pictures is smaller. Larger; it is expected that the two tasks of identity and attribute are fully utilized by combining the two classification and verification methods, and the verification technology is embedded in the attribute, which makes up for the shortcomings of the attribute recognition model to a certain extent.

Description of drawings

FIG. 1 is an overview diagram of a multi-task learning method according to an embodiment of the present invention.

FIG. 2 is an architecture based on a multi-task learning network according to an embodiment of the present invention.

Detailed ways

The technical solutions of the present invention will be further described in detail below with reference to the accompanying drawings.

As shown in Figure 1, a multi-task learning-based pedestrian re-identification method includes the following steps.

Step 1) obtaining cross-camera pedestrian images, and constructing a pedestrian re-identification training data set, which contains a preset number of pedestrian images;

In this embodiment, the Gaussian mixture model is used to detect the foreground of the pedestrian image; according to the detection result of the Gaussian mixture model, if there is a video frame with a moving foreground, a pre-trained pedestrian detector is used to detect the pedestrian, accurately locate the pedestrian position, and from the The corresponding area image is intercepted from the video frame as the pedestrian image; if the Gaussian mixture model does not detect the moving foreground, the pedestrian detector will not be executed; the same pedestrian extracted from different cameras is marked as the same class by manual labeling, and assigned a number , each type of pedestrian image is not less than the preset number of samples; different types of pedestrian images are represented by different numbers; the attribute labels of each pedestrian are manually labeled; iterate the above sample collection process, when the training data set scale contains Data collection can be stopped at a preset number of pedestrian images.

Step 2) Using ResNet-50 as the backbone network, the pre-training model based on ImageNet is pre-trained on the pedestrian re-identification data, and the pedestrian re-identification pre-training network model is obtained; the identity label and the attribute label are respectively used to re-identify the pedestrian. Perform fine-tuning to generate identity recognition network and attribute recognition network;

In this embodiment, the networks described are all convolutional neural networks, including several convolutional units and pooling layers, wherein each convolutional unit consists of a batch normalization layer, a convolutional layer and a nonlinear The composition of the activation layer; in recent years, the convolutional network in deep learning has shown a good effect in extracting high-level image features, but the information extracted by the convolution kernel lacks sufficient target prior information; therefore, the attribute task and the recognition task are combined. Combined with attribute tasks to supplement identity recognition to improve the accuracy of pedestrian recognition;

Obtain the deep convolutional network model pre-trained on the ImageNet dataset and train it on the person re-identification data; when pre-training the deep convolutional neural network model on the person re-identification data, the sample identity information and The attribute information is used to fine-tune the deep convolutional network model separately. The ResNet50 network model pre-trained on the ImageNet dataset is added with an average pooling layer, and a dropout layer is inserted after the average pooling layer; the identity network and the attribute network are two subnet structures. Same, parameter sharing.

Step 3) The identity recognition network is divided into an identity classification task and an identity verification task. In the identity classification task, a loss function is used to perform identity prediction, and the feature vector output from the sample image is used to calculate the classification loss;

的子集,若训练图片为x,则输出z＝[z₁,z₂……z_k],每个身份标签k∈{1,2……k}的概率能够预测为P,损失函数定义为

其中

是预测概率，P_i是目标概率；除P_i＝k＝1外，所有p_i＝0,f为2048维的特征。In this embodiment, the multi-task learning network can input a single picture for learning, and output the global features of the picture; the global features extract the identity features through the identity network layer FC_IC layer, and use the loss function for identity prediction, which can well express the difference between samples. similarity. I={x _n , k _n , a _n } is a picture group, wherein the identity label is k _n and the attribute label a _n , each image x _n , contains a subset of the identity label k∈{1...k} and attributes Label

A subset of , if the training image is x, then the output z=[z ₁ , z ₂ ...... z _k ], the probability of each identity label k∈{1, 2 ...... k} can be predicted as P, the loss function is defined for

in

is the predicted probability, and P _i is the target probability; except for P _{i = k} =1, all p _i =0,f are 2048-dimensional features.

Step 4) In the identity verification task, given two pictures, input them into the pedestrian re-identification pre-training network model, obtain the global feature vectors of the two pictures, and calculate the loss function of the image pairs; select N image pairs, and calculate the total verification. loss;

In this example, the network forward propagation is used to solve the objective function relative to the parameters of each layer of the network, and stochastic gradient descent is used to update and learn the parameters of each layer;

In this example, given two images I, J, the high-level features f _a and f _b corresponding to the pictures are extracted through the FC_I layer of the identity recognition network, and the identity label corresponding to each picture is y. According to the extracted high-level features f _a , f _b , using a contrastive loss function for similarity estimation; the image pair loss function is defined as:

Euclidean distance is used to measure distance, represented by ||·|| ₁ ; ||·|| ₁ represents the first norm of the vector; |·| represents the absolute value, λ is the weighting parameter, which controls the strength of the regularizer , θ>0 is a margin threshold parameter; in the present invention, we set the parameters as λ=0.01 and θ=1024; if the two pictures are similar, then y=0; otherwise, y=1; the first penalty term Penalizes similar images mapped to different codes; the second penalty term penalizes different images mapped to similar image codes when the Euclidean distance is below a threshold θ; the last term is used to make the output match the required discrete values (+1/- 1), thus ensuring the convergence of the network; assuming that N pairs are randomly selected from a small batch, our ultimate goal is to minimize the total loss function, which can be defined as L _m (f _a ,f _b ,y ), which is the sum of the loss functions from 1 to N.

Step 5) In the attribute classification task, use M loss functions for attribute prediction, and then calculate the loss of m attributes of M networks;

的子集，我们用M个损失函数进行属性预测，属性i分配给属性类别j∈{1，2……m}的概率为P，则

属性分类的交叉熵成本函数可被定义为：

也就是说，总属性分类损失是每个属性分类损失之和。假设y_i,m是属性i的真实标签；当j＝y_i,m时，P_i,j＝1。In this example, similar to step 3), given I={x _n , k _n , a _n }, and the attribute label

A subset of , we use M loss functions for attribute prediction, and the probability that attribute i is assigned to attribute category j∈{1, 2...m} is P, then

The cross-entropy cost function for attribute classification can be defined as:

That is, the total attribute classification loss is the sum of each attribute classification loss. Suppose y _i,m is the true label of attribute i; when j=y _i,m , P _i,j =1.

Step 6) The attribute verification task is to use the contrast loss function to calculate the total loss of the attribute verification task in order to judge whether the extracted attribute features conform to the pedestrian characteristics;

的子集，假设属性i有j∈{1，2…m}种类别，我们能定义属性验证损失为：

In this example approach, the attribute verification task can be achieved by combining multiple different contrastive losses. Similar to step 4), given I={x _n , k _n , a _n }, and attribute labels

A subset of , assuming that attribute i has j ∈ {1, 2…m} categories, we can define the attribute validation loss as:

Step 7) The total loss generated by the four tasks is adjusted by balancing parameters α and β, where parameter α balances the contribution between identity and attributes, while β balances the contribution between classification and verification;

In this example, the final multi-task loss is generated by combining the four losses generated in the above steps 3), 4), 5), and 6); the multi-task learning network can be trained to simultaneously predict the identity and attributes; the final loss The function L is the sum of the four loss function values according to a certain weight, the parameter α balances the loss between identity and attributes, and β balances the loss between classification and verification.

The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited to this, any person familiar with the technology can understand the transformation or replacement that comes to mind within the technical scope disclosed by the present invention, All should be included within the scope of the present invention, therefore, the protection scope of the present invention should be subject to the protection scope of the claims.

Claims (7)

1. A pedestrian re-identification method based on multi-task learning is characterized in that: the method comprises the following steps:

step 1) acquiring a cross-camera pedestrian image, and constructing a pedestrian re-identification training data set, wherein the data set comprises a preset number of pedestrian images;

step 2) taking ResNet-50 as a backbone network, and pre-training a pre-training model based on ImageNet on pedestrian re-recognition data to obtain a pedestrian re-recognition pre-training network model; respectively carrying out fine adjustment on the pedestrian re-identification pre-training network by adopting an identity tag and an attribute tag to generate an identity identification network and an attribute identification network;

step 3) dividing the identity recognition network into an identity classification task and an identity verification task, dividing the attribute recognition network into an attribute classification task and an attribute verification task, performing identity prediction by using a loss function in the identity classification task, and calculating classification loss for a feature vector output by a sample picture;

step 4) in the authentication task, giving two pictures, inputting the two pictures into a pedestrian re-recognition pre-training network model, acquiring global feature vectors of the two pictures, and calculating a loss function of the picture pair; selecting N image pairs, and calculating the total verification loss;

step 5), in the attribute classification task, performing attribute prediction by using M loss functions, and calculating the loss of M attributes of M networks;

step 6), the attribute verification task is to use a comparison loss function to calculate the total loss of the attribute verification task in order to judge whether the extracted attribute features accord with the pedestrian features;

step 7) the total loss generated by the four tasks is adjusted by balancing parameters alpha and beta, wherein the parameter alpha balances the contribution between the identity and the attribute, and the beta balances the contribution between the classification and the verification.

2. The pedestrian re-identification method based on multitask learning according to claim 1, characterized by comprising the following steps: the step 1) specifically comprises the following steps:

1.1) detecting a pedestrian image foreground by adopting a Gaussian mixture model;

1.2) according to the detection result of the step 1.1, if a video frame with a moving foreground exists, detecting the pedestrian by adopting a pre-trained pedestrian detector, accurately positioning the position of the pedestrian, and intercepting a corresponding area image from the video frame as a pedestrian image;

1.3) according to the detection result of the step 1.1, if the Gaussian mixture model does not detect the motion foreground, the pedestrian detector is not executed;

1.4) marking the same pedestrian extracted from different cameras as the same class by adopting a manual marking mode, and giving a serial number, wherein the image of each class of pedestrian is not less than the preset number of samples; different numbers are used for representing different pedestrian images; manually labeling the attribute label of each pedestrian; the sample collection process described above is iterated, and data collection may be stopped when the training data set size contains a preset number of pedestrian images.

3. The pedestrian re-identification method based on multitask learning according to claim 1, characterized by comprising the following steps: the step 2) specifically comprises the following steps:

2.1) obtaining a deep convolution network model trained in advance on the ImageNet data set, and training the deep convolution network model on the pedestrian re-identification data;

2.2) when the deep convolutional neural network model is pre-trained on pedestrian re-identification data, the deep convolutional network model is respectively subjected to fine tuning by utilizing sample identity information and attribute information, an average pooling layer is added to a ResNet50 network model pre-trained on an ImageNet data set, and a dropout layer is inserted behind the average pooling layer.

4. The pedestrian re-identification method based on multitask learning according to claim 1, characterized by comprising the following steps: the step 3) specifically comprises the following steps:

3.1) acquiring an image group from the data set, wherein the image group comprises an identity tag and an attribute tag;

3.2) using a loss function to predict the identity, and acquiring the prediction probability of the global image of the pedestrian according to the identity recognition classification network;

3.3) defining a loss function in the identity classification network according to the prediction probability.

5. The pedestrian re-identification method based on multitask learning according to claim 1, characterized by comprising the following steps: the step 4) specifically comprises the following steps:

4.1) determining two images I, J, extracting the corresponding advanced features f of the images through an identity recognition network_a,f_b；

4.2) according to the characteristics extracted in the step 4.1, carrying out similarity estimation by using a contrast loss function;

4.3) calculating the verification loss of N image pairs in each batch according to the loss function of a group of images.

6. The pedestrian re-identification method based on multitask learning according to claim 1, characterized by comprising the following steps: the step 5) specifically comprises the following steps:

5.1) given a sample image I, assume that there are m classes of attributes;

5.2) performing attribute prediction by using a loss function, and acquiring the prediction probability of the pedestrian image attribute according to an attribute classification network;

5.3) defining a loss function in the attribute classification network according to the prediction probability.

7. The pedestrian re-identification method based on multitask learning according to claim 1, characterized by comprising the following steps: step 7) specifically comprises:

7.1 combining the four losses generated in the step 3), the step 4), the step 5) and the step 6) to generate a final multitasking loss;

7.2 the loss between identity and attribute is balanced using parameter α;

7.3 the loss between classification and validation is balanced using the parameter β.

Images