CN112101217B - Person Re-identification Method Based on Semi-supervised Learning - Google Patents

Abstract

The invention discloses a pedestrian re-identification method based on semi-supervised learning, which comprises the following steps that S100 learns a projection matrix U epsilon R ^d×c to project an original d-dimensional feature space into a c-dimensional subspace, so that U ^TX∈R^c×N meets the following conditions in a new subspace: the Euclidean distance between pairs of samples from the same pedestrian is smaller, and the Euclidean distance between pairs of samples from different pedestrians is larger; samples from the same pedestrian are defined as homogeneous samples, and samples from different pedestrians are defined as heterogeneous samples; s200, projecting the new sample into a new subspace by adopting a projection matrix U epsilon R ^d×c to obtain a predicted sample sequence, wherein the predicted sample sequence is arranged according to the Euclidean distance between the new sample and the samples in the training sample set from small to large. The method fully utilizes the accurately marked labeled sample, and the positive sample is restrained by using the contrast loss function, and simultaneously, the negative sample pair can be fully utilized, so that the identification speed is high, and the identification accuracy is higher.

Description

Person Re-identification Method Based on Semi-supervised Learning

Technical Field

The present invention relates to the technical field of pedestrian re-identification, and in particular to a pedestrian re-identification method based on semi-supervised learning.

Background technique

Person re-identification, also known as pedestrian re-identification, is a technology that uses computer vision technology to determine whether a specific pedestrian exists in an image or video sequence. It is widely considered to be a sub-problem of image retrieval. Given a monitored pedestrian image, the image of the pedestrian is retrieved across devices. It aims to make up for the visual limitations of fixed cameras and can be combined with pedestrian detection and pedestrian tracking technologies. It can be widely used in intelligent video surveillance, intelligent security and other fields.

Although computer vision practitioners have proposed a large number of algorithms for pedestrian re-identification tasks from different perspectives in recent years, trying to continuously improve the recognition rate on public datasets, pedestrian re-identification is still an extremely challenging task due to some practical factors.

At present, semi-supervised learning methods are generally used to solve the pedestrian re-identification task. The process is roughly as follows: first, the unlabeled samples are automatically labeled; second, the labeled samples and the automatically labeled samples are uniformly trained, and then the model is optimized to make the model have better discrimination ability. However, there are two problems with the automatic labeling of unlabeled samples and the use of labeled samples:

① The idea of the method used for automatic labeling of unlabeled samples is to use the K-Nearest Neighbor (KNN) algorithm to label in the new space. This means that if the discriminative ability of the learned new space is not enough, the error after automatic labeling will be relatively large. When using data with large labeling errors for model training, it is very likely that the model will not have better generalization ability due to the increase in training samples, but will instead lead to the worse discriminative ability of the model as the training progresses.

② During training, only the positive sample pairs in the training set are constrained, and no attention is paid to the negative sample pairs, resulting in insufficient utilization of the training samples.

Summary of the invention

In view of the above problems existing in the prior art, the technical problem to be solved by the present invention is: the existing method of using semi-supervised learning to solve the pedestrian re-identification task has the problem that the automatic labeling error is easily affected by the new space obtained by learning and the training samples are not fully utilized.

In order to solve the above technical problems, the present invention adopts the following technical solution: a pedestrian re-identification method based on semi-supervised learning, comprising the following steps:

S100: Learn a projection matrix U∈R ^d×c to project the original d-dimensional feature space into a c-dimensional subspace, so that U ^T X∈R ^c×N satisfies in the new subspace: the Euclidean distance between sample pairs from the same pedestrian is smaller, and the Euclidean distance between sample pairs from different pedestrians is larger; samples from the same pedestrian are defined as samples of the same type, and samples from different pedestrians are defined as samples of different types;

S200: Project the new sample to the new subspace using the projection matrix U∈R ^d×c to obtain a prediction sample sequence, which is arranged in ascending order according to the Euclidean distance between the new sample and the samples in the training sample set.

Preferably, the method for learning the projection matrix U∈R ^d×c in S100 is specifically:

S110: Establish a training sample set, the training sample set includes a plurality of samples, the plurality of samples include labeled samples and unlabeled samples, and the labeled samples have the same sample label for the same pedestrian;

Let X = [ _XL , _XU ] ∈ ^{Rd × N} represent all training samples, where N is the number of images in the training set, d is the length of the feature vector, represents N _L labeled samples, Represents N _U unlabeled samples;

S120: Establish the objective function as follows:

Where L(U) is a regression function, is a weighted regression function, Ω(U) is a regularized constraint, and α,λ＞0 are balance coefficients;

S130: The loss function of the labeled sample is a contrast loss function: for the N _P sample pairs and/> If/> and/> Samples from the same pedestrian, then in the new projection space/> and/> The Euclidean distance d _n between them should be as small as possible, close to 0; otherwise, d _n should be at least greater than a pre-set threshold margin>0. If the above conditions are not met, losses will occur.

S140: Label the unlabeled samples. Use the K nearest neighbor method to label the unlabeled samples. The loss function for the unlabeled samples is:

If U ^T x _i and U ^T x _j satisfy K nearest neighbors and x _i and x _j are from different cameras, then take

Otherwise, _Wij = 0; (8);

After labeling the unlabeled samples, the labeled sample pairs are used to further constrain the existing subspace, and the constraint weight is the cosine distance between the two samples in the new projection space;

S150: Regularization term: Use the L2,1 norm to constrain the projection matrix U:

Ω(U)＝||U|| _2,1 (4).

Preferably, the labeled sample loss function of S130 is:

in:

Preferably, the sampling strategy for the _NP samples sampled in S130 is a sampling strategy for maximizing the top-k recognition rate, that is, for each image, all samples of its k nearest neighbors are sampled.

Preferably, the method of labeling the unlabeled samples using the K nearest neighbor method in S140 is:

The K nearest neighbors N(x,k) of sample x are defined as follows:

N(x,k)＝{x ₁ ,x ₂ ,...,x _k },|N(p,k)|＝k (5);

Where |·| represents the number of samples in the set, then the K nearest neighbors R(x,k) are defined as follows:

R(x,k)＝{ _xi | ₍ xi∈N(x,k))∧(x∈N( _xi ,k))} (6).

Compared with the prior art, the present invention has at least the following advantages:

(1) The present invention uses K nearest neighbors to make the automatic labeling results of unlabeled samples more reliable.

(2) Make full use of accurately annotated labeled samples. Use the contrast loss function commonly used in training deep neural networks to constrain positive samples while making full use of negative samples. It should be noted that the labeled sample loss function can be replaced by any loss used for recognition or classification.

(3) In order to facilitate the migration of models to deep models in the future, this paper uses an end-to-end training method, and the training strategy uses the stochastic gradient descent method. This paper proposes a batch generation strategy that maximizes the top-k recognition rate to solve the problems of slow convergence of paired training strategies under random batches and prevent model overfitting.

BRIEF DESCRIPTION OF THE DRAWINGS

Figure 1 K mutual nearest neighbor sampling strategy for person re-identification problem. First row: a searched image and its 10 nearest neighbor images, where P1-P4 are positive samples and N1-N6 are negative samples. Second row: every two columns are the 10 nearest neighbor images of the corresponding first row image. The thick rectangular box without chamfer and the thin rectangular box with chamfer represent the searched image and the positive sample image respectively.

In Figure 2, the negative sample closest to the image to be retrieved is the most difficult negative sample; the first positive sample that is just smaller than the most difficult negative sample is a moderate positive sample; the samples in the box are the sampling strategy of this paper.

Fig. 3 Appropriate positive sample sampling.

Detailed ways

The present invention will be further described in detail below in conjunction with the accompanying drawings.

Referring to Figure 1-3, the pedestrian re-identification method based on semi-supervised learning includes the following steps:

S100: Learn a projection matrix U∈Rd ^×c to project the original d-dimensional feature space into the c-dimensional subspace, so that ^{UTX∈Rc ×N} satisfies in the new subspace: the Euclidean distance between sample pairs from the same pedestrian is smaller, and the Euclidean distance between sample pairs from different pedestrians is larger; samples from the same pedestrian are defined as samples of the same type, and samples from different pedestrians are defined as samples of different types.

The method of learning the projection matrix U∈Rd ^×c is as follows:

S110: Establish a training sample set, the training sample set includes a plurality of samples, the plurality of samples include labeled samples and unlabeled samples, and the labeled samples have the same sample label for the same pedestrian;

Let X = [ _XL , _XU ] ∈ ^{Rd × N} represent all training samples, where N is the number of images in the training set, d is the length of the feature vector, represents N _L labeled samples, represents N _U unlabeled samples;

S120: Establish the objective function as follows:

Where L(U) is a regression function, the purpose of which is to make the labeled samples closer to each other in the new space mapped to the same label sample pair and farther away from each other. is a weighted regression function that can use unlabeled samples to improve the discriminative power of the model. Ω(U) is a regularized constraint that can select more discriminative features in the original feature space to avoid overfitting. α,λ＞0 are balance coefficients.

S130: Labeled sample loss function: The purpose of this constraint is to make full use of the label information of the labeled samples. In order to simultaneously utilize the positive and negative sample pair constraints, we use the training contrast loss function

in:

For the N _P sample pairs and/> If/> and/> Samples from the same pedestrian, then in the new projection space/> and/> The Euclidean distance d _n between them should be as small as possible, close to 0; otherwise, d _n should be at least greater than a pre-set threshold margin>0. If the above conditions are not met, losses will occur;

S140: Label the unlabeled samples. Use the K nearest neighbor method to label the unlabeled samples. The loss function for the unlabeled samples is:

In order to effectively utilize the discriminative information of unlabeled samples and reduce the adverse effects of incorrect labeling on the model, we use K mutual nearest neighbors instead of K nearest neighbors to label unlabeled samples, and only constrain positive sample pairs in this item. The specific loss function is as follows:

If U ^T x _i and U ^T x _j satisfy K nearest neighbors and x _i and x _j are from different cameras, then take

Otherwise, _Wij = 0 (8);

The significance of this item is that it is determined that the K nearest neighbor sample pairs in the learned discriminative subspace are very likely to come from the same pedestrian. Then, after labeling the unlabeled samples, the labeled sample pairs are used to further constrain the existing subspace, and the constraint weight is the cosine distance between the two samples in the new projection space.

S150: Regularization term: The purpose of adding the regularization term is to avoid overfitting and make the learned projection matrix more sparse. Here, we use the L2,1 norm to constrain the projection matrix U:

Ω(U)＝||U|| _2,1 (4).

As an improvement, the sampling strategy of the N _P samples sampled in S130 is a sampling strategy that maximizes the top-k recognition rate, that is, for each image, all samples of its k nearest neighbors are sampled. In this way, while avoiding overfitting, the discriminant information of the labeled samples can be maximized.

When using stochastic gradient descent for optimization, all samples need to be fed into the model in batches. All samples are randomly sampled, and a small number of categories are randomly selected each time, with two images for each category. When calculating the loss, all sample pairs that may be composed of all images in each batch are involved in the calculation. In this way, although many pairs of samples can be calculated at a time, due to the randomness of category sampling, the optimization direction of this sampling may not be the direction that can make the target drop fastest. After each optimization, the distance of all samples under the current model will be calculated. In order to make the target drop faster, only one pair of the most difficult negative samples under the current model is selected for each image, as shown in Figure 2.

It is worth noting that some positive sample pairs have too large intra-class differences due to drastic changes. If these samples are trained, the model is likely to overfit, as shown in Figure 3. In order to avoid this overfitting, a moderate positive sample (moderate positive sample) is sampled for each image. The sampling method is shown in Figure 2, that is, the first positive sample that is just smaller than the most difficult negative sample is sampled. In order to utilize as much information as possible from labeled samples, we propose a sampling strategy that maximizes the top-k recognition rate. As shown in Figure 2, for each image, all samples of its k nearest neighbors are sampled. In this way, while avoiding overfitting, the discriminant information of labeled samples can be maximized.

As an improvement, the method of labeling the unlabeled samples using the K nearest neighbor method in S140 is:

As shown in Figure 1, P1-P4 are four positive samples of the image to be retrieved, but they are not ranked in the top four nearest neighbor images. If the K nearest neighbor result is used directly, a large error will be introduced. However, it is worth noting that the image to be retrieved and the four positive samples are each other's K nearest neighbors, which we call K mutual nearest neighbors. If this method is used to annotate unlabeled data, the error introduction will be reduced to a certain extent.

The K nearest neighbors N(x,k) of sample x are defined as follows:

N(x,k)＝{x ₁ ,x ₂ ,...,x _k },|N(p,k)|＝k (5);

Where |·| represents the number of samples in the set, then the K nearest neighbors R(x,k) are defined as follows:

R(x,k)＝{ _xi | ₍ xi∈N(x,k))∧(x∈N( _xi ,k))} (6).

S200: Project the new sample to the new subspace using the projection matrix U∈R ^d×c to obtain a prediction sample sequence, which is arranged in ascending order according to the Euclidean distance between the new sample and the samples in the training sample set. The prediction sample that is ranked first indicates that the new sample is most likely to be the same person as the prediction sample.

Experiment and analysis:

Feature selection: In order to quickly verify the effectiveness of the proposed method, this paper uses LOMO features and GOG features which are commonly used in pedestrian re-identification tasks.

Parameter setting: Theano framework was used to implement the above algorithm, where the minimum margin was set to 0.5, the balance coefficients α and λ were set to 0.005 and 0.0001 respectively, the subspace dimension c was mapped to 512, and the batch size, learning rate and k were set to 32, 1 and 10 respectively.

VIPeR database test results and analysis

The VIPeR database is one of the most popular databases for person re-identification tasks. It contains 1264 images of 632 pedestrians captured by two cameras with 90° viewing angles and different lighting conditions. We randomly selected 316 pedestrians as the training set and the remaining 316 pedestrians as the test set, and conducted semi-supervised and fully supervised experimental settings respectively.

Semi-supervised experiment: For the semi-supervised setting, we randomly select 1/3 of the pedestrian images in the training set and erase the labels as unlabeled samples, and the remaining 2/3 of the pedestrian images as labeled samples. The experimental results are shown in Table 4.1. We compared the proposed method with SSCDL and DLLAP, and found that the proposed method has greatly improved the performance, especially after combining LOMO features with GOG features, the Rank-1 recognition rate can reach 47.5%.

Table 4.1 Comparison of recognition rates of semi-supervised learning methods on the VIPeR database

Rank 1 5 10 20 SSCDL 25.6 53.7 68.2 83.6 DLLAP 32.5 61.8 74.3 84.1 LOMO+Our 34.2 65.2 76.4 85.4 GOG+Our 42.4 73.4 83.9 91.0 LOMO+GOG+Our 47.5 78.3 86.9 92.1

Fully supervised experiments: We also conducted a fully supervised setting for the proposed method, that is, using the labels of all training samples. The experimental results are shown in Table 4.2. Compared with DLLAP and L1Graph, it can be found that the proposed method has a significant improvement when using GOG features and the combined use of LOMO and GOG features. Compared with the semi-supervised setting, it can be seen that when using LOMO and GOG features, only using the labels of 2/3 of the training samples can achieve a recognition rate of 47.5%, which is only 3% lower than the fully supervised case, which fully proves the effectiveness of the proposed method.

Table 4.2 Comparison of recognition rates under full supervision on the VIPeR database

Rank 1 5 10 20 DLLAP ^[41] 38.5 70.8 78.5 86.1 L1Graph ^[42] 41.5 - - - LOMO+Our 36.1 68.2 79.6 88.5 GOG+Our 48.6 77.1 87.3 92.9 LOMO+GOG+Our 50.5 79.6 88.8 94.3

The method of the present invention uses a contrastive loss function to make full use of the label information of labeled samples, and uses the K mutual nearest neighbor method to replace the K nearest neighbor method to annotate unlabeled samples. The experimental results on the public dataset VIPeR for person re-identification confirm the effectiveness of the method.

Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solution of the present invention can be modified or replaced by equivalents without departing from the purpose and scope of the technical solution of the present invention, which should be included in the scope of the claims of the present invention.

Claims (3)

1. A pedestrian re-identification method based on semi-supervised learning, characterized in that it includes the following steps: S100: Learn a projection matrix U∈R ^d×c to project the original d-dimensional feature space into a c-dimensional subspace, so that U ^T X∈R ^{c ×N} satisfies in the new subspace: the Euclidean distance between sample pairs from the same pedestrian is smaller, and the Euclidean distance between sample pairs from different pedestrians is larger; samples from the same pedestrian are defined as samples of the same type, and samples from different pedestrians are defined as samples of different types; The method for learning the projection matrix U∈R ^d×c in S100 is specifically: S110: Establish a training sample set, the training sample set includes a plurality of samples, the plurality of samples include labeled samples and unlabeled samples, and the labeled samples have the same sample label for the same pedestrian; Let X = [ _XL , _XU ] ∈ ^{Rd × N} represent all training samples, where N is the number of images in the training set, d is the length of the feature vector, represents N _L labeled samples, /> represents N _U unlabeled samples; S120: Establish the objective function as follows: Where L(U) is a regression function, is a weighted regression function, W(U) is a regularized constraint, and α,λ>0 are balance coefficients; S130: The loss function of the labeled sample is a contrast loss function: for the N _P sample pairs and/> If/> and/> Samples from the same pedestrian, then in the new projection space/> and/> The Euclidean distance d _n between them should be as small as possible, close to 0; otherwise, d _n should be at least greater than a pre-set threshold margin>0. If the above conditions are not met, losses will occur. The labeled sample loss function of S130 is: in: S140: Label the unlabeled samples. Use the K nearest neighbor method to label the unlabeled samples. The loss function for the unlabeled samples is: If U ^T x _i and U ^T x _j satisfy K nearest neighbors and x _i and x _j are from different cameras, then take Otherwise, _Wij = 0; (8); After labeling the unlabeled samples, the labeled samples are used to further constrain the existing subspace. The weight of the constraint is the cosine distance between the two samples in the new projection space. S150: Regularization term: Use the L2,1 norm to constrain the projection matrix U: W(U)＝||U|| _2,1 (4); S200: Project the new sample to the new subspace using the projection matrix U∈R ^d×c to obtain a prediction sample sequence, which is arranged in ascending order according to the Euclidean distance between the new sample and the samples in the training sample set. 2. The pedestrian re-identification method based on semi-supervised learning as described in claim 1 is characterized in that the sampling strategy of the N _P samples sampled in S130 is a sampling strategy that maximizes the top-k recognition rate, that is, for each image, all samples of its k nearest neighbors are sampled. 3. The pedestrian re-identification method based on semi-supervised learning according to claim 1, characterized in that the method of labeling the unlabeled samples using the K-nearest neighbor method in S140 is: The K nearest neighbors N(x,k) of sample x are defined as follows: N(x,k)＝{x ₁ ,x ₂ ,...,x _k },|N(p,k)|＝k (5); Where |·| represents the number of samples in the set, then the K nearest neighbors R(x,k) are defined as follows: R(x,k)＝{ _xi | ₍ xi∈N(x,k))∧(x∈N( _xi ,k))} (6).