Background
Vision is the largest source of information for humans to obtain outside information. With the development of electronic technology, various visual sensors are applied in real life, and a large number of images are generated to record various information, so that it becomes impossible to simply rely on human to find required information from a large number of pictures. Therefore, it is desirable to extract information in an image by a computer. In computer vision, people need to model related information transmitted in images, and feature learning is to learn vectors to accurately and efficiently express information in the images. The goal of metric learning is to learn a distance to accurately and robustly characterize the correlation between features.
Metric learning can be widely applied to various visual applications, such as image retrieval, face recognition, pedestrian re-recognition, target tracking, object recognition, blood relationship verification and the like. Large-scale monitoring systems not only play a very large role in the criminal investigation field, but are also increasingly used in security maintenance of enterprises, schools, and the like. In order to extract effective information from the surveillance video, it is often necessary to analyze the people therein. The monitoring system is generally composed of a plurality of cameras, and therefore cooperation among the plurality of cameras is an important issue. The similarity between two pedestrians can be measured through the learned measure, and then the positions of the same person at different times can be obtained. In addition, the face recognition can be carried out on people entering the monitoring system, the similarity comparison is carried out between the learned measurement and the blacklist in the database, and dangerous people can be found in time.
Training of the depth metric learning method typically includes two phases, data sampling and network updating. In the data sampling phase, a small batch is first sampled, either randomly or in some pattern, from the entire training data set and then organized into a structure. In the network updating stage, the loss of the small batch is calculated according to the definition of the optimization target, and then the network is updated by using a back propagation algorithm.
The current research on the aspect of depth metric learning is divided into two directions, wherein one direction is improvement on a network optimization target, including a sample structure based on which the optimization target is selected and direct improvement on a loss function. For example, Wang et al consider the third order geometric information in the triplet, optimize the distance relationship between samples by limiting the angle at the negative sample, avoid using the absolute distance between samples, achieve scale invariance, and improve the robustness of the optimization target with respect to feature changes. Movshovit et al propose a triple loss function based on proxy points, which enables one proxy point to represent information of a plurality of sample points by reasonably selecting and continuously updating the proxy points, so that the loss function based on several proxy points can limit the relationship among a plurality of sample points, utilize the overall information of the sample, and avoid sampling all the sample points. Another direction is to improve the data sampling strategy, including how to construct small batches from a large number of samples in the training set, and how to sample useful samples for training from the small batches. For example, Sohn proposes a small lot organization called N-pair, which can more efficiently use all sample information in a small lot. Song et al propose to expand the distance vector of the sample pairs in the small lot to a distance matrix consisting of the distances between every two samples, which can make more full use of the correlation between the samples, thereby improving the efficiency of using the small lot.
While many of the current sampling strategies have proven effective in experimentation, there are two problems with them. The first problem is that they are all strategies pre-designed according to a priori knowledge, but at present there is no good standard to determine what sampling strategy is optimal; the second problem is that it is fixed during the training process of the metrology network, and thus cannot adaptively sample the samples that are most helpful for the metrology network training. The development of meta-learning in recent years makes it possible to train a sampler by means of meta-learning.
Disclosure of Invention
The present invention is directed to solving, at least to some extent, one of the technical problems in the related art.
Therefore, an object of the present invention is to provide a depth metric learning method based on adaptive sampling, which can improve the performance of the final learned metric and improve the generalization capability of the learned metric, thereby improving the performance of the depth metric learning method.
Another objective of the present invention is to provide an adaptive sampling-based depth metric learning system.
In order to achieve the above object, the present invention provides a depth metric learning method based on adaptive sampling, including the following steps: step S1, sampling by a sampler to obtain a sampling sample; step S2, updating the measurement network according to the sampling sample; step S3, training the sampler through the updated performance of the measurement network under the verification set; step S4, training the measurement network through the trained sampler; and step S5, obtaining the representation of the test sample in the measurement space through the trained measurement network according to the sample picture.
According to the depth measurement learning method based on the self-adaptive sampling, disclosed by the embodiment of the invention, a learnable sampler can interact with a measurement network, so that the performance of finally learned measurement can be improved; the sampler is trained by taking the updated performance of the metric on the verification set as a meta-objective function in a meta-learning mode, so that the generalization capability of the learned metric is improved; meanwhile, the depth metric learning method can be integrated into a generator module in most depth metric learning models at present, and the performance of the depth metric learning method is improved.
In addition, the depth metric learning method based on adaptive sampling according to the above embodiment of the present invention may further have the following additional technical features:
further, in an embodiment of the present invention, the step S1 includes: and sampling samples in different orders by adopting corresponding sampling strategies according to the sample input sequence, and updating the sampler in the training process of the measurement network.
Further, in an embodiment of the present invention, the update formula of the metric network is:
wherein, θ represents the parameter of the metric network, α is the training learning rate thereof, T represents a tuple, { T } tr represents all tuples in a small batch of training sets, s is the sampler, Φ is the parameter thereof, and L is the training loss function of the metric network.
Further, in an embodiment of the present invention, the training formula of the sampler is:
where φ represents the parameters of the sampler, β is its training learning rate, T' represents a tuple, { T } va represents all tuples in a small batch validation set, and L is the training loss function of the metric network.
Further, in an embodiment of the present invention, the training formula of the metric network is:
where θ represents the parameters of the metric network, α is its training learning rate, T represents a tuple, { T } tr represents all tuples in a small batch of training sets, s is the sampler, φ*For it after renewalL is a training loss function of the metric network.
In order to achieve the above object, another aspect of the present invention provides an adaptive sampling-based depth metric learning system, including: the sampling module is used for obtaining a sampling sample by sampling through the sampler; an update module for updating the metric network according to the sampling samples; a first training module, configured to train the sampler through the updated performance of the metric network in a validation set; a second training module for training the metric network through the trained sampler; and the acquisition module is used for acquiring the representation of the test sample in the measurement space through the trained measurement network according to the sample picture.
According to the depth measurement learning system based on the self-adaptive sampling, the learnable sampler can interact with the measurement network, and the performance of the final learned measurement can be improved; the sampler is trained by taking the updated performance of the metric on the verification set as a meta-objective function in a meta-learning mode, so that the generalization capability of the learned metric is improved; meanwhile, the depth metric learning method can be integrated into a generator module in most depth metric learning models at present, and the performance of the depth metric learning method is improved.
In addition, the depth metric learning system based on adaptive sampling according to the above embodiment of the present invention may also have the following additional technical features:
further, in an embodiment of the present invention, the sampling module is further configured to sample samples in different orders according to the sample input order by using corresponding sampling strategies, and update the sampler in a training process of the metric network.
Further, in an embodiment of the present invention, the update formula of the metric network is:
wherein, θ represents the parameter of the metric network, α is the training learning rate thereof, T represents a tuple, { T } tr represents all tuples in a small batch of training sets, s is the sampler, Φ is the parameter thereof, and L is the training loss function of the metric network.
Further, in an embodiment of the present invention, the training formula of the sampler is:
where φ represents the parameters of the sampler, β is its training learning rate, T' represents a tuple, { T } va represents all tuples in a small batch validation set, and L is the training loss function of the metric network.
Further, in an embodiment of the present invention, the training formula of the metric network is:
where θ represents the parameters of the metric network, α is its training learning rate, T represents a tuple, { T } tr represents all tuples in a small batch of training sets, s is the sampler, φ*For its parameters after updating, L is the training loss function of the metric network.
Additional aspects and advantages of the invention will be set forth in part in the description which follows and, in part, will be obvious from the description, or may be learned by practice of the invention.
Detailed Description
Reference will now be made in detail to embodiments of the present invention, examples of which are illustrated in the accompanying drawings, wherein like or similar reference numerals refer to the same or similar elements or elements having the same or similar function throughout. The embodiments described below with reference to the drawings are illustrative and intended to be illustrative of the invention and are not to be construed as limiting the invention.
The following describes a depth metric learning method and system based on adaptive sampling proposed in an embodiment of the present invention with reference to the drawings, and first, a depth metric learning method based on adaptive sampling proposed in an embodiment of the present invention will be described with reference to the drawings.
Fig. 1 is a flowchart of a depth metric learning method based on adaptive sampling according to an embodiment of the present invention.
As shown in fig. 1, the depth metric learning method based on adaptive sampling includes the following steps:
step S1, obtaining a sample by sampling with a sampler.
Further, in an embodiment of the present invention, the step S1 includes: and sampling samples in different orders by adopting corresponding sampling strategies according to the sample input sequence, and updating the sampler in the training process of the measurement network.
It should be noted that, the embodiment of the present invention uses a learnable sampler to perform adaptive sampling, and uses a meta-learning method to train the sampler.
Specifically, as shown in fig. 2, for a small batch, the samples are divided into a training set and a testing set. For the samples in the training set, a series of weights are obtained by sequentially passing the samples through the sampler, and the loss function of the small batch is calculated by weighting the corresponding samples by the weights:
firstly, because different sample input sequences have an influence on the training of the network, the sampler needs to use different strategies to sample samples in different sequences; secondly, the sampler needs to update itself continuously in the training process of the measurement network, and a better effect is achieved through interaction between the sampler and the measurement network. Thus, the present invention uses a learnable long-short term memory model as a sampler that outputs different results for different input sequences.
Step S2, updating the metric network according to the sample.
Further, in one embodiment of the present invention, the update formula of the metric network is:
wherein, θ represents the parameter of the metric network, α is the training learning rate thereof, T represents a tuple, { T } tr represents all tuples in a small batch of training sets, s is the sampler, Φ is the parameter thereof, and L is the training loss function of the metric network.
That is, on the training set, a one-step random gradient update is performed on the metric network using a weighted loss function.
Step S3, training the sampler through the updated performance of the metric network under the validation set.
Further, in one embodiment of the present invention, the training formula of the sampler is:
where φ represents the parameters of the sampler, β is its training learning rate, T' represents a tuple, { T } va represents all tuples in a small batch validation set, and L is the training loss function of the metric network.
That is, after a metric update, embodiments of the present invention evaluate it for performance on a validation set, specifically using an unweighted loss function to measure its performance. Since the updated metrics are affected by the sampler, embodiments of the invention may train the sampler using the loss function on the validation set:
it is noted that training on the training set and testing on the validation set this process simulates training and testing of the network in the usual sense, so maximizing test performance can improve the generalization ability of the measured network.
Step S4, training the metric network by the trained sampler.
Further, after the sampler is updated, the embodiment of the present invention trains the original metric network by using the sampler, wherein the training formula of the metric network is as follows:
in the formula, theta represents the parameter of the measurement network, alpha is the training learning rate of the measurement network, T represents a multi-tuple, { T } tr represents all the multi-tuples in a small-batch training set, s is a sampler, phi*For its parameters after updating, L is the training loss function of the metric network.
And step S5, obtaining the representation of the test sample in the measurement space through the trained measurement network according to the sample picture.
In other words, after the entire network is trained, the representation of the test sample in the metric space is directly obtained from the sample picture through the metric network without using a sampler, which is only used during the training process of the network.
In summary, in the embodiments of the present invention, a pre-designed fixed sampling method is mostly used in the current depth metric learning, and the pre-designed fixed sampling method cannot be adjusted along with the training process, so that the sampling method cannot be optimized. Therefore, the embodiment of the invention uses a long-short term memory model as a learnable sampler and trains the sampler by means of meta-learning. First, the samples in the small lot are divided into a training set and a verification set, and for each sample in the training set, a weight is generated using a sampler. Second, the weighted loss function is used to train the metric model and tested on the validation set to update the sampler. Finally, the metric model is trained using the updated samplers. After the network training is finished, a sampler is not needed for the test sample, so that no additional calculation process is introduced in the practical application.
Stated another way, firstly, a depth network is used for a picture to obtain a vector representation of the picture in a metric space; the sampler then samples each sample with a weight using a long short term memory model (LSTM) and calculates a weighted loss function; updating the metric network using the weighted loss function and training the sampler by its behavior on the validation set; finally, optimizing the measurement network by using the trained sampler; and updating the measurement network and the sampler network in each small batch, wherein the measurement network and the sampler network interact and influence with each other in the training process, and finally learning a measurement with accuracy and strong generalization capability.
According to the depth measurement learning method based on the self-adaptive sampling provided by the embodiment of the invention, a learnable sampler can interact with a measurement network, so that the performance of finally learned measurement can be improved; the sampler is trained by taking the updated performance of the metric on the verification set as a meta-objective function in a meta-learning mode, so that the generalization capability of the learned metric is improved; meanwhile, the depth metric learning method can be integrated into a generator module in most depth metric learning models at present, and the performance of the depth metric learning method is improved.
Next, a depth metric learning system based on adaptive sampling proposed according to an embodiment of the present invention is described with reference to the drawings.
Fig. 3 is a structural diagram of a depth metric learning system based on adaptive sampling according to an embodiment of the present invention.
As shown in fig. 3, the adaptive sampling-based depth metric learning system 10 includes: a sampling module 100, an update module 200, a first training module 300, a second training module 400, and an acquisition module 500.
The sampling module 100 is configured to obtain a sample by sampling with a sampler.
Further, in an embodiment of the present invention, the sampling module 100 is further configured to sample samples in different orders according to the sample input order by using corresponding sampling strategies, and update the sampler during the training process of the metrology network.
The update module 200 is configured to update the metric network based on the sample.
Further, in one embodiment of the present invention, the update formula of the metric network is:
wherein, θ represents the parameter of the metric network, α is the training learning rate thereof, T represents a tuple, { T } tr represents all tuples in a small batch of training sets, s is the sampler, Φ is the parameter thereof, and L is the training loss function of the metric network.
The first training module 300 is used to train the sampler through the updated metric network performance under the validation set.
Further, in one embodiment of the present invention, the training formula of the sampler is:
where φ represents the parameters of the sampler, β is its training learning rate, T' represents a tuple, { T } va represents all tuples in a small batch validation set, and L is the training loss function of the metric network.
The second training module 400 is used to train the metric network through the trained sampler.
Wherein, the training formula of the measurement network is as follows:
in the formula, theta represents the parameter of the measurement network, alpha is the training learning rate of the measurement network, T represents a multi-tuple, { T } tr represents all the multi-tuples in a small-batch training set, s is a sampler, phi*For its parameters after updating, L is the training loss function of the metric network.
The obtaining module 500 is configured to obtain a representation of the test sample in the metric space through the trained metric network according to the sample picture.
It should be noted that the foregoing explanation of the depth metric learning method based on adaptive sampling is also applicable to the system, and is not repeated here.
According to the depth measurement learning system based on the self-adaptive sampling provided by the embodiment of the invention, a learnable sampler can interact with a measurement network, so that the performance of finally learned measurement can be improved; the sampler is trained by taking the updated performance of the metric on the verification set as a meta-objective function in a meta-learning mode, so that the generalization capability of the learned metric is improved; meanwhile, the depth metric learning method can be integrated into a generator module in most depth metric learning models at present, and the performance of the depth metric learning method is improved. Furthermore, the terms "first", "second" and "first" are used for descriptive purposes only and are not to be construed as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one such feature. In the description of the present invention, "a plurality" means at least two, e.g., two, three, etc., unless specifically limited otherwise.
In the present invention, unless otherwise expressly stated or limited, the terms "mounted," "connected," "secured," and the like are to be construed broadly and can, for example, be fixedly connected, detachably connected, or integrally formed; can be mechanically or electrically connected; they may be directly connected or indirectly connected through intervening media, or they may be connected internally or in any other suitable relationship, unless expressly stated otherwise. The specific meanings of the above terms in the present invention can be understood by those skilled in the art according to specific situations.
In the present invention, unless otherwise expressly stated or limited, the first feature "on" or "under" the second feature may be directly contacting the first and second features or indirectly contacting the first and second features through an intermediate. Also, a first feature "on," "over," and "above" a second feature may be directly or diagonally above the second feature, or may simply indicate that the first feature is at a higher level than the second feature. A first feature being "under," "below," and "beneath" a second feature may be directly under or obliquely under the first feature, or may simply mean that the first feature is at a lesser elevation than the second feature.
In the description herein, references to the description of the term "one embodiment," "some embodiments," "an example," "a specific example," or "some examples," etc., mean that a particular feature, structure, material, or characteristic described in connection with the embodiment or example is included in at least one embodiment or example of the invention. In this specification, the schematic representations of the terms used above are not necessarily intended to refer to the same embodiment or example. Furthermore, the particular features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples. Furthermore, various embodiments or examples and features of different embodiments or examples described in this specification can be combined and combined by one skilled in the art without contradiction.
Although embodiments of the present invention have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present invention, and that variations, modifications, substitutions and alterations can be made to the above embodiments by those of ordinary skill in the art within the scope of the present invention.