CN110175615A - The adaptive visual position recognition methods in model training method, domain and device - Google Patents

Abstract

The invention discloses a model training method, a domain-adaptive visual position recognition method and a device, belonging to the technical field of computer vision, including: establishing an image feature extraction model based on a deep neural network; constructing a training set according to a standard data set; Each training sample includes a target image and its positive samples and s negative samples; the image feature extraction model is trained using the training set; in the image feature extraction model, the feature extraction network includes a plurality of cascaded first networks; the first network It is composed of one or more second networks and a maximum pooling layer connected in sequence, and the maximum pooling layer is used for feature selection; the second network includes sequentially connected convolution layers for feature extraction; batch normalization layer for Perform zero-mean normalization processing; the activation function layer is used for activation processing; the local feature aggregation network is used to aggregate local features to obtain the feature vector of the image. The invention can improve the robustness of visual position recognition.

Description

Model training method, domain adaptive visual position recognition method and device

technical field

The invention belongs to the technical field of computer vision, and more specifically relates to a model training method, a domain adaptive visual position recognition method and a device.

Background technique

Visual location recognition specifically refers to extracting features from an image, and then identifying the geographic location of the image based on the extracted image features. With the development of autonomous driving, the increasing demand for autonomously navigating mobile robots, and the increasing popularity of virtual reality and augmented reality, the research on visual position recognition is widely used in the field of computer vision, the robot community and other related fields. aroused widespread concern.

In the early period of computer vision research, image features were mainly extracted by artificially designed methods of extracting image feature points, such as scale-invariant feature transform (SIFT) feature points. The design of the extracted features is very dependent on experience, and some experts and scholars even took decades to design a better feature, and these hand-designed algorithms for extracting image feature points are in sharp lighting changes (such as day to night) and The effect is very poor when the scene changes (pedestrians and vehicles in the scene are edited), and the performance of visual position recognition methods that rely on these features, such as the visual bag of words model (V-BOW), will also drop sharply. In recent years, with the rise of deep learning and widely used in object recognition, object detection, object tracking, semantic segmentation and other fields, some visual location recognition methods based on deep learning have been proposed. For example, Convolutional Neural Network-based Place Recognition, which uses deep convolutional neural networks to extract image features, because deep convolutional neural networks can be trained end-to-end according to specific tasks, the extracted Image features are more robust. Another example is the CNN architecture NetVLAD (NetVLAD: CNN architecture for weakly supervised place recognition) used for weakly supervised place recognition. This method takes advantage of the traditional local feature aggregation (VLAD) method to effectively aggregate the local features of the image to obtain a compact The image expresses the feature vector, and this method also makes the image features extracted by the deep neural network more robust.

Compared with the traditional visual position recognition method based on manually designed image feature points, the image features extracted by the visual position recognition method based on deep neural network are more robust, and the visual position recognition is more accurate. However, the deep neural network needs to be trained before use, and due to the influence of viewing angle, illumination and other factors, the feature distribution of the image used for training is often quite different from the feature distribution of the actual image to be recognized. In this case, The accuracy of visual place recognition is not guaranteed. In general, the robustness of existing visual place recognition methods is low.

Contents of the invention

Aiming at the defects and improvement needs of the prior art, the present invention provides a model training method, a domain-adaptive visual position recognition method and device, the purpose of which is to improve the robustness of visual position recognition.

In order to achieve the above object, according to the first aspect of the present invention, a kind of image feature extraction model training method is provided, comprising:

(1) Establish an image feature extraction model based on a deep neural network to obtain the feature vector of the image;

The image feature extraction model includes a feature extraction network and a local feature aggregation network;

The feature extraction network includes a plurality of cascaded first networks; the first network is sequentially connected by one or more second networks and a maximum pooling layer, and the maximum pooling layer is used for the output of the previous second network. Image feature selection; the second network includes sequentially connected convolutional layers, batch normalization layers, and activation function layers. The convolutional layer is used to extract features from the image, and the batch normalization layer is used to zero-mean the output image of the convolutional layer. Normalization processing, the activation function layer is used to activate the image output by the batch normalization layer;

The local feature aggregation network is used to aggregate all the local features in the image output by the feature extraction network to obtain the feature vector of the image;

(2) In the standard data set, obtain positive samples and s negative samples of each target image, so as to form a training sample by a target image and its positive samples and negative samples, thereby obtaining a training set composed of all training samples;

The positive sample of the target image is the image closest to its feature distance among the adjacent images, and the position distance d between the target image and its adjacent image satisfies T _NL ≤ d<T _NH ; the position distance between the target image and its negative sample satisfies d≥T _F ;

(3) Utilize training set to train image feature extraction model, thereby obtain each model parameter;

Among them, the position information of each image in the standard data set is known, and the target image is multiple images pre-screened in the standard data set; the feature distance between images is the distance between the feature vectors of the image; T _NL , T _NH and T _F Both are preset thresholds, 0<T _NL <T _NH , T _NH ≤T _F ; s≥1.

In the image feature extraction model training method described above, in the image feature extraction model established, each convolution used for feature extraction is followed by a batch normalization layer (Batch Normalization) to carry out zero-mean normalization on the image output by the convolution layer In this way, while speeding up the model training, the image features extracted by the image feature extraction model have a similar distribution, thereby effectively avoiding the poor model training effect due to the large difference in the feature distribution of the images in the training set , which can further improve the problem of low robustness of visual position recognition when the distribution of image features is quite different.

Further, the local feature aggregation network includes: dimensionality reduction convolution layer, soft-max layer, aggregation layer, internal normalization layer and overall normalization layer;

The dimensionality reduction convolutional layer is a convolutional layer, which is used to reduce the latitude of the image to be aggregated to be equal to the number of preset cluster centers, so that each channel of the image to be aggregated represents local features and each The weight of the difference between cluster centers;

The soft-max layer is used to normalize the weight of the difference between local features and each cluster center;

The aggregation layer is used to aggregate VLAD (vector of locally aggregated descriptors) vectors according to local features, cluster centers, and normalized weights; VLAD vectors are composed of N D-dimensional vectors, N is the number of cluster centers, and D is the dimension of the cluster center;

The internal normalization layer is used to normalize the vectors of each D dimension in the VLAD vector, so that the distribution of the vectors of each D dimension is in the same order of magnitude;

The overall normalization layer is used to concatenate the D-dimensional vectors processed by the internal normalization layer into a column vector, and normalize the column vector so that each local feature of the image to be aggregated is distributed in the same Order of magnitude; thus, the convergence speed of the neural network model and the accuracy of the network model can be improved;

Among them, the image to be aggregated is the image output by the feature extraction network.

Further, s>1; the training accuracy of the model can be improved by selecting multiple negative samples, so that the image feature vector obtained by the above image feature extraction model is used for visual position recognition, which has high robustness.

Further, in step (3), when using the training set to train the image feature extraction model, the loss function used is:

Among them, n is the total number of training samples, k is the number of training samples, i is the number of negative samples, q _k , p _k and n _ki represent the target image, positive sample and i-th negative sample in the kth training sample respectively, Indicates the feature distance between the target image q _k and its positive sample p _k , is the feature distance between the target image q _k and its negative sample n _ki , m is a predefined hyperparameter, max means taking the maximum value, and min means taking the minimum value;

The above loss function, based on the idea of triplet loss, minimizes the feature distance between the target image and the positive sample and maximizes the feature distance to the negative sample through training; This item selects the negative sample with the largest loss, so that based on the idea of mining difficult examples, more attention is paid to the negative samples that are difficult to distinguish during the model training process, and then when using the above image feature extraction model for visual position recognition, Avoid the interference of negative samples similar to the image to be recognized.

According to the second aspect of the present invention, there is also provided a domain-adaptive visual position recognition method based on the image feature extraction model training method provided in the first aspect of the present invention, including:

Determine the target domain to which the image to be recognized belongs, and obtain multiple images at different positions in the target domain, and use the acquired image and the image to be recognized as the image to be retrieved;

Taking the image to be retrieved as input, use the image feature extraction model to obtain the feature vector of each image to be retrieved; when obtaining the image feature vector, for each convolutional layer, count all the images to be retrieved after passing through the convolutional layer, the obtained features The mean and standard deviation of the graph are used as the parameters of the batch normalization layer after the convolutional layer; the rest of the model parameters in the image feature extraction model are the model parameters obtained from training;

Use the image feature acquisition model to obtain the feature vector of each image in the test data set;

Obtain the image with the closest feature distance to the image to be recognized in the test data set according to the acquired feature vector, and determine the position information of the image as the position information of the image to be recognized, thereby completing the visual position recognition of the image to be recognized;

Among them, the position information of each image in the test data set is known, and the domain is a set of factors that affect the distribution of image features;

According to the actual application, the delineation of the domain will be completed according to the influence of factors such as illumination, viewing angle, and season on the distribution of image features. In the same domain, the distribution of features of images is similar; Larger impact, and the images taken during the day have similar feature distributions, and the images taken at night have similar feature distributions, then two domains can be obtained according to the lighting conditions;

In the above-mentioned domain adaptive visual position recognition method, when using the image feature extraction model to obtain the feature vector of the image to be recognized, the parameters of each batch of normalization layers in the model do not depend on the training set, but use multiple Images in the same domain acquire corresponding parameters. Since images in the same domain have similar feature distributions, the present invention can realize domain adaptation. When the feature distributions of the images in the training set and the image to be recognized differ greatly, The visual position recognition can still be accurately completed, that is to say, the present invention can improve the robustness of the visual position recognition.

Further, when using the image feature acquisition model to obtain the feature vectors of each image in the test data set, the setting method of each model parameter is:

Use the model parameters obtained from training to set each model parameter;

Or, for each convolutional layer, all the images in the statistical test data set pass through the convolutional layer thickness, and the mean and standard deviation of the obtained feature map are used as the parameters of the batch normalization layer after the convolutional layer; image features The rest of the model parameters in the extracted model are the model parameters obtained from training.

According to a third aspect of the present invention, an image feature extraction model training device is provided, including: a model building module, a training set construction module, and a model training module;

The model building module is used to set up the image feature extraction model based on the deep neural network, and the image feature extraction model is used to obtain the feature vector of the image;

The training set construction module is used to obtain positive samples and s negative samples of each target image in the standard data set, so as to form a training sample from a target image and its positive samples and negative samples, so as to obtain a training sample composed of all training samples Training set;

The model training module is used to utilize the training set to train the image feature extraction model, thereby obtaining each model parameter;

Among them, the image feature extraction model includes a feature extraction network and a local feature aggregation network;

The feature extraction network includes a plurality of cascaded first networks; the first network is sequentially connected by one or more second networks and a maximum pooling layer, and the maximum pooling layer is used for the output of the previous second network. Image feature selection; the second network includes sequentially connected convolutional layers, batch normalization layers, and activation function layers. The convolutional layer is used to extract features from the image, and the batch normalization layer is used to zero-mean the output image of the convolutional layer. Normalization processing, the activation function layer is used to activate the image output by the batch normalization layer;

The local feature aggregation network is used to aggregate all the local features in the image output by the feature extraction network to obtain the feature vector of the image;

The positive sample of the target image is the image closest to its feature distance among the adjacent images, and the position distance d between the target image and its adjacent image satisfies T _NL ≤ d<T _NH ; the position distance d between the target image and its negative sample satisfies d≥T _F ;

The position information of each image in the standard data set is known, and the target image is multiple images pre-screened in the standard data set; the feature distance between images is the distance between the feature vectors of the image; T _NL , _{T NH} and TF are Preset threshold, 0<T _NL <T _NH , T _NH ≤T _F ; s≥1.

According to the fourth aspect of the present invention, there is also provided a domain-adaptive visual position recognition device based on the image feature extraction model training method provided in the first aspect of the present invention, including: a retrieval set acquisition module, a first feature extraction module , a second feature extraction module and a recognition module;

The retrieval set acquisition module is used to determine the target domain to which the image to be recognized belongs, and obtain multiple images at different positions in the target domain, and use the acquired image and the image to be recognized as the image to be retrieved;

The first feature extraction module is used to take the image to be retrieved as input, and use the image feature extraction model to obtain the feature vector of each image to be retrieved; when obtaining the image feature vector, for each convolutional layer, count all the images to be retrieved after the convolution After the layer, the mean and standard deviation of the obtained feature map are used as the parameters of the batch normalization layer after the convolution layer; the rest of the model parameters in the image feature extraction model are the model parameters obtained from training;

The second feature extraction module is used to obtain the feature vector of each image in the test data set by using the image feature acquisition model;

The recognition module is used to obtain the image with the closest feature distance to the image to be recognized in the test data set according to the feature vectors extracted by the first feature extraction module and the second feature extraction module, and determine the position information of the image as the image to be recognized Position information, so as to complete the visual position recognition of the image to be recognized;

Among them, the location information of each image in the test data set is known, and the domain is a set of factors that affect the distribution of image features.

Generally speaking, through the above technical solutions conceived by the present invention, the following beneficial effects can be obtained:

(1) In the image feature extraction model training method provided by the present invention, in the established image feature extraction model, after each convolution for feature extraction, a batch normalization layer is used to perform a batch normalization on the output image of the convolution layer Zero-mean normalization processing, so that while accelerating model training, the image features extracted by the image feature extraction model have a similar distribution, thereby effectively avoiding the large differences in the feature distribution of the images in the training set. The effect is not good, and then it can improve the problem of low robustness of visual position recognition when the distribution of image features is large.

(2) The image feature extraction model training method provided by the present invention, in its preferred solution, completes the construction of training samples by selecting a plurality of negative samples, which can improve the training accuracy of the model, so that the above-mentioned image feature extraction model is used to obtain When the image feature vector is used for visual position recognition, it has high robustness.

(3) The image feature extraction model training method provided by the present invention, in its preferred scheme, simultaneously constructs a loss function based on the idea of triple loss and difficult example mining, so that more attention is paid to the negative samples that are difficult to distinguish during the model training process , and then it can avoid the interference of negative samples similar to the image to be recognized when using the above image feature extraction model for visual position recognition.

(4) In the domain adaptive visual position recognition method provided by the present invention, when using the image feature extraction model to obtain the feature vector of the image to be recognized, the parameters of each batch of normalization layers in the model do not depend on the training set, but use multiple An image belonging to the same domain as the image to be recognized acquires corresponding parameters, thereby realizing domain adaptation and improving the robustness of visual position recognition.

Description of drawings

Fig. 1 is a schematic diagram of an image feature extraction model provided by an embodiment of the present invention;

FIG. 2 is a flowchart of a domain-adaptive visual position recognition method provided by an embodiment of the present invention;

FIG. 3 is a schematic diagram of a domain-adaptive visual position recognition method provided by an embodiment of the present invention.

Detailed ways

In order to make the object, technical solution and advantages of the present invention clearer, the present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described here are only used to explain the present invention, not to limit the present invention. In addition, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not constitute a conflict with each other.

The image feature extraction model training method provided by the present invention includes:

(1) Establish an image feature extraction model based on a deep neural network to obtain the feature vector of the image;

As shown in Figure 1, the image feature extraction model includes a feature extraction network and a local feature aggregation network;

The feature extraction network includes a plurality of cascaded first networks; the first network is sequentially connected by one or more second networks and a maximum pooling layer (Pool), and the maximum pooling layer is used for the previous second The image output by the network is selected for feature selection; the second network includes a sequentially connected convolutional layer (Conv), batch normalization layer (BN) and activation function layer (Relu). The convolutional layer is used to extract features from the image, and the batch normalization layer It is used to perform zero-mean normalization processing on the image output by the convolution layer, and the activation function layer is used to perform activation processing on the image output by the batch normalization layer; in this embodiment, the convolution kernel size of the convolution layer in each second network Specifically, it is 3x3; the number of second networks included in each first network may be the same or different;

The local feature aggregation network is used to aggregate all the local features in the image output by the feature extraction network to obtain the feature vector of the image;

In an optional embodiment, as shown in Figure 1, the local feature aggregation network includes: a dimensionality reduction convolutional layer (Conv), a soft-max layer (Soft-max), an aggregation layer (VLAD), an internal normalization layer (Intra-normalization) and overall normalization layer (L2-normalization);

The dimensionality reduction convolutional layer is a convolutional layer with a convolution sum size of 1x1, which is used to reduce the latitude of the image to be aggregated to be equal to the number of preset cluster centers, so that each image to be aggregated A channel represents the weight of the difference between the local feature and each cluster center; where the image to be aggregated is the image output by the feature extraction network;

The soft-max layer is used to normalize the weight of the difference between local features and each cluster center;

The aggregation layer is used to aggregate VLAD vectors according to local features, cluster centers, and normalized weights; VLAD vectors are composed of N D-dimensional vectors, N is the number of cluster centers, and D is the dimension of the cluster centers;

Suppose there are N cluster centers, represented by CluCenter, CluCenter=[c ₁ ,c ₂ ,...c _j ,...c _N ], where the dimension of each cluster center is D, c _j (j ∈{1,2,…,N}) represents the jth cluster center;

The feature extraction network outputs n local features of each image, represented by Features, Features=[f ₁ ,f ₂ ,...f _i ...f _n ], where f _i (i∈{1,2 ,...,n}) represents the i-th local feature;

The weight of the difference between the i-th local feature and the j-th cluster center is represented by a _ij , and the vector VLADvector _j of the j-th D dimension in the VLAD vector (ie, the j-th element of the VLAD vector) can be obtained as:

The internal normalization layer is used to normalize the vectors of each D dimension in the VLAD vector, so that the distribution of the vectors of each D dimension is in the same order of magnitude;

The overall normalization layer is used to concatenate the D-dimensional vectors processed by the internal normalization layer into a column vector, and normalize the column vector so that each local feature of the image to be aggregated is distributed in the same Order of magnitude; thus, the convergence speed of the neural network model and the accuracy of the network model can be improved;

In this embodiment, both the internal normalization layer and the overall normalization layer complete the normalization operation through the L2 norm normalization method;

(2) In the standard data set, obtain positive samples and s negative samples of each target image, so as to form a training sample by a target image and its positive samples and negative samples, thereby obtaining a training set composed of all training samples;

Wherein, the position information of each image in the standard data set is known, and the target image is a plurality of images pre-screened in the standard data set;

The positive sample of the target image is the image closest to its feature distance among the adjacent images, and the position distance d between the target image and its adjacent image satisfies T _NL ≤ d<T _NH ; the position distance between the target image and its negative sample satisfies d≥T _F ; where , T _NL , T _NH and TF are preset thresholds, 0<T _NL <T _NH , T _NH ≤T _F ; _s≥1 ; the feature distance between images is the distance between feature vectors of images;

In this embodiment, the standard dataset used for model training is the TokyoTimeMachine Google Street View dataset; this dataset includes images collected from multiple different locations, each location is collected from 12 angles, and a total of about 47,000 images , each image has geographic coordinate information; in this data set, the target image is 10,000 randomly selected images, that is, the total number of training samples is n=10,000; in other applications, other data sets can also be selected according to actual application requirements as a standard dataset;

Thresholds T _NL , _{T NH} and TF can be set according to the standard data set adopted and actual application scenarios. Generally, T _NH ≤ 25, 25 ≤ T _F ; in this embodiment, the threshold setting is specifically T _NL = 1, T _NH = 10, T _F = 25; set the upper and lower limits of the distance between the target image and the positive sample through the threshold T _NL and T _NH , which can ensure that the positive sample is similar to the target image but different, avoiding the model Overfitting can ensure a better model training effect;

In this embodiment, in each training sample, the number of negative samples is specifically set to be s=4; by selecting multiple negative samples, the training accuracy of the model can be improved, so that the image feature vector obtained by the above-mentioned image feature extraction model is used for visual It has high robustness in position recognition;

The training set trainSet constructed in this embodiment can be specifically expressed as:

Among them, for any k-th training sample S _k , q _k , p _k and n _ki (i∈{1,2,3,4}) represent the target image, positive sample and i-th negative sample in the training sample, respectively. sample;

(3) Use the training set to train the image feature extraction model, so as to obtain each model parameter.

In the image feature extraction model training method described above, in the image feature extraction model established, each convolution used for feature extraction is followed by a batch normalization layer (Batch Normalization) to carry out zero-mean normalization on the image output by the convolution layer In this way, while speeding up the model training, the image features extracted by the image feature extraction model have a similar distribution, thereby effectively avoiding the poor model training effect due to the large difference in the feature distribution of the images in the training set , which can further improve the problem of low robustness of visual position recognition when the distribution of image features is quite different.

In order to further improve the robustness of visual position recognition, in the step (3) of the above image feature extraction model training method, when using the training set to train the image feature extraction model, the loss function adopted is specifically:

in, Indicates the feature distance between the target image q _k and its positive sample p _k , is the feature distance between the target image q _k and its negative sample n _ki , m is a predefined hyperparameter, max means taking the maximum value, and min means taking the minimum value;

The above loss function, based on the idea of triplet loss, minimizes the feature distance between the target image and the positive sample and maximizes the feature distance to the negative sample through training; This item selects the negative sample with the largest loss, so that based on the idea of mining difficult examples, more attention is paid to the negative samples that are difficult to distinguish during the model training process, and then when using the above image feature extraction model for visual position recognition, Avoid the interference of negative samples similar to the image to be recognized.

The present invention also provides a domain-adaptive visual position recognition method based on the above image feature extraction model training method, as shown in Figure 2, including:

Determine the target domain to which the image to be recognized belongs, and obtain multiple images at different positions in the target domain, and use the acquired image and the image to be recognized as the image to be retrieved;

Taking the image to be retrieved as input, use the image feature extraction model to obtain the feature vector of each image to be retrieved; when obtaining the image feature vector, for each convolutional layer, count all the images to be retrieved after passing through the convolutional layer, the obtained features The mean and standard deviation of the graph are used as the parameters of the batch normalization layer after the convolutional layer; the rest of the model parameters in the image feature extraction model are the model parameters obtained from training;

Utilize the image feature acquisition model to obtain the feature vector of each image in the test data set; in the present embodiment, the test data set for visual position recognition is specifically the tokyo247 data set, wherein each image has geographic coordinate information;

Obtain the image with the closest feature distance to the image to be recognized in the test data set according to the acquired feature vector, and determine the position information of the image as the position information of the image to be recognized, thereby completing the visual position recognition of the image to be recognized;

Among them, the position information of each image in the test data set is known, and the domain is a set of factors that affect the distribution of image features;

According to the actual application, the delineation of the domain will be completed according to the influence of factors such as illumination, viewing angle, and season on the distribution of image features. In the same domain, the distribution of features of images is similar; greater impact, and the images taken during the day have similar feature distributions, and the images taken at night have similar feature distributions, then two domains can be obtained according to the lighting conditions; according to which factors to complete the delineation of the domain, and the same The similarity of image feature distribution in the domain can be determined according to the actual application requirements, as long as the accuracy of the final visual position recognition meets the requirements;

In the above-mentioned domain adaptive visual position recognition method, when using the image feature extraction model to obtain the feature vector of the image to be recognized, the parameters of each batch of normalization layers in the model do not depend on the training set, but use multiple Images in the same domain acquire corresponding parameters. Since images in the same domain have similar feature distributions, the present invention can realize domain adaptation. When the feature distributions of the images in the training set and the image to be recognized differ greatly, The visual position recognition can still be accurately completed, that is to say, the present invention can improve the robustness of the visual position recognition.

In the above visual position recognition method, since the test data set tokyo247 used for visual position recognition has similar feature distribution to the images in the standard data set TokyoTimeMachine used for model training, in this embodiment, image features are used to obtain the model When obtaining the feature vector of each image in the test data set, directly utilize the model parameters obtained from the training of the above-mentioned image feature extraction model training method to set each model parameter;

In other application scenarios, in order to avoid dependence on the training set to the greatest extent, when using the image feature acquisition model to obtain the feature vectors of each image in the test data set, the model parameters can also be set in the following way: For each volume Convolutional layer, all the images in the statistical test data set pass through the convolutional layer thickness, the mean and standard deviation of the obtained feature map are used as the parameters of the batch normalization layer after the convolutional layer; the rest of the models in the image feature extraction model Parameters are the model parameters obtained from training.

Figure 3 shows an example of visual location recognition by using the image, in which the training set image represents the standard data set used for model training, the query image is the image to be retrieved, and the gallery image is the test set database image.

The present invention also provides an image feature extraction model training device for realizing the above-mentioned image feature extraction model training method, the device includes: a model building module, a training set construction module and a model training module;

The model building module is used to set up the image feature extraction model based on the deep neural network, and the image feature extraction model is used to obtain the feature vector of the image;

The training set construction module is used to obtain positive samples and s negative samples of each target image in the standard data set, so as to form a training sample from a target image and its positive samples and negative samples, so as to obtain a training sample composed of all training samples Training set;

The model training module is used to utilize the training set to train the image feature extraction model, thereby obtaining each model parameter;

Among them, the image feature extraction model includes a feature extraction network and a local feature aggregation network;

The feature extraction network includes a plurality of cascaded first networks; the first network is sequentially connected by one or more second networks and a maximum pooling layer, and the maximum pooling layer is used for the output of the previous second network. Image feature selection; the second network includes sequentially connected convolutional layers, batch normalization layers, and activation function layers. The convolutional layer is used to extract features from the image, and the batch normalization layer is used to zero-mean the output image of the convolutional layer. Normalization processing, the activation function layer is used to activate the image output by the batch normalization layer;

The local feature aggregation network is used to aggregate all the local features in the image output by the feature extraction network to obtain the feature vector of the image;

The positive sample of the target image is the image closest to its feature distance among the adjacent images, and the position distance d between the target image and its adjacent image satisfies T _NL ≤ d<T _NH ; the position distance d between the target image and its negative sample satisfies d≥T _F ;

The position information of each image in the standard data set is known, and the target image is multiple images pre-screened in the standard data set; the feature distance between images is the distance between the feature vectors of the image; T _NL , _{T NH} and TF are Preset threshold, 0<T _NL <T _NH , T _NH ≤T _F ; s≥1;

In this embodiment, for the specific implementation manners of each module, reference may be made to the description in the foregoing method embodiments, which will not be repeated here.

The present invention also provides a domain-adaptive visual position recognition device, which is used to realize the above-mentioned domain-adaptive visual position recognition method. The device includes: a retrieval set acquisition module, a first feature extraction module, a second feature extraction module, and identification module;

The retrieval set acquisition module is used to determine the target domain to which the image to be recognized belongs, and obtain multiple images at different positions in the target domain, and use the acquired image and the image to be recognized as the image to be retrieved;

The first feature extraction module is used to take the image to be retrieved as input, and use the image feature extraction model to obtain the feature vector of each image to be retrieved; when obtaining the image feature vector, for each convolutional layer, count all the images to be retrieved after the convolution After the layer, the mean and standard deviation of the obtained feature map are used as the parameters of the batch normalization layer after the convolution layer; the rest of the model parameters in the image feature extraction model are the model parameters obtained from training;

The second feature extraction module is used to obtain the feature vector of each image in the test data set by using the image feature acquisition model;

The recognition module is used to obtain the image with the closest feature distance to the image to be recognized in the test data set according to the feature vectors extracted by the first feature extraction module and the second feature extraction module, and determine the position information of the image as the image to be recognized Position information, so as to complete the visual position recognition of the image to be recognized;

Among them, the position information of each image in the test data set is known, and the domain is a set of factors that affect the distribution of image features;

In this embodiment, for the specific implementation manners of each module, reference may be made to the description in the foregoing method embodiments, which will not be repeated here.

Those skilled in the art can easily understand that the above descriptions are only preferred embodiments of the present invention, and are not intended to limit the present invention. Any modifications, equivalent replacements and improvements made within the spirit and principles of the present invention, All should be included within the protection scope of the present invention.

Claims (8)

1. An image feature extraction model training method is characterized by comprising the following steps:

(1) establishing an image feature extraction model based on a deep neural network, and obtaining feature vectors of an image;

the image feature extraction model comprises a feature extraction network and a local feature aggregation network;

the feature extraction network comprises a plurality of first networks in cascade; the first network is formed by sequentially connecting one or more second networks and a maximum pooling layer, and the maximum pooling layer is used for carrying out feature selection on images output by the previous second networks; the second network comprises a convolution layer, a batch standardization layer and an activation function layer which are sequentially connected, wherein the convolution layer is used for carrying out feature extraction on images, the batch standardization layer is used for carrying out zero-mean standardization processing on the images output by the convolution layer, and the activation function layer is used for carrying out activation processing on the images output by the batch standardization layer;

the local feature aggregation network is used for aggregating all local features in the image output by the feature extraction network so as to obtain a feature vector of the image;

(2) obtaining a positive sample and s negative samples of each target image in the standard data set so as to form a training sample by one target image and the positive sample and the negative sample thereof, thereby obtaining a training set formed by all the training samples;

the positive sample of the target image is the image which is closest to the characteristic distance of the target image in the adjacent images, and the position distance d between the target image and the adjacent images meets T_NL≤d＜T_NH(ii) a The position distance between the target image and the negative sample thereof satisfies d ≥ T_F；

(3) Training the image feature extraction model by using the training set so as to obtain each model parameter;

the position information of each image in the standard data set is known, and the target image is a plurality of images screened in advance in the standard data set; the characteristic distance between the images is the distance between the characteristic vectors of the images; t is_NL、T_NHAnd T_FAre all preset threshold values, T is more than 0_NL＜T_NH，T_NH≤T_F；s≥1。

2. The image feature extraction model training method of claim 1, wherein the local feature aggregation network comprises: the integrated structure comprises a dimensionality reduction convolution layer, a soft-max layer, a polymerization layer, an internal normalization layer and an integral normalization layer; the dimensionality reduction convolutional layer is a convolutional layer and is used for reducing the dimensionality of the image to be aggregated to be equal to the number of preset clustering centers so that each channel of the image to be aggregated represents the weight of the difference between the local characteristic and each clustering center;

the soft-max layer is used for normalizing the weight of the difference between the local feature and each cluster center;

the aggregation layer is used for aggregating according to the local features, the clustering center and the weight after normalization to obtain a VLAD vector;

the VLAD vector consists of vectors of N D dimensions, wherein N is the number of the clustering centers, and D is the dimension of the clustering centers;

the inner normalization layer is used for normalizing the vector of each D dimension in the VLAD vector so that the distribution of the vector of each D dimension is in the same order of magnitude;

the integral normalization layer is used for serially connecting the D-dimension vectors processed by the internal normalization layer into a column vector, and then normalizing the column vector so as to enable each local feature of the image to be aggregated to be distributed in the same order of magnitude;

and the image to be aggregated is the image output by the feature extraction network.

3. The method of training an image feature extraction model of claim 1, wherein s > 1.

4. The method for training an image feature extraction model according to claim 3, wherein in the step (3), when the training set is used to train the image feature extraction model, the loss function used is:

wherein n is the total number of training samples, k is the serial number of the training samples, i is the serial number of the negative sample, q_k、p_kAnd n_kiRespectively representing a target image, a positive sample and an ith negative sample in a kth training sample,representing a target image q_kWith its positive sample p_kThe characteristic distance between the two or more of them,is a target image q_kWith its negative sample n_kiM is a predefined hyper-parameter, max represents taking the maximum value, and min represents taking the minimum value.

5. A domain-adaptive visual position recognition method based on the image feature extraction model training method of any one of claims 1 to 4, comprising:

determining a target domain to which an image to be identified belongs, acquiring a plurality of images at different positions in the target domain, and taking the acquired image and the image to be identified as an image to be retrieved;

taking the image to be retrieved as input, and obtaining a feature vector of each image to be retrieved by using the image feature extraction model; when the image characteristic vector is obtained, for each convolution layer, counting the mean value and the standard deviation of the characteristic graph obtained after all the images to be retrieved pass through the convolution layer, and taking the mean value and the standard deviation as the parameters of a batch standard layer behind the convolution layer; the rest model parameters in the image feature extraction model are model parameters obtained by training;

acquiring a feature vector of each image in the test data set by using the image feature acquisition model;

obtaining an image which is closest to the characteristic distance of the image to be recognized in the test data set according to the obtained characteristic vector, and determining the position information of the image as the position information of the image to be recognized, so as to complete the visual position recognition of the image to be recognized;

and the position information of each image in the test data set is known, and the domain is a factor set influencing the image characteristic distribution.

6. The method of claim 1, wherein when the image feature acquisition model is used to obtain the feature vector of each image in the test dataset, the setting of each model parameter is as follows:

setting each model parameter by using the model parameters obtained by training;

or, for each convolution layer, counting the thickness of all images in the test data set passing through the convolution layer, and taking the mean value and standard deviation of the obtained characteristic diagram as parameters of a batch of normalization layers behind the convolution layer; and the rest model parameters in the image feature extraction model are model parameters obtained by training.

7. An image feature extraction model training device, comprising: the system comprises a model establishing module, a training set constructing module and a model training module;

the model establishing module is used for establishing an image feature extraction model based on a deep neural network, and the image feature extraction model is used for acquiring a feature vector of an image;

the training set construction module is used for obtaining a positive sample and s negative samples of each target image in the standard data set so as to form a training sample by one target image and the positive sample and the negative sample thereof, thereby obtaining a training set formed by all the training samples;

the model training module is used for training the image feature extraction model by using the training set so as to obtain each model parameter;

the image feature extraction model comprises a feature extraction network and a local feature aggregation network;

the feature extraction network comprises a plurality of first networks in cascade; the first network is formed by sequentially connecting one or more second networks and a maximum pooling layer, and the maximum pooling layer is used for carrying out feature selection on images output by the previous second networks; the second network comprises a convolution layer, a batch standardization layer and an activation function layer which are sequentially connected, wherein the convolution layer is used for carrying out feature extraction on images, the batch standardization layer is used for carrying out zero-mean standardization processing on the images output by the convolution layer, and the activation function layer is used for carrying out activation processing on the images output by the batch standardization layer;

the local feature aggregation network is used for aggregating all local features in the image output by the feature extraction network so as to obtain a feature vector of the image;

the positive sample of the target image is the image which is closest to the characteristic distance of the target image in the adjacent images, and the position distance d between the target image and the adjacent images meets T_NL≤d＜T_NH(ii) a The position distance d between the target image and the negative sample thereof satisfies that d is more than or equal to T_F；

The position information of each image in the standard data set is known, and the target image is a plurality of images screened in advance in the standard data set; the characteristic distance between the images is the distance between the characteristic vectors of the images; t is_NL、T_NHAnd T_FAre all preset threshold values, T is more than 0_NL＜T_NH，T_NH≤T_F；s≥1。

8. A domain-adaptive visual position recognition device based on the image feature extraction model training method of any one of 1-4 is characterized by comprising the following steps: the system comprises a retrieval set acquisition module, a first feature extraction module, a second feature extraction module and an identification module;

the retrieval set acquisition module is used for determining a target domain to which an image to be identified belongs, acquiring a plurality of images at different positions in the target domain, and taking the acquired images and the image to be identified as the image to be retrieved;

the first feature extraction module is used for taking the image to be retrieved as input and obtaining a feature vector of each image to be retrieved by using the image feature extraction model; when the image characteristic vector is obtained, for each convolution layer, counting the mean value and the standard deviation of the characteristic graph obtained after all the images to be retrieved pass through the convolution layer, and taking the mean value and the standard deviation as the parameters of a batch standard layer behind the convolution layer; the rest model parameters in the image feature extraction model are model parameters obtained by training;

the second feature extraction module is used for acquiring a feature vector of each image in the test data set by using the image feature acquisition model;

the identification module is used for acquiring an image which is closest to the characteristic distance of the image to be identified in the test data set according to the characteristic vectors extracted by the first characteristic extraction module and the second characteristic extraction module, and determining the position information of the image as the position information of the image to be identified, so that the visual position identification of the image to be identified is completed;

and the position information of each image in the test data set is known, and the domain is a factor set influencing the image characteristic distribution.