CN117115483A - Template matching method, medium and equipment based on heterogeneous image feature fusion - Google Patents

Abstract

The application relates to the technical field of image processing, in particular to a template matching method based on heterogeneous image feature fusion, which comprises the following steps: a, acquiring an image set and preprocessing to obtain a background image and a template image; b, establishing a matching model, wherein the matching model comprises an encoder, a feature mapper and a decoder which are sequentially connected, and sending the background image and the template image in the acquired image set into the matching model for matching; training the established matching model based on the preprocessed image set, and calculating a loss function until the loss function converges to a preset value to obtain a matching model comprising a trained encoder, a feature mapper and a decoder; and D, matching the test images by using the trained matching model. Template matching between heterologous images can be realized end to end, labels can be automatically generated in the training process, self-supervision learning is realized, a large amount of human resource labeling is not required to be consumed, and the method is more efficient and accurate than the existing method.

Description

A template matching method, medium and equipment based on heterogeneous image feature fusion

Technical field

The invention relates to the technical field of image processing, and in particular to a template matching method based on heterogeneous image feature fusion.

Background technique

Template matching refers to determining the spatial mapping relationship between the two images of the template image and the background image, so as to find a specific position in the background image. Generally, the main difficulty in image template matching is that the imaging between the template and the background image is often a heterogeneous image. Heterogeneous images refer to imaging images from different light sources, and their imaging characteristics are very different.

Existing template matching methods often manually design image feature points, and then extract the image feature points to find matching relationships. For example, image corner points are extracted as image feature points through SHIFT, ORB and other methods, and then the feature point matching algorithm is used to extract the image feature points. Determine the correspondence between the image feature points, and further calculate the homography transformation matrix used to describe the image mapping relationship, thereby determining the matching relationship between the two images.

This type of method has achieved good results in some daily scenarios, but template matching of heterogeneous images with large differences in imaging characteristics will fail. In addition, the feature points extracted by traditional methods often have some abnormal points, called "outer points", which need to be iteratively eliminated through a complex random selection consistency method, which is time-consuming and inefficient. Moreover, the classic decoder in the existing technology consists of six decoding modules with the same structure, which has a large number of parameters and low reasoning efficiency.

The traditional method needs to extract feature points first, and then determine the matching relationship between feature points through the feature point descriptors. It also needs to eliminate abnormal points through the RANSC method. Template matching can be completed in two steps, which consumes a lot of computing resources and is inefficient. .

Contents of the invention

In view of the above problems existing in the prior art, the present invention provides a template matching method based on heterogeneous image feature fusion, which can realize template matching between heterogeneous images end-to-end, and can automatically generate labels during the training process. , realizing self-supervised learning without consuming a lot of human resources for labeling, and is more efficient and accurate than existing methods.

In the first aspect, embodiments of the present application provide a template matching method based on heterogeneous image feature fusion, including the steps:

A. Obtain an image set and perform preprocessing to obtain a background image and a template image. The image set includes a training image set and a test image;

B. Establish a matching model. The matching model includes an encoder, a feature mapper and a decoder connected in sequence, and the background image and template image in the acquired image set are sent to the matching model for matching;

C. Train the established matching model based on the preprocessed image set and calculate the loss function until the loss function converges to the preset value to obtain a matching model including the trained encoder, feature mapper and decoder. ;

D. Use the trained matching model to match the test image.

In an optional solution of the first aspect, in step A, the preprocessing specifically includes:

A1, randomly select two images from the image set, recorded as the first image and the second image, capture an image of random size on the first image, recorded as the first screenshot, and record the position of the first screenshot on the first image. Key point coordinates are used as training positive samples;

A2, capture an image on the second image at the same position as the first image, record it as the second screenshot, and use it as a training negative sample.

In yet another optional solution of the first aspect, in step A, the preprocessing further includes performing data enhancement processing on the first image and the second image, where the data enhancement processing refers to image transformation, including zooming and translation. , rotation, perspective, randomly adding noise, random contrast transformation, random brightness changes and randomly selecting an area for occlusion.

In another alternative to the first aspect, in step B, before the background image and template image in the acquired image set are sent to the matching model, the grid is also divided, and a grid with a preset ratio is selected for random masking. , the purpose is to reduce redundant information in the image and improve the robustness of the matching model.

In yet another alternative to the first aspect, in step B, the feature mapper in the matching model includes a convolutional layer connected in sequence and a linear mapping layer whose output is a preset dimension; the decoder includes a convolutional layer connected in sequence and a linear mapping layer whose output is a preset dimension. Self-attention structure, residual structure and cross-self-attention structure;

Cross self-attention structure: used for the dot multiplication of the feature vector of the background image and the feature vector of the template image, and the dot multiplication of the feature vector of the template image and the feature vector of the background image to obtain the weight.

In yet another alternative to the first aspect, in step B, before the image is input to the decoder, the position information is also encoded, and the encoding formula is as follows:

Among them, PE represents the position encoding vector, pos represents the grid number, d_index represents the element position of the encoding feature vector, i represents the element number in the encoding feature vector, d represents the dimension of the encoding feature vector, the value of dimension d is n, and n is an integer. .

In yet another alternative to the first aspect, in step C, the loss function, denoted as L, has a calculation formula of:

L＝λ ₁ L _confidence +λ ₂ L _loc

Among them, L _confidence represents the loss value of whether there is a key point at the position, λ ₁ represents the weight of the loss value of whether there is a key point at the position, L _loc represents the loss value of the key point coordinates, λ ₂ represents the weight of the loss value of the key point coordinates, c _i represents the true value of whether the i-th key point has moved, which is 1 if it has moved, and 0 if it has not moved. p _i represents whether there is a predicted value of the i-th key point, and x _i represents the true value of the abscissa of the i-th key point. value, y _i represents the true value of the ordinate of the i-th key point, Represents the predicted value of the abscissa of the i-th key point, /> Represents the predicted value of the ordinate of the i-th key point.

In yet another alternative to the first aspect, step D specifically includes:

D1, obtain the preprocessed test image, including the preprocessed background image and template image;

D2, input the preprocessed background image and template image into the trained matching model to obtain the matching result;

D3, analyze the matching results and set the preset confidence level of the key points. If the number of key points with a confidence level greater than the preset confidence value is greater than or equal to 3, the matching is successful, and the matching position is determined by the three key points with the highest confidence level. , if the number of key points with a confidence greater than the preset confidence value is less than 3, the background image and the template image do not match.

In a second aspect, embodiments of the present application provide a computer storage medium. The computer storage medium stores a computer program. The computer program includes program instructions. When executed by a processor, the program instructions can implement the first aspect or the third aspect of the embodiment of the present application. Any implementation of the aspect provides a template matching method based on heterogeneous image feature fusion.

In a third aspect, embodiments of the present application provide an electronic device, including a memory and a processor. The memory and the processor are communicatively connected to each other. The memory stores computer instructions. The processor executes the instructions. The computer instructions are used to execute a template matching method based on heterogeneous image feature fusion provided by the first aspect of the embodiment of the present application or any implementation of the first aspect.

The beneficial technical effects of the present invention include:

1. Through the improved multi-head self-attention structure, the high-dimensional features of heterogeneous images are fused, and the spatial matching relationship between the template image and the background image is discovered, which can effectively complete the template matching of heterogeneous feature images, which is different from existing images. The matching method is a method of manually extracting image corners as feature points and then matching them, which overcomes the problem of not being suitable for template matching scenarios. The algorithm is more adaptable and can complete template matching in one step in an end-to-end manner.

2. Labels are automatically generated during the training process to achieve self-supervised learning without spending a lot of human resources on labeling. Compared with existing methods, image matching is faster, more accurate, and more efficient.

Description of drawings

In order to explain the embodiments of the present invention or the technical solutions in the prior art more clearly, the drawings needed to be used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings in the following description are only These are some embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without exerting creative efforts.

Figure 1 is a flow chart of a template matching method based on heterogeneous image feature fusion in the present invention;

Figure 2 is a block diagram of a template matching model based on heterogeneous image feature fusion in the present invention;

Figure 3 is a background image of the verification experiment in the embodiment of the present invention;

Figure 4 is a template image of the verification experiment in the embodiment of the present invention;

Figure 5 is a schematic diagram of the matching results of the verification experiment in the embodiment of the present invention.

Detailed ways

The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present application.

In the following introduction, the terms "first" and "second" are used for descriptive purposes only and shall not be understood as indicating or implying relative importance. The following description provides multiple embodiments of the present application. Different embodiments can be replaced or combined. Therefore, the present application can also be considered to include all possible combinations of the same and/or different embodiments described. Thus, if one embodiment contains features A, B, C, and another embodiment contains features B, D, then the application should also be considered to include all other possible combinations containing one or more of A, B, C, D embodiment, although this embodiment may not be explicitly documented in the following content.

The following description provides examples and does not limit the scope, applicability, or examples set forth in the claims. Changes may be made in the function and arrangement of the elements described without departing from the scope of the disclosure. Various procedures or components may be omitted, substituted, or added as appropriate from each example. For example, the described methods may be performed in a different order than that described, and various steps may be added, omitted, or combined. Additionally, features described with respect to some examples may be combined into other examples.

Example 1:

Referring to Figure 1, a template matching method based on heterogeneous image feature fusion includes steps:

Step A: Obtain an image set and perform preprocessing. The image set includes a training image set and a test image. The training image set includes: an open source image data set, which can be COCO, ImageNet, etc., and a continuous image data set, that is, continuous image frames extracted from the video. Preprocessing includes: label file generation and data enhancement.

Step A specifically includes:

Step A1: Randomly select two different images from the training image set, marked as the first image and the second image.

Step A2: Randomly take a randomly sized first screenshot from the first image, and record the 9 point coordinates of the first screenshot on the original image: upper left corner (x1, y1), upper right corner (x2, y2), lower left corner ( x3,y3), lower right corner (x4,y4), upper left area center point (x5,y5), upper right area center point (x6,y6), lower left area center point (x7,y7), lower right area center point (x8 , y8), screenshot center point (x9, y9), normalize the coordinates of the 9 points according to the image width and height, and store them in the label file as training positive samples.

Step A3: Take a screenshot of the second image at the same position as the first image to obtain the second screenshot, and generate an empty label file as a training negative sample.

Step A4: Perform data enhancement on the first screenshot and the second screenshot to obtain the first transformed screenshot and the second transformed screenshot.

Data enhancement refers to image transformation of the background image, including scaling, translation, rotation, perspective transformation, random addition of noise, random contrast transformation, random brightness change, random selection of an area for occlusion and other operations, thereby increasing the diversity of data and improving The algorithm has multi-scenario adaptability and randomly selects at least two data enhancement methods for each training sample to process the image. It should be noted that when the data enhancement method will affect the geometric structure of the image, such as scaling, translation, rotation, perspective transformation and other operations, the coordinate points in the corresponding label file need to be transformed accordingly.

Referring to Figure 2, step B, a matching model is established. The matching model includes an encoder, a feature mapper and a decoder connected in sequence, and the background image and template image in the training image set are sent to the matching model for matching.

This invention uses a method based on a multi-head self-attention mechanism to fuse heterogeneous image features to achieve template matching. The network structure is based on the encoder and decoder, and is improved and designed according to the characteristics of the problems and tasks to be solved by this invention to better Get the matching relationship between the background image and the template image. The input of the algorithm is the background image and the template image, and the two images are represented by a series of feature vectors through the encoder and feature mapper. Then the improved decoder is used to fuse the two image features to obtain the feature encoding output.

Before inputting into the encoder, the input background image and template image are preprocessed. The preprocessing is a scaling operation to obtain a preprocessed background image and a preprocessed template image, where the preprocessed background image is The width is w1*16 and the height is h1*16, where w1 and h1 are integers between 14 and 40. Specifically, w1 and h1 are both set to 20, and the resolution of the preprocessed background image is 320*320. The width of the preprocessed template image is w2*16 and the height is h2*16, where w2 and h2 are integers between 3 and 7. Specifically, w2 and h2 are both set to 5. The preprocessed template image is The resolution is 80*80.

The preprocessed background image and the preprocessed template image are divided into grids. The size of each grid is 16×16 pixels. The preprocessed background image is divided into w1*h1 grids. After preprocessing The template image is divided into w2*h2 grids. Further, the preprocessed background image is divided into 400 grids, and the preprocessed template image is divided into 25 grids.

It should be noted that during the training phase, grids with proportions r1 and r2 will be selected for random masking to reduce redundant image information and improve the robustness of the algorithm. After passing the divided image grid through the corresponding image feature mapping structure, w1*h1 background image feature vectors with n dimensions and w2*h2 template feature vectors with n dimensions can be obtained. Specifically, r1=0.7, r2=0.4, and then 400 background image feature vectors with a dimension of 768 and 25 template feature vectors with a dimension of 768 can be obtained.

The encoder includes a self-attention structure and a residual structure connected in sequence. The background image feature vector and the template feature vector obtain the vectors of K, Q and V through three matrix operations respectively. The Q corresponding to each vector and the K points of all vectors Multiply to obtain the weight, then multiply the obtained weight with V, and finally obtain the weighted sum V value as the output vector of the encoder. Specifically, the Q of the background image feature vector is multiplied by K points of the template feature vector to obtain the weight. At the same time, the Q of each template feature vector is multiplied by the K points of the background image feature to obtain the weight, and then the obtained weight is multiplied by V. Finally, the weighted sum V value is obtained as the output vector of the encoder.

The obtained output vector of the encoder is feature mapped through the feature mapper, and then input into the improved decoder structure. The feature mapper includes a convolution layer and a linear mapping layer connected in sequence, that is, a fully connected layer. The convolution layer includes a convolution with a convolution kernel size of 16×16 and a step size of 16×16, an activation function and a batch regression. Unification operation, specifically, the activation function is ReLU, and the convolutional layer is followed by a fully connected layer with an output of n, n=768.

The feature mapper is a fully connected layer with an output dimension of n×n. Specifically, the feature map structure is a fully connected layer with an output dimension of 768×768.

Since each grid has a specific position in the corresponding image, its position information needs to be added to the obtained background image feature vector and template feature vector. Before inputting to the decoder, the position information is also encoded to obtain a position encoding vector of the same dimension. The encoding formula is as follows:

Among them, PE represents the position encoding vector, pos represents the current grid number, and follows the row precedence principle. d_index represents the element position of the encoding feature vector. i represents the element number in the encoding feature vector, which is divided by 2 and rounded down. d represents encoding. The dimension of the feature vector, the value of dimension d is n, n is an integer, and further, the value of dimension d is 768.

After obtaining the position information encoding vector, each position information encoding vector is added to the corresponding background image feature vector or template feature vector to obtain the background image input feature and template input feature. The obtained background image input features and template input features are input into the improved decoder to obtain an output feature vector.

The decoder includes self-attention structure, residual structure and cross self-attention structure connected in sequence. The input of the decoder is 9 query key values (vectors), and V is obtained through the self-attention structure and residual structure. Together with the feature encoding outputs K and Q of the encoder, it is used as the input of the cross self-attention structure, and the output feature vector is obtained through vector addition and normalization operations.

The first w1*h1 Qs of the cross self-attention structure, that is, the feature vectors related to the background image, are only multiplied with the last w2*h2 K points to obtain the weight. The last w2*h2 Qs, that is, the feature vectors related to the template image, are only Multiply the first w1*h1 K points to get the weight, of which 9 query key values are trainable parameters.

Specifically, the first 400 Q of the cross self-attention structure, that is, the feature vectors related to the background image, are only multiplied with the last 25 K points to obtain the weight. The last 25 Q, that is, the feature vectors related to the template image, are only multiplied with the first 400 The weights are obtained by multiplying K points, among which 9 query key values are trainable parameters.

The inputs of the decoder are the background image input features, template input features and query vectors after position encoding. Among them, there are 9 query vectors obtained during the algorithm training process. The output of the decoder is 9 decoding output vectors. The 9 decoding output vectors are further extracted through the feedforward network structure FFN to obtain 9 vectors with a dimension of 3. Each vector is expressed as (p, x , y), p represents the confidence that the key point exists, x represents the abscissa value of the key point, and y represents the ordinate value of the key point.

The advantage of the established decoder is that it has fewer parameters, making the model more lightweight and improving inference efficiency. The cross-correlation between background image features and template image features makes it easier for the matching model to learn the spatial relationship between images and achieve better template matching effects.

Step C: Train the established matching model based on the preprocessed image set, and calculate the loss function until the loss function converges to the preset value to obtain a matching including the trained encoder, feature mapper and decoder. Model.

Step C specifically includes:

Step C1: Perform pre-training. The pre-trained network structure is an encoder and a decoder connected in sequence. The feature mapping layer is first removed, that is, the output of the encoder is directly used as the input of the decoder.

Step C2: Randomly select two images from the open source image data set. In the training positive sample, the first image is the background image, and the first screenshot is the template image. Scale the background image and template image, and separate the width and height of the background image. Scale to w1*16 and h1*16, where w1 and h1 are integers between 14 and 40; scale the width and height of the template image to w2*16 and h2*16 respectively, where w2 and h2 are between 3 and 7 integer. Specifically, w1, h1, w2 and h2 selected for each training are random integers that meet the requirements to improve the adaptability of the trained algorithm model to multi-scale images.

Step C3: Divide the scaled background image and scaled template image into grids. The size of each grid is 16×16 pixels. Select the ratios of r1 and r1 for the scaled background image and scaled template image respectively. The grid of r2 is randomly covered, where 0.5≤r1≤0.8, 0.3≤r2≤0.5, specifically, r1=0.7, r2=0.4.

Step C4: Input the image into the pre-trained network structure, perform forward inference analysis to obtain the matching result, read the label file corresponding to the input image, and calculate the loss function value L until the loss function converges to the preset value to obtain Pretrained matching model.

The loss function, denoted as L, is calculated as:

L＝λ ₁ L _confidence +λ ₂ L _loc

Among them, L _confidence represents the loss value of whether there is a key point at the position, λ ₁ represents the weight of the loss value of whether there is a key point at the position, L _loc represents the loss value of the key point coordinates, and satisfies 5＜λ ₁ <10, λ ₂ represents the key The weight of the loss value of the point coordinate satisfies 0.5 < λ ₂ < 2.0. c _i represents the true value of whether the i-th key point has moved. It is 1 if it has moved, and 0 if it has not moved. p _i represents whether there is a true value of the i-th key point. The predicted value satisfies 0 ≤ p _i ≤ 1.0, x _i represents the true value of the abscissa of the i-th key point, y _i represents the true value of the ordinate of the i-th key point, Represents the predicted value of the abscissa of the i-th key point, /> Indicates the predicted value of the ordinate of the i-th key point, where λ ₁ =8.0 and λ ₂ =1.5.

Further, determine whether the preset training times have been reached. If the preset training times have not been reached, determine whether it has converged to the preset value. If it has converged to the preset value, stop training to obtain the pre-trained matching model. If not, If it converges to the preset value, the training will continue. If the preset training times are reached, the training will be stopped to obtain the pre-trained matching model.

In one embodiment, most of the model parameters are frozen at this stage so that they are not updated during the training process. The number of trainable parameters is small, so training can converge quickly. Includes the following steps:

Step C5: Perform correction training and obtain the pre-trained matching model. The network structure is an encoder, feature mapper and decoder connected in sequence, that is, the output of the encoder is input to the decoder after being feature mapped.

Step C6: Use the parameters of the obtained pre-trained matching model to initialize the network parameter values of the encoder and decoder, randomly initialize the feature mapping structure parameters, set the learning rate to 0.1 times of the pre-training stage, and freeze the encoder and decoder parameter values, i.e. the parameters are not updated during the correction training phase.

Step C7: Select two adjacent images from the continuous image data set, obtain difficult training sample pairs through data preprocessing, and repeat steps C2 to C4 to perform parameter training on the network to obtain a trained matching model.

Step D, use the trained matching model to match the test image.

Step D specifically includes:

Step D1: Scale the background image to the size of W1*H1, W1 and H1 are both integer multiples of 16, scale the template image to the size of W2*H2, W2 and H2 are both integer multiples of 16, satisfying W2<W1, H2<H1, specifically, the background image is scaled to 320*320, and the template image is scaled to 80*80.

Step D2: Input the preprocessed background image and template image into the trained matching model to obtain the matching result.

Step D3: Analyze the matching results and determine whether the confidence in the nine output vectors is greater than the preset confidence value. If the number of key points with a confidence greater than the preset confidence value is less than 3, it means that it cannot be included in the background image. Find the matching position of the template map. If the number of key points with confidence greater than the threshold T is not less than 3, it means that the template matching is successful. The matching position is determined by the three key points with the highest confidence.

In order to verify the effect of the method of the present invention, a test data set was established, and the method of the present invention was tested and verified.

Referring to Figure 3 to Figure 5, the background image to be matched is an RGB three-channel color image with a resolution of 1048*777, that is, the figure, and the template image is a near-infrared image with a resolution of 200*200, that is, the two Each image is sent to the matching model for template matching, and the template image is superimposed on the RGB color test image according to the matching relationship. The matching result is as shown in the figure.

The position of the rectangular frame in the figure is the corresponding matching position of the template image on the background image. The size of the circle indicates the confidence level of the matching point. The larger the circle, the higher the confidence level of the matching point. It can be found from the matching results that two images of different sizes and different imaging characteristics can be matched correctly, which illustrates the effectiveness of the method of the present invention.

Example 2:

A computer-readable storage medium stores computer instructions, and the computer instructions are used to cause the computer to execute a template matching method based on heterogeneous image feature fusion as proposed in Embodiment 1. The artificial intelligence acceleration hardware can be Cambrian MLU270 artificial intelligence acceleration hardware and Intel CPU. The model conversion tool chain converts the network model provided in Embodiment 1 into an .om format file so that the model parameters can be correctly loaded into a computer-readable file. Perform inference calculations in storage media.

Embodiment three:

An electronic device, which can be a single server or an embedded computing platform. It includes a memory and a processor. The memory and the processor are communicatively connected to each other. The memory stores computer instructions. The processor executes the computer instructions to execute a template matching method based on heterogeneous image feature fusion as proposed in Embodiment 1.

The memory is a machine-readable storage medium. The electronic device also includes a deep learning parallel computing acceleration chip and a network interface. The deep learning parallel computing acceleration chip, machine-readable storage medium, network interface and processor are connected through the PCIe bus system. Deep learning Parallel computing acceleration chips are used to accelerate forward inference calculations of deep learning models. Machine-readable storage media are used to store programs, instructions or codes. The processor can be used to control the sending and receiving actions of the network interface, so that data can be sent and received through the network. Deep The learning parallel computing acceleration chip realizes parallel processing of frontline inference calculations in deep learning networks and speeds up calculations.

The above-described embodiments are only descriptions of preferred embodiments of the present invention and do not limit the scope of the present invention. Without departing from the design spirit of the present invention, those of ordinary skill in the art may make various modifications to the technical solutions of the present invention. All deformations and improvements shall fall within the protection scope of the present invention.

Claims (10)

1. A template matching method based on heterogeneous image feature fusion is characterized by comprising the following steps of

A, acquiring an image set and preprocessing to obtain a background image and a template image, wherein the image set comprises a training image set and a test image;

b, establishing a matching model, wherein the matching model comprises an encoder, a feature mapper and a decoder which are sequentially connected, and sending the background image and the template image in the acquired image set into the matching model for matching;

training the established matching model based on the preprocessed image set, and calculating a loss function until the loss function converges to a preset value to obtain a matching model comprising a trained encoder, a feature mapper and a decoder;

and D, matching the test images by using the trained matching model.

2. The template matching method based on heterogeneous image feature fusion according to claim 1, wherein in step a, the preprocessing specifically includes:

a1, randomly selecting two images in an image set, marking the images as a first image and a second image, cutting an image with random size on the first image, marking the image as a first screenshot, and recording the coordinates of key points of the first screenshot on the first image as training positive samples;

a2, capturing an image at the same position on the second image as the first image, and recording the image as a second screenshot as a training negative sample.

3. The method of claim 2, wherein in step a, the preprocessing further includes performing data enhancement processing on the first image and the second image, where the data enhancement processing refers to image transformation, and includes scaling, translation, rotation, perspective, random noise addition, random contrast transformation, random brightness change, and random selection of a region for occlusion.

4. The template matching method based on heterogeneous image feature fusion according to claim 1, wherein in the step B, before the background image and the template image in the acquired image set are sent to the matching model, mesh division is further performed, and meshes with preset proportions are selected for random covering.

5. The template matching method based on heterogeneous image feature fusion according to claim 1, wherein in the step B, a feature mapper in the matching model comprises a convolution layer and a linear mapping layer with output of a preset dimension, which are sequentially connected; the decoder comprises a self-attention structure, a residual error structure and a crossed self-attention structure which are connected in sequence;

cross self-attention structure: the feature vector for the background image is multiplied by the feature vector point of the template image, and the feature vector of the template image is multiplied by the feature vector point of the background image to obtain the weight.

6. The method of claim 1, wherein in step B, before the image is input to the decoder, the position information is further encoded, where the encoding formula is as follows:

wherein PE represents a position coding vector, pos represents a grid number, d_index represents a coding feature vector element position, i represents an element number in the coding feature vector, d represents a dimension of the coding feature vector, the value of the dimension d is n, and n is an integer.

7. The template matching method based on heterogeneous image feature fusion according to claim 1, wherein in the step C, the loss function is denoted as L, and the calculation formula is as follows:

L＝λ ₁ L _confidence +λ ₂ L _loc

wherein L is _confidence Loss value, lambda, indicating whether or not there is a key point in the location ₁ Weights, L, representing loss values for locations for which there are keypoints _loc Loss value, lambda, representing coordinates of key points ₂ Weights, c, representing loss values for keypoint coordinates _i True value indicating whether the ith key point is moved, move to 1, not move to 0, p _i A predicted value indicating whether the ith key point exists, x _i True value, y, representing the abscissa of the ith keypoint _i A true value representing the ordinate of the ith keypoint,predicted value representing the abscissa of the ith keypoint,/-)>Representing the predicted value of the ordinate of the ith keypoint.

8. The template matching method based on heterogeneous image feature fusion according to claim 1, wherein the step D specifically includes:

d1, acquiring a preprocessed test image, wherein the preprocessed test image comprises a preprocessed background image and a preprocessed template image;

d2, inputting the preprocessed background image and template image into a trained matching model to obtain a matching result;

and D3, analyzing the matching result, setting preset confidence levels of key points, if the confidence levels are greater than the preset confidence values, the number of the key points is greater than or equal to 3, then the matching is successful, the matching position is determined by three key points with the highest confidence levels, and if the confidence levels are greater than the preset confidence values, the number of the key points is less than 3, then the background image and the template image are not matched.

9. A computer-readable storage medium storing computer instructions for causing the computer to perform a heterogeneous image feature fusion-based template matching method according to any one of claims 1 to 8.

10. An electronic device comprising a memory and a processor, the memory and the processor being communicatively coupled to each other, the memory storing computer instructions, the processor executing the computer instructions to perform a heterogeneous image feature fusion-based template matching method as claimed in any one of claims 1-8.