CN115601397A - Ship track tracking and predicting method based on monocular camera - Google Patents

Abstract

The invention belongs to the technical field of ocean mapping, and particularly relates to a ship track tracking and predicting method based on a monocular camera, which comprises the following steps of: step 1: constructing an image data set of the target ship, and step 2: labeling and training a data set, and step 3: evaluating and analyzing the model training result, and step 4: a target ship tracking experiment based on the YOLOv5 and Deep Sort algorithm, and the step 5: visual positioning and data preprocessing of the target ship, and step 6: building an LSTM track prediction model, and step 7: selecting and evaluating the time step of the prediction model, and step 8: and (5) predicting, verifying and analyzing the target ship track. The method can complete the rapid and accurate detection and tracking of the marine ship target, and has important significance for identifying dangerous behaviors of ships and improving the efficiency of maritime supervision.

Description

Ship trajectory tracking and prediction method based on monocular camera

technical field

The invention belongs to the technical field of marine surveying and mapping, and in particular relates to a ship trajectory tracking and prediction method based on a monocular camera.

Background technique

Ship detection and identification and trajectory prediction are early predictions of dangerous behavior of ships, which are of great significance for identifying potential navigation hazards of ships. In addition, object recognition and tracking technology based on monocular cameras can also be applied to the field of military monitoring. Rights and interests, supervision of special areas, detection of illegal activities at sea, etc. have a wide range of application space. At present, in the field of civil navigation, the identification of target ships and the judgment of collision risk still rely on manual plotting of various sensor images and calculation of collision risk. This method not only takes up additional human resources, but also manually evaluates the risk of ship collision. It is also easily influenced by subjective factors.

Contents of the invention

The object of the present invention is to provide a method for tracking and predicting ship trajectory based on a monocular camera, so as to solve the problems in the background technology.

In order to achieve the above object, the present invention provides the following technical solutions: a method for tracking and predicting ship trajectory based on a monocular camera, comprising the following steps:

Step 1: Construct the target ship image dataset,

Using data enhancement technology, using commonly used enhancement operations to generate more target ship images;

Step 2: Dataset labeling and training,

After sorting out all the image data, it is necessary to label all the ship images in the training set and verification set;

Step 3: Evaluation and analysis of model training results,

Using the average precision mean mAP, calculate the precision Precision and recall rate Recall of all categories, and calculate the average value of AP of all categories;

Step 4: Target ship tracking experiment based on YOLOv5 and Deep Sort algorithm,

Integrate the YOLOv5-based target detector with the Deep Sort algorithm to realize the continuous tracking of the custom target ship, and output the pixel coordinates of the target in the tracking video, which is convenient for the next step of target ship orientation calculation;

Step 5: Target ship visual positioning and data preprocessing,

Based on the principle of homogeneous coordinate transformation and the pinhole imaging model, the conversion model of the pixel coordinates of the target ship and the world coordinates is established. Through this process, the pixel coordinates of the target ship tracked in step 4 can be converted into relative coordinates with the optical center of the camera as the origin. ;

Step 6: Construction of LSTM trajectory prediction model,

Select the LSTM network to predict the trajectory of the target ship. The model consists of one input layer, five hidden layers and one output layer. The input dimension of the input sequence of the input layer is 2, that is, the relative coordinates converted from the pixel coordinates in step 5 ;The output sequence output dimension of the output layer is 2, and the hidden layer is composed of 2 LSTM layers, 2 Dropout layers and 1 Dense layer; according to experience, the probability of the Dropout layer is set to 0.3 to prevent the occurrence of overfitting ;Set the parameter of the Dense layer to 2, that is, the final result output is the predicted relative position information of the target ship. In order to introduce a certain nonlinearity, the ReLU activation function is selected as the activation function of the Activation layer, which is set between the hidden layer and the output layer;

Step 7: Selection and evaluation of the time step of the prediction model,

Select the first 80% of the relative coordinate information calculated in step 6 as the training set, and the remaining 20% as the verification set for model training;

Step 8: Target ship trajectory prediction and verification analysis,

Use the best prediction model selected in step 7 to predict the relative trajectory information of the target ship, and use the original visual positioning trajectory for verification, and analyze the error of the model prediction. If the ideal prediction accuracy is not achieved, you should continue to adjust the parameters of the network model , to retrain to improve the accuracy of target ship trajectory prediction.

Preferably, in step 2, the website www.makesense.ai is used to mark the ship in the image. Then set the relevant parameters of the training model, in which the number of iterations epochs of the data set during the training process is set to 300 times; the initial learning rate lr0 is set to 0.01, and when the learning rate of the model reaches the preset initial value, the method of cosine descent is used to decrease, At the end of the training, it drops to 0.002; the batch number batch-size of each gradient update is set to 16, that is, after viewing 16 pictures at a time, a weight update is performed.

Preferably, in step 3, the calculation formulas of precision and recall are as follows:

Among them, TP indicates that the true category of the sample in the detection frame is correct, and the model prediction result is also correct, that is, both the detection target and the detection result are ship; TN indicates that the true category of the sample in the detection frame is wrong, and the model prediction result is also Wrong, that is, the detection target is not a ship and the detection result is a ship; FP indicates that the true category of the sample in the detection frame is wrong, but the model prediction result is correct, that is, neither the detection target nor the detection result is a ship; FN indicates that the detection frame The true category of the sample is correct, but the model prediction result is wrong, that is, the detection target is ship but the detection result is not ship. From the training results, it can be judged that the recognition performance of the model is excellent. If the average precision (mAP) is low, the image data set should be supplemented and marked, and the network model should be adjusted and retrained to obtain better recognition performance.

Preferably, in the step 5, the relative coordinates obtained through the calculation are preprocessed by mean filtering to make the trajectory smoother.

Preferably, in the step 7, multiple sets of experiments are carried out for the selection of the time step in the model training process, that is, the time step is set to n, and the azimuth coordinates at the n+1th moment are predicted, that is, input t-n+1 to The azimuth coordinates at time t, the coordinates at time t+1 are predicted, and the mean square error (MSE) is used to evaluate the trajectory prediction model of the target ship. The smaller the value of MSE, the higher the prediction accuracy of the LSTM model for the data used in this experiment. The specific calculation formula is as follows:

表示利用LSTM模型预测t时刻坐标点的位置，(x^(t)，y^(t))表示该时刻在t时刻实际坐标点的位置，T为样本数量。in,

Indicates the position of the coordinate point predicted by the LSTM model at time t, (x ^(t) , y ^(t) ) indicates the position of the actual coordinate point at time t at this time, and T is the number of samples.

The beneficial effects of the present invention are:

1. The present invention first completes the identification and tracking of the target ship in combination with YOLOv5 and Deep Sort algorithms. First of all, through the self-made target ship data set, select ship images under different types of sea and sky, and perform data enhancement processing, and then train the network model, and through the analysis of the training results, select a better network model and apply it to ship recognition track. This method can quickly and accurately detect and track ships at sea, which is of great significance for identifying dangerous behaviors of ships and improving the efficiency of maritime supervision.

2. The present invention performs visual positioning on the basis of identifying the target ship and utilizes the LSTM model to predict the track of the target ship. By building the LSTM network model and adjusting the parameters of the network model many times to obtain the ideal prediction effect, this ship navigation safety assessment method is accurate and intuitive, which is of great significance for ensuring the safety of maritime navigation.

Description of drawings

Fig. 1 is the target ship tracking algorithm flow chart based on Deep Sort algorithm among the present invention;

Fig. 2 is a schematic diagram after relative coordinate mean filtering among the present invention;

Fig. 3 is the internal schematic diagram of the LSTM unit in the present invention;

Fig. 4 is the comparison chart of LSTM predicted track and original visual positioning track in the present invention;

Fig. 5a-Fig. 5d are the target ship detection effect diagrams of different attitudes by YOLOv5 algorithm in the present invention;

Fig. 6a-Fig. 6d are the target ship tracking effect diagrams of different attitudes of YOLOv5 and Deep Sort algorithms in the present invention;

Fig. 7 is a flow chart of the middle body of the present invention.

detailed description

The specific implementation manner of the present invention will be described in detail below in conjunction with the accompanying drawings and preferred embodiments.

In describing the present invention, it should be understood that the terms "center", "longitudinal", "transverse", "upper", "lower", "front", "rear", "left", "right", The orientations or positional relationships indicated by "vertical", "horizontal", "top", "bottom", "inner", "outer", etc. are based on the orientation or positional relationships shown in the drawings, and are only for the convenience of describing the present invention Creation and simplification of description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operate in a specific orientation, and therefore should not be construed as limiting the invention. In addition, the terms "first", "second", etc. are used for descriptive purposes only, and should not be understood as indicating or implying relative importance or implicitly specifying the quantity of the indicated technical features. Thus, a feature defined as "first", "second", etc. may expressly or implicitly include one or more of that feature. In the description of the present invention, unless otherwise specified, "plurality" means two or more.

In the description of the present invention, it should be noted that unless otherwise specified and limited, the terms "installation", "connection", "connection", "fixed connection" and "fixed connection" should be understood in a broad sense, for example, it can be It can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection or an indirect connection through an intermediary, and it can be the internal communication of two components. Those of ordinary skill in the art can understand the specific meanings of the above terms in the present invention based on specific situations.

Below in conjunction with accompanying drawing, the patent of the present invention is further described.

1. Target ship identification and tracking method based on YOLOv5 and Deep Sort algorithm

In this patent, the YOLOv5 algorithm is selected to complete the identification of the target ship. This algorithm is a single-stage deep learning algorithm that uses a convolutional neural network for target detection. It is detected in a single forward propagation through the neural network, and has It has higher accuracy and faster detection than other target detection algorithms, which makes the YOLOv5 algorithm very popular in deep learning algorithms. In addition, the YOLOv5 algorithm has designed four different structures of S, M, L, and X to meet the needs of workers in different fields. YOLOv5s has the smallest network and the fastest calculation speed, but also the lowest average precision. The model size is only 14MB, which is suitable for embedded devices. The target tracking part uses the Deep Sort algorithm, which uses the detection results of YOLOv5 as input, predicts the future position of the detected target through the Kalman filter algorithm, and makes the targets better related, and then uses the Hungarian algorithm to identify whether the object in the current frame is Have the same association and ID attributes as the object in the previous frame to match to the same object between adjacent frames. The predecessor of the Deep Sort algorithm is the Sort algorithm. The Sort algorithm uses the IoU to perform matching between Detect and Track, but it only considers the distance matching between the bounding boxes, and does not consider the matching of content features, which easily leads to ID transformation. , in addition, when the target is lost and cannot be retrieved, this method can only re-update the ID through detection, which cannot meet the basic needs of target tracking. In contrast, Deep Sort uses two indicators to evaluate the similarity, which are the target's motion state information and appearance information. The Deep Sort algorithm uses the result of the fusion of these two indicators as the final judgment basis to improve the tracking failure after the target is occluded and the problems of a large number of ID changes. The identification and tracking methods for the target ship are as follows:

(1) Construct target ship image dataset

Before identifying the target ship, it is necessary to construct an image data set of the target ship, and then use the YOLOv5 network model for training. In order to improve the robustness of system identification, the image data set should include the sea and sky types of various conditions during the actual voyage of the target ship, such as rain, snow, strong wind and waves, etc. should be considered. In order to improve the generalization performance of the framework, this system employs data augmentation techniques to generate more target ship images using commonly used augmentation operations. Specifically, through data enhancement techniques (translation, rotation, color transformation, scale transformation, etc.), 20 different ship images can be obtained for each input training image. The system generates 20 ship visual features (edge, Outline, color, etc.) are hand-selected samples of ship variants in images that are significantly different from the original samples. 70% of the pictures are randomly selected as the training set, and the remaining pictures are used as the verification set.

(2) Dataset labeling and training

After sorting out all the image data, it is necessary to label all the ship images in the training set and verification set, and use the www.makesense.ai website to label the ships in the images. Then set the relevant parameters of the training model, in which the number of iterations epochs of the data set during the training process is set to 300 times; the initial learning rate lr0 is set to 0.01, and when the learning rate of the model reaches the preset initial value, the method of cosine descent is used to decrease, At the end of the training, it drops to 0.002; the batch number batch-size of each gradient update is set to 16, that is, after viewing 16 pictures at a time and performing a weight update, the model can have a better recognition effect. As shown in Figure 5a-5d.

(3) Evaluation and analysis of model training results

When evaluating the performance of network model training results, the average precision mAP is often used, which mainly calculates the precision Precision and recall rate Recall of all categories, and calculates the average value of AP of all categories.

Among them, the calculation formulas of precision and recall are as follows:

Among them, TP indicates that the true category of the sample in the detection frame is correct, and the model prediction result is also correct, that is, both the detection target and the detection result are ship; TN indicates that the true category of the sample in the detection frame is wrong, and the model prediction result is also Wrong, that is, the detection target is not a ship and the detection result is a ship; FP indicates that the true category of the sample in the detection frame is wrong, but the model prediction result is correct, that is, neither the detection target nor the detection result is a ship; FN indicates that the detection frame The true category of the sample is correct, but the model prediction result is wrong, that is, the detection target is ship but the detection result is not ship. From the training results, it can be judged that the recognition performance of the model is excellent. If the average precision (mAP) is low, the image data set should be supplemented and marked, and the network model should be adjusted and retrained to obtain better recognition performance.

(4) Target ship tracking experiment based on YOLOv5 and Deep Sort algorithm

The main design idea of the ship detection and tracking proposed by this system is to integrate the YOLOv5-based target detector with the Deep Sort algorithm to realize the continuous tracking of the custom target ship, and output the pixel coordinates of the target in the tracking video, which is convenient The next step is to calculate the orientation of the target ship. The design process is shown in Figure 1 and Figure 6a-6d.

2. Target ship trajectory prediction method based on LSTM algorithm

(1) Target ship visual positioning and data preprocessing

Based on the principle of homogeneous coordinate transformation and the pinhole imaging model, the conversion model of the pixel coordinates of the target ship and the world coordinates (the origin of the world coordinates can be customized, here we assume that the optical center of the camera is the origin of the world coordinates) is established. Through this process, the pixel coordinates of the target ship obtained by tracking above can be converted into relative coordinates with the optical center of the camera as the origin.

Due to camera shake, frame size error during manual labeling, and background interference on the horizontal plane, there may be a small positional deviation between the size of the target detection frame and the outline of the target ship, resulting in an incorrect estimation of the image coordinates of the tracking target ship, which in turn will reduce the relative coordinates. solution accuracy. We use mean filtering to preprocess the calculated relative coordinates to make the trajectory smoother, so as to reduce the influence of external factors on the relative coordinate solution accuracy, as shown in Figure 2.

(2) Construction of LSTM trajectory prediction model

Since ships are easily affected by external environmental factors during navigation, their trajectories have nonlinear characteristics. It is difficult for traditional mathematical equations to consider the influence of sea wind and water currents on the ship's motion state. Therefore, here we choose neural networks to carry out predict. As we all know, the cyclic neural network (RNN) has problems such as gradient disappearance, gradient explosion, and difficulty in learning long-term patterns, and is usually difficult to train. The LSTM model can solve this problem well and is easier to train. Therefore, this system chooses LSTM network for target ship trajectory prediction.

The model consists of an input layer, five hidden layers (LSTM layer, Dropout layer, and Dense layer) and an output layer. The input dimension of the input sequence of the input layer is 2 (that is, the relative coordinates converted from the pixel coordinates above); the output dimension of the output sequence of the output layer is 2 (same as above). The hidden layer consists of 2 LSTM layers, 2 Dropout layers and 1 Dense layer; according to experience, the probability of the Dropout layer is set to 0.3 to prevent over-fitting; the parameter of the Dense layer is set to 2, that is, the final The output of the result is the predicted relative position information of the target ship. In order to introduce a certain nonlinearity, the ReLU activation function is selected here as the activation function of the Activation layer, which is set between the hidden layer and the output layer. The internal principle of the LSTM unit is shown in Figure 3.

(3) Selection and evaluation of the time step of the prediction model

Here we select the first 80% of the relative coordinate information calculated above as the training set, and the remaining 20% as the verification set for model training. Generally speaking, the different time steps of the input layer will also have a certain impact on the accuracy of the algorithm's prediction. The larger the input dimension, the more likely the model will overfit. In the process of model training, multiple sets of experiments are carried out for the selection of the time step, that is, the time step is set to n, and the azimuth coordinates at the n+1th time are predicted, that is, the azimuth coordinates at the time t-n+1 to t are input, and the prediction t+ 1 time coordinates. The mean square error (MSE) is used to evaluate the trajectory prediction model of the target ship. The mean square error refers to the expected value of the square of the difference between the predicted value of the ship coordinates and the real coordinate value of the ship. The smaller the value of MSE, the better the LSTM model is for the data used in this experiment. The prediction accuracy is high, and the specific calculation formula is as follows:

表示利用LSTM模型预测t时刻坐标点的位置，(x^(t)，y^(t))表示该时刻在t时刻实际坐标点的位置，T为样本数量。in,

Indicates the position of the coordinate point predicted by the LSTM model at time t, (x ^(t) , y ^(t) ) indicates the position of the actual coordinate point at time t at this time, and T is the number of samples.

(4) Target ship trajectory prediction and verification analysis

Use the best prediction model selected in the previous step to predict the relative trajectory information of the target ship, and verify it with the original visual positioning trajectory, analyze the error of the model prediction, if the ideal prediction accuracy is not reached, continue to adjust the network model Parameters, retraining to improve the accuracy of target ship trajectory prediction. As shown in Figure 4.

Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present invention, rather than limiting them; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that: It is still possible to modify the technical solutions described in the foregoing embodiments, or perform equivalent replacements for some or all of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the technical solutions of the various embodiments of the present invention. scope.

Claims (5)

1. A ship track tracking and predicting method based on a monocular camera is characterized by comprising the following steps:

step 1: a target vessel image data set is constructed,

generating more target ship images by adopting a data enhancement technology and utilizing common enhancement operation;

step 2: the labeling and training of the data set are carried out,

after finishing sorting all the image data, marking all the ship images in the training set and the verification set;

and step 3: the model training result is evaluated and analyzed, and the model training result is analyzed,

calculating Precision and Recall rate Recall of all categories by adopting an average Precision mean mAP, and calculating an average value of all categories of APs;

and 4, step 4: based on the target ship tracking experiment of the YOLOv5 and Deep Sort algorithm,

fusing a target detector based on YOLOv5 with a Deep Sort algorithm to realize continuous tracking of a user-defined target ship, and outputting pixel coordinates of a target in a tracking video to facilitate next-step target ship azimuth calculation;

and 5: the visual positioning and data preprocessing of the target ship,

establishing a conversion model of the pixel coordinates and the world coordinates of the target ship based on a homogeneous coordinate conversion principle and a small-hole imaging model, and converting the pixel coordinates of the target ship obtained by tracking in the step 4 into relative coordinates taking the optical center of the camera as an origin through the process;

and 6: building an LSTM track prediction model,

selecting an LSTM network to predict the track of the target ship, wherein the model consists of 1 input layer, five hidden layers and an output layer, the input dimensionality of an input sequence of the input layer is 2, namely the relative coordinate converted from the pixel coordinate in the step 5; the output dimension of an output sequence of the output layer is 2, and the hidden layer consists of 2 LSTM layers, 2 Dropout layers and 1 Dense layer; the probability of setting Dropout layer to 0.3 is empirically set for preventing the occurrence of the overfitting phenomenon; setting the parameter of a Dense layer as 2, namely outputting the final result as predicted relative position information of a target ship, and selecting a ReLU Activation function as an Activation function of an Activation layer to introduce certain nonlinearity, wherein the ReLU Activation function is arranged between a hidden layer and an output layer;

and 7: selecting and evaluating the time step of the prediction model,

selecting the first 80% of the relative coordinate information calculated in the step 6 as a training set, and the remaining 20% as a verification set, and training the model;

and 8: the target ship track is predicted, verified and analyzed,

and (4) predicting the relative track information of the target ship by using the optimal prediction model selected in the step (7), verifying the original visual positioning track, analyzing the error of model prediction, and if the ideal prediction precision is not reached, continuously adjusting parameters of the network model and retraining so as to improve the precision of the target ship track prediction.

2. The vessel trajectory tracking and prediction method based on the monocular camera as recited in claim 1, wherein in the step 2, the vessel in the image is labeled by using www. Then setting relevant parameters of the training model, wherein the iteration times epochs of the data set in the training process are set to be 300 times; the initial learning rate lr0 is set to 0.01, when the model learning rate reaches a preset initial value, the model learning rate is reduced in a cosine reduction mode, and when the training is finished, the model learning rate is reduced to 0.002; the batch size of each gradient update is set to 16, namely, the weight update is performed once after 16 pictures are seen at a time.

3. The vessel track tracking and predicting method based on the monocular camera according to claim 1, wherein in step 3, the calculation formulas of the precision and the recall rate are as follows:

wherein TP indicates that the real category of the sample in the detection frame is correct and the model prediction result is also correct, namely the detection target and the detection result are both ship; TN indicates that the true class of the sample in the detection frame is wrong, and the model prediction result is also wrong, namely the detection target is not ship and the detection result is ship; FP means that the real class of the sample in the detection frame is wrong, and the model prediction result is correct, namely, neither the detection target nor the detection result is ship; FN indicates that the true class of the sample in the detection box is correct, while the model prediction result is wrong, i.e. the detection target is ship and the detection result is not ship. The training result can judge that the model recognition performance is excellent, if the average precision mean mAP is low, the image data set is supplemented and labeled, the network model is subjected to parameter adjustment, and retraining is carried out to obtain good recognition performance.

4. The ship track tracking and predicting method based on the monocular camera according to claim 1, wherein in the step 5, the calculated relative coordinates are preprocessed by using mean filtering to make the track smoother.

5. The monocular camera-based vessel trajectory tracking and prediction method of claim 1, further comprising: in the step 7, multiple sets of experiments are performed for selecting the time step in the model training process, that is, the time step is set to be n, the azimuth coordinate at the n +1 th moment is predicted, that is, the azimuth coordinate from the t-n +1 to the t moment is input, the coordinate at the t +1 moment is predicted, and a track prediction model of the target ship is estimated by using a Mean Square Error (MSE), wherein the MSE refers to an expected value of the square of the difference between a ship coordinate prediction value and a ship real coordinate value, and the smaller the MSE value is, which indicates that the LSTM model has higher accuracy for the data used in the experiment, and a specific calculation formula is as follows:

wherein,

indicating the prediction of the position of the coordinate point at time t using the LSTM model, (x) ^(t) ，y ^(t) ) Which represents the position of the actual coordinate point at time T, where T is the number of samples.

Images