CN112949634B - A method for detecting bird nests in railway contact network - Google Patents

Abstract

The invention provides a method for detecting a railway contact net nest. The method comprises the following steps: obtaining an interest domain picture containing a bird nest region through reverse reasoning according to the picture containing the bird nest, taking the interest domain picture as a template picture, forming a template library according to all the template pictures, and training a second-stage YOLO detector by using the template library; sequentially matching the pictures which do not contain the bird nest with each template picture in the template library to obtain an interest domain picture data set, and training a first-stage YOLO detector by using the interest domain picture data set; the picture to be detected is input into a first-stage YOLO detector after training, the first-stage YOLO detector outputs a region-of-interest picture, the region-of-interest picture is input into a second-stage YOLO detector after training, and the second-stage YOLO detector outputs a bird nest detection result of the picture to be detected. The invention can solve the problem of difficult recognition caused by the lack of obvious characteristics due to less bird nest information in the overhead contact system, and can effectively and automatically recognize and detect the railway overhead contact system.

Description

A method for detecting bird nests in railway contact network

Technical Field

The invention relates to the technical field of railway contact network foreign body detection, and in particular to a railway contact network bird nest detection method.

Background Art

At present, the research on target identification has become one of the most popular hot topics in the field of computer vision detection. The marking of targets in dynamic videos is mainly achieved by analyzing the image sequence collected by the image sensor, extracting the target scenes of interest in the image sequence, marking the pixel area of the same target, and identifying the position, size and profile of the target. Typical target marking and recognition methods include target feature description, feature information extraction and target feature matching. By extracting the feature information of the target to be represented, such as position, color, contour, texture, etc., and then evaluating the detection target based on these feature information, it is determined whether the target can match the feature information to complete the target labeling.

At present, the detection methods of foreign bodies on high-speed railway contact networks in the prior art include: detection based on the Faster R-CNN detection model, detection based on relative position invariance, and detection of bird nests on railway contact networks using HOG (Histogram of Oriented Gradient) features. Among them, the detection based on the Faster R-CNN detection model introduces an RPN (Region Proposal Network) to generate candidate regions for the target object. Faster R-CNN can be regarded as consisting of an RPN network that generates target candidate regions and a Faster R-CNN detection that uses these candidate regions for prediction and classification. First, an image is input and propagated forward to the last shared convolutional layer. On the one hand, the obtained feature map is transmitted to the RPN network, and on the other hand, it continues to propagate forward to generate a higher-dimensional feature map. In the RPN network, after a series of processing, multiple candidate regions and candidate region scores are generated, and non-maximum filtering is used for the candidate region frame to reduce the number of candidate regions. The candidate regions larger than the set threshold and the previously generated high-dimensional feature map are input into the RoI pooling layer to extract the features of the corresponding regions. Finally, the regional features are connected to the fully connected layer to output the target classification and score, as well as the bounding box target positioning box regression value.

The detection based on relative position invariance utilizes machine vision processing technology. After a preliminary analysis of the color, texture and shape features of the image, combined with the characteristics of the bird's nest platform, the preprocessed detection image is used to obtain the image edge using the Sobel horizontal edge detection operator. The probabilistic Hough transform straight line detection method is then used to correct the image angle. The hard beam in the foreground is detected in combination with the relationship between the line segment lengths in the image to be analyzed. Finally, the image is binarized and the area of the white area between the hard beams is counted to determine whether there is a bird's nest on the beam.

The detection of bird nests on the railway contact network using HOG features is to roughly extract the areas where bird nests may appear in the picture based on prior knowledge, then find the HOG features of the extracted area, and finally accurately identify the bird nests in the picture based on the HOG features of the picture through support vector machines (SVM). Because neural networks have unparalleled advantages over other algorithms in image processing, some domestic scholars have combined neural networks with traditional image processing technology to detect targets of interest in the picture. This method can effectively improve the speed and accuracy of detection.

The disadvantages of the above-mentioned detection method of foreign bodies in the high-speed railway contact network in the prior art are: in the actual operation process of bird nest detection, due to the various changes in the operating environment of the train and the fact that the bird nests of various shapes are located in the complex contact network, the accuracy and recall rate of the recognition system built based on the HOG and DMP models are far from the expected standards. This is because traditional recognition models such as HOG and DPM use artificially extracted features as detection templates and then perform sliding frame matching detection. This type of method is easily affected by shape and texture features, so it is difficult to extract standard detection features from bird nests in this environment.

In addition, due to the need for manual feature extraction, it is even more difficult to achieve accurate identification when the amount of data is insufficient. As a result, it has disadvantages such as narrow application and lack of universality, which will bring many problems to the identification of bird nests in high-speed railway contact networks.

Summary of the invention

The embodiment of the present invention provides a method for detecting bird nests on a railway contact network, so as to realize effective automatic bird nest identification and detection on the railway contact network.

In order to achieve the above object, the present invention adopts the following technical scheme.

A method for detecting bird nests in a railway contact network, comprising:

According to the pictures containing bird nests in the railway contact network picture data set, the interest domain pictures containing the bird nest area are obtained by reverse reasoning, the interest domain pictures are used as template pictures, a template library is formed according to all template pictures, and the template library is used to train the second-level YOLO detector;

Matching the pictures that do not contain bird nests in the railway contact network picture dataset with each template picture in the template library in sequence to obtain a domain of interest picture dataset, and using the domain of interest picture dataset to train the first-level YOLO detector;

The picture to be detected is input into the trained first-level YOLO detector, and the first-level YOLO detector outputs the interest domain picture. The interest domain picture is input into the trained second-level YOLO detector, and the second-level YOLO detector outputs the bird nest detection result of the picture to be detected.

Preferably, the image of the domain of interest containing the bird's nest area is obtained by reverse reasoning based on the image containing the bird's nest in the railway contact network image data set, the image of the domain of interest is used as a template image, and a template library is formed based on all the template images, including:

The pictures containing bird nests in the railway contact network picture dataset are preliminarily segmented to obtain basic areas with set similarities. The basic areas are preliminarily merged according to the differences between the areas to obtain a series of preliminary candidate areas. The preliminary candidate areas are surrounded by rectangular frames. The preliminary candidate areas are merged according to the similarities between the rectangular frames to obtain final candidate areas. The bird nest positions in the final candidate areas are manually marked, and the marked bird nest area attributes are represented by rectangles. The final candidate area containing the bird nest area is used as the interest domain, and the interest domain image is used as the template image. A template library is constructed based on all template images.

Preferably, the pictures containing bird nests in the railway contact network picture data set are preliminarily segmented to obtain basic regions with set similarities, and the basic regions are preliminarily merged according to the differences between the regions to obtain a series of preliminary candidate regions, including:

A picture containing a bird's nest is represented by an undirected graph G = <V, E>, where the vertex of the undirected graph represents a pixel point in the picture, the weight of the edge e = ( _vi , _vj ) represents the dissimilarity between adjacent vertices i and j, and the color distance of the pixel is used Represents the dissimilarity w(e) between two pixels, and a basic region is the set of points with the minimum dissimilarity;

The intra-class variance of the basic region is defined as:

The inter-class difference between two basic regions C ₁ and C ₂ is defined as the minimum connecting edge between the two basic regions:

If two basic regions are not connected by edges, Diff(C ₁ , C ₂ ) = ∞

When the condition Diff(C ₁ , C ₂ )≤min(Int(C ₁ )+τ(C ₁ ), Int(C ₂ )+τ(C ₂ )) is satisfied, it is determined that the two basic regions C ₁ and C ₂ can be merged;

Where τ(C) is the threshold function, which makes the area composed of isolated points have weight:

τ(C)＝k/||C||

Each basic region is preliminarily merged to obtain a series of preliminary candidate regions.

Preferably, the step of enclosing the preliminary candidate regions with rectangular frames, merging the preliminary candidate regions according to the similarity between the rectangular frames, and obtaining the final candidate regions comprises:

The preliminary candidate area is enclosed by a rectangular frame, and the position of the rectangular frame C is represented by a four-tuple (x, y, w, h), where x and y represent the coordinates of the upper left corner of the rectangular frame, and w and h represent the width and height of the rectangular frame;

The color distance between the rectangular box _ci of the preliminary candidate region _ri and the rectangular box _cj of the preliminary candidate region _rj is:

In the formula Represents the pixel ratio of the kth bins of the color histogram;

The texture distance between the rectangular box c _i of the preliminary candidate region _ri and _the rectangular _box c _j of the preliminary candidate region r _j is:

In the formula Indicates the ratio of pixels in the kth dimension of the texture histogram;

For the preliminary candidate regions _ri and _rj :

Where size( _ri ) represents the size of the rectangular box corresponding to region _ri , and size(im) represents the size of the original image to be segmented.

For the preliminary candidate regions _ri and _rj :

Where size(BB _ij ) represents the size of the bounding rectangle of regions _ri and _rj ;

The total difference between the preliminary candidate regions _ri and _rj is:

S (r _i , r _j ) = a ₁ S _color (r _i , r _j ) + a ₂ S _texture (r _i , r _j ) + a ₃ S _size (r _i , r _j ) + a ₄ S _fill ( r _i , r _j )

a ₁ , a ₂ , a ₃ , a ₄ are the corresponding weight values;

When the total difference S( _ri , _rj ) between the preliminary candidate regions _ri and _rj is greater than the set merging threshold, the preliminary candidate regions _ri and _rj are merged to obtain the final candidate region.

Preferably, the bird nest positions in the final candidate area are manually annotated, and the annotated bird nest area attributes are represented by rectangles. The final candidate area containing the bird nest area is used as the domain of interest, and the domain of interest image is used as the template image. A template library is formed according to all template images, including:

The final candidate area C is represented by a rectangle, and the position attribute of the rectangle is represented by a four-tuple (x, y, w, h);

The bird nest position in the final candidate area is marked, and the marked bird nest area attributes are represented by a rectangle. The position attributes of the rectangle are (bx, by, bw, bh). The region of interest is the candidate area containing the bird nest area, and the position coordinates of the region of interest satisfy:

And the threshold condition is met:

The interest domain picture is used as a template picture, and a template library is constructed based on all template pictures.

Preferably, the pictures not containing bird nests in the railway contact network picture data set are matched with each template picture in the template library in sequence to obtain the interest domain picture data set, including:

Match the to-be-matched images that do not contain bird nests in the railway contact network image dataset with each template image in the template library in turn. Let the template image be T, the to-be-matched image that does not contain bird nests be I, let the width of the template image be w, the height be h, R represents the matching result, and the matching method is expressed as follows:

Where:

The larger the R value is, the higher the similarity between the rectangular area of the image to be matched at the position (x, y) and the size (w, h) is with the template. The maximum template similarity is taken as the template matching result, and the template matching value is required to be higher than the threshold parameter.

Note Rs(T,I)=max _x,y∈I R(x,y)

Each template image corresponds to a best matching value R. The position of the rectangular matching box corresponding to R is (x, y, w, h). The initial template matching results constitute the result set S:

Where c is the matching threshold parameter;

Arrange the result set S in descending order by Rs value. The condition for the intersection of two rectangular matching boxes s and t is:

max(x(s),x(t))≤min(x(s)+w(s),x(t)+w(t))

max(y(s),y(t))≤min(y(s)+h(s),y(t)+h(t))

Traverse the result set S in sequence. If the current rectangular matching box intersects with the annotated rectangular matching box, the annotation is abandoned. Otherwise, the current rectangular matching box is annotated in VOC format. All annotated rectangular matching boxes constitute the interest domain dataset.

Preferably, the first-stage YOLO detector and the second-stage YOLO detector include: YOLOv3-spp, YOLOv4 and Faster R-CNN.

Preferably, the confidence and expectation of the first-stage YOLO detector are as follows:

Where Pr(Zone) is the probability that the current grid has the object to be tested (domain of interest). During the training process, if the grid to be tested contains the domain of interest, Pr(Zone) is 1, otherwise it is 0. is the intersection-over-union ratio of the grid-predicted annotation box and the rectangular box where the interest domain is actually located, B is the number of annotation boxes predicted for each grid, S ² is the total number of grids divided into the image, The average IOU of all predicted boxes predicted for all grids where the object is located, I(Zone) is the size of the region of interest, I(image) is the size of the original image, and E(Zone) is the sum of the total IOU given by the image;

The confidence of the second-level YOLO detector is:

Where Pr(Birdnest) is the probability that the current grid has a bird nest. If the grid to be tested contains a bird nest, Pr(Birdnest) is 1, otherwise it is 0. is the intersection-and-union ratio of the grid-predicted annotation box and the rectangular box where the bird's nest actually is. P (distribution) is the probability that the bird's nest in the image exists in the domain of interest. Obviously, all bird's nests exist in the domain of interest, and this item is 1.

The prediction expectation of bird nests in the domain of interest is:

Where Pr _i (birdnest) is the probability that the i-th grid in the sub-image contains a bird nest in the sub-image of the interest domain. is the intersection-over-union ratio of the jth rectangular box predicted by the i-th grid of the sub-image and the bird's nest image, confidence(Zone) is the confidence of the interest region marked by a rectangular box predicted by the original image grid, A prediction box representing the division grid of the original image can mark the degree of certainty of the bird's nest;

The expectation of the cascade prediction is:

In the formula is the average intersection-over-union ratio of the predicted annotation box contained in the grid of the bird's nest in the sub-image of the interest domain and the rectangle where the bird's nest is located, where Represents the average IOU value of the grid prediction anchor box;

The accuracy of cascade prediction is:

P=F(birdnest, Zone, N)*F(Zone, image, M)>F(birdnest, image, N).

It can be seen from the technical solutions provided by the above-mentioned embodiments of the present invention that the method for automatic identification and rapid tracking of bird nests on the high-speed railway contact network proposed in the embodiments of the present invention can effectively solve the problem of accurate and rapid identification and tracking of bird nests on the high-speed railway contact network; it can effectively solve the identification difficulties caused by the small amount of bird nest information in the contact network and the lack of obvious shape or texture features; thereby enabling effective automatic identification and detection of bird nests on the railway contact network.

Additional aspects and advantages of the present invention will be given in part in the following description, which will become obvious from the following description, or may be learned through practice of the present invention.

BRIEF DESCRIPTION OF THE DRAWINGS

In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

FIG1 is a schematic diagram showing a method for detecting bird nests in a railway overhead contact network according to an embodiment of the present invention;

FIG2 is a processing flow chart of a method for detecting bird nests in a railway contact network provided by an embodiment of the present invention;

FIG3 is a flowchart of a process for performing preliminary segmentation and merging of images to generate basic regions provided by an embodiment of the present invention;

FIG. 4( a ) is a schematic diagram of an original image provided by an embodiment of the present invention, and FIG. 4( b ) is a schematic diagram of a series of basic regions obtained after preliminary segmentation and merging.

FIG5 is a flowchart of a process for merging basic regions according to differences between regions to obtain a series of candidate regions provided by an embodiment of the present invention;

FIG6 is a schematic diagram of marking a region of interest with a rectangular frame according to an embodiment of the present invention;

FIG7 is a processing flow chart of template matching between a to-be-detected image and a template image provided by an embodiment of the present invention;

FIG8(a) is a schematic diagram of an original picture provided by an embodiment of the present invention, and FIG8(b) is a schematic diagram of template matching of the original picture;

FIG9 is a schematic diagram of dividing a picture into S×S grids provided by an embodiment of the present invention;

FIG10 is a schematic diagram of a single-stage network prediction provided by an embodiment of the present invention;

FIG11 is a schematic diagram of a first-level network prediction of a cascade network provided by an embodiment of the present invention;

FIG12 is a schematic diagram of a second-level network prediction of a cascade network provided by an embodiment of the present invention;

FIG13 is a schematic diagram of an IOU curve in a YOLOv3-GIOU training process provided by an embodiment of the present invention;

FIG14 is a schematic diagram of an IOU curve in a Yolov4-CIOU training process provided by an embodiment of the present invention;

FIG15 is a schematic diagram of an IOU comparison between a YOLOv3-SPP second-level detection and a direct detection provided by an embodiment of the present invention;

FIG16 is a schematic diagram of an IOU comparison between a YOLOv4 second-level detection and a direct detection provided by an embodiment of the present invention.

DETAILED DESCRIPTION

The embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and cannot be interpreted as limiting the present invention.

It will be understood by those skilled in the art that, unless expressly stated, the singular forms "one", "said", and "the" used herein may also include plural forms. It should be further understood that the term "comprising" used in the specification of the present invention refers to the presence of the features, integers, steps, operations, elements and/or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and/or groups thereof. It should be understood that when we refer to an element as being "connected" or "coupled" to another element, it may be directly connected or coupled to the other element, or there may be intermediate elements. In addition, the "connection" or "coupling" used herein may include wireless connection or coupling. The term "and/or" used herein includes any unit and all combinations of one or more associated listed items.

It will be understood by those skilled in the art that, unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as those generally understood by those skilled in the art in the art to which the present invention belongs. It should also be understood that terms such as those defined in common dictionaries should be understood to have meanings consistent with the meanings in the context of the prior art, and will not be interpreted with idealized or overly formal meanings unless defined as herein.

To facilitate understanding of the embodiments of the present invention, several specific embodiments will be further explained below with reference to the accompanying drawings, and each embodiment does not constitute a limitation on the embodiments of the present invention.

The method based on convolutional neural network of the present invention uses convolution kernel as feature extractor, and the picture can be directly used as the input of the network. The bird nest features obtained through training avoid the complicated feature extraction and data reconstruction process in the traditional recognition algorithm, so the accuracy and recall rate can be significantly improved.

The present invention first uses the Selective Search algorithm to cluster multiple candidate regions, and finds the domain of interest from the candidate regions according to the location of the bird's nest. Template matching is performed on all the obtained domains of interest in the picture set, and the domains of interest of all pictures are marked. A YOLO network is constructed to perform recognition training on the domains of interest of all pictures, and a YOLO network is constructed to perform recognition training on the bird's nest in the domain of interest. The first level recognition is performed on the unknown picture sample to find the domain of interest, and then the second level recognition is performed to find the bird's nest from the domain of interest. This cascade recognition can greatly improve the recognition accuracy and improve the calculation efficiency.

Embodiment 1

The implementation principle diagram of a railway contact network bird nest detection method provided by an embodiment of the present invention is shown in FIG1 , and the processing flow is shown in FIG2 , which includes the following processing steps:

Step S210: obtaining a domain of interest image containing a bird nest area through reverse reasoning based on images containing bird nests in the railway contact network image dataset, using the domain of interest image as a template image, and constructing a template library based on all template images.

The domain of interest is a region, and a region is a whole with high similarity. First, we need to find all the regions with high similarity in the image. Finding regions is the process of segmenting and merging images.

First, the images containing bird nests in the railway contact network image data set are preliminarily segmented to obtain a large number of basic regions with similarities. Then the basic regions are merged to obtain a series of candidate regions. A processing flow chart for preliminarily segmenting and merging images to generate basic regions provided by an embodiment of the present invention is shown in Figure 3. The processing process includes: a picture can be represented by an undirected graph G = <V, E>, where the vertex of the undirected graph represents a pixel point of the picture, and the weight of the edge e = (v _i , v _j ) represents the dissimilarity of adjacent vertex pairs i and j. The color distance of the pixels can be used The equal pixel attribute represents the dissimilarity w(e) between two pixels. A basic region is a point set with the minimum dissimilarity, so the basic region is the minimum spanning tree containing the point set. The initial segmentation of the image is to find the forest composed of the minimum spanning tree of the image.

The difference determines whether the basic regions are merged. The intra-class difference of the basic regions is defined as:

Where C represents a basic region in the merging process, e represents the connecting edge inside the region, its weight represents the dissimilarity between pixels, and the intra-class difference Int(C) is the maximum value of the connecting edge inside the region.

The difference between classes is defined as the minimum connecting edge between two basic regions:

In the formula, C ₁ and C ₂ represent two different regions, _vi and v _j represent two vertices of the connecting edge between two different regions, and the inter-class difference Diff(C ₁ , C ₂ ) is the minimum value of the connecting edge connecting the two regions.

In particular, if two basic regions are not connected by edges, Diff(C ₁ , C ₂ ) = ∞

So we get the basis for merging two basic regions when the conditions are met:

Diff(C ₁ , C ₂ )≤min(Int(C ₁ )+τ(C ₁ ), Int(C ₂ )+τ(C ₂ ))

Then it is judged that the basic areas _C1 and _C2 can be merged

Where τ(C) is the threshold function, which makes the area composed of isolated points have weight:

τ(C)＝k/||C||

In the formula, k is a manually set parameter, ||C|| is the number of vertices in the basic area C, and the size of the algorithm segmentation area can be controlled by adjusting the size of k.

In this way, the image point sets are merged to obtain multiple basic regions. Figure 4 (a) is a schematic diagram of an original image provided by an embodiment of the present invention, and Figure 4 (b) is a schematic diagram of a series of basic regions obtained after preliminary segmentation and merging.

FIG5 is a flowchart of a process for merging basic regions according to differences between regions to obtain a series of candidate regions provided by an embodiment of the present invention. The specific processing process includes: merging the basic regions formed above, and inferring the domain of interest where the bird's nest is located. The basic regions and their merged results are represented by rectangles, and the position of the rectangle can be represented by a four-tuple of (x, y, w, h). Where x, y represent the coordinates of the upper left corner of the rectangular box, and w, h represent the width and height of the rectangular box.

For area C, the corresponding rectangular position attribute calculation method is as follows:

First, the differences between regions are calculated. The differences between regions can be evaluated by four evaluation indicators:

Color distance, determined by the minimum value of each bin of the color histogram of the two regions

The number of pixels in each bin of the rectangular image of different color channels corresponding to the statistical area _ri is obtained to obtain the n-dimensional color histogram, where Represents the pixel ratio of the kth bins of the color histogram.

The texture distance is determined by the minimum value of each bin of the fast SIFT feature histogram of the two regions.

The n-dimensional texture histogram is obtained by counting the number of pixels of each SIFT feature in each bin of different color channels of the rectangular image corresponding to the region _ri , where Indicates the ratio of pixels in the kth dimension of the texture histogram.

Prioritize the merging of small areas and give higher merging weights to small areas.

Where size( _ri ) represents the size of the rectangular image corresponding to region r _i , and size(im) represents the size of the original image to be segmented.

Prioritize merging areas with large overlapping areas of the bounding rectangles.

Where size(BB _ij ) represents the size of the bounding rectangle of regions _ri and _rj . The other parameters have the same meaning as above.

Weighting the above differences, we get the total difference between regions:

S (r _i , r j ) = a ₁ S _color (r _i , r _j ) + a ₂ S _texture (r _i , r _j ) + a ₃ S _size (r _i , r _j ) + a ₄ S _fill (r _i , _rj )

a ₁ , a ₂ , a ₃ , a ₄ are the corresponding weight values.

The previous conditional segmentation is a segmentation of the original image to be segmented based on pixel differences. It gives which basic area each pixel belongs to, and the basic area is a continuous set of pixels with irregular shapes. The segmentation performed by the judgment condition is based on the result of the first basic segmentation. The second segmentation first uses a rectangular frame to enclose the irregular point set of the first segmentation to obtain a series of rectangular areas. The range of the rectangular area is larger than the enclosed basic area. These rectangular areas need to be judged more strictly for similarity again, and then these rectangular areas are merged according to the similarity to obtain the final candidate area. The candidate area containing the bird's nest is the area of interest to be found. The first segmentation only performs a rough segmentation based on whether the pixels are "similar", while the second segmentation adds "size", "features", "shape" and other considerations on the basis of the first segmentation to obtain the final rectangular segmentation area containing the bird's nest.

The bird nest locations in the input data set are manually annotated. The annotated bird nest area attributes are still represented by rectangles, and its location attributes are set to (bx, by, bw, bh). The region of interest is the candidate region containing the bird nest area, and its location coordinates should satisfy:

To prevent over-merging, the domain of interest should also meet the threshold conditions:

The candidate area where the bird's nest is located is the inferred interest area, as shown in Figure 6, where the rectangular box marks the inferred interest area.

Step S220: matching the pictures that do not contain bird nests in the railway contact network picture dataset with each template picture in the template library in sequence to obtain a domain of interest picture dataset.

FIG7 is a processing flow chart of template matching between a to-be-detected image and a template image provided by an embodiment of the present invention. The specific processing process includes: using the image of the interest domain derived by the above inference as a template element image, and all template element images constitute a template library. The above template library is used to perform template matching on the images that do not contain bird nests in the railway contact network image dataset, that is, traversing each image, and using the normalized correlation coefficient to match and mark all interest domains of the to-be-detected image.

Let the template image be T, the image to be matched that does not contain the bird's nest be I, let the width of the template image be w, the height be h, and R represent the matching result. Then the matching method can be expressed as follows:

Where:

The larger the R value is, the higher the similarity between the rectangular area of the image to be matched at the position (x, y) and the size (w, h) is with the template, and the maximum template similarity is taken as the template matching result. And the template matching value is required to be higher than the threshold parameter.

Note Rs(T,I)=max _x,y∈I R(x,y)

Template matching first matches the image to be matched with each template image in the template library in turn. Each template image corresponds to a best matching value R, and the position of the rectangular matching box corresponding to R is (x, y, w, h). The initial template matching results constitute the result set S:

Where c is the matching threshold parameter.

The result set S is sorted in descending order by Rs value. The rectangular matching boxes in the result set may intersect.

For two rectangular matching boxes s and t, the condition for the intersection of rectangles is:

max(x(s),x(t))≤min(x(s)+w(s),x(t)+w(t))

max(y(s),y(t))≤min(y(s)+h(s),y(t)+h(t))

Traverse the result set S in sequence. If the current rectangular matching box intersects with the annotated rectangular matching box, the annotation is abandoned. Otherwise, the current rectangular matching box is annotated in VOC format. All annotated rectangular matching boxes constitute the interest domain data set. The best matching result obtained by the template matching algorithm is a rectangular box, and its matching result is (x, y, w, h). For a single-class object, the format of its label.txt file is Oxywh (one line). One line represents a rectangular annotation box. Then use the txttoxml script to convert the txt file annotation format into an xml file in VOC format for deep learning training.

FIG8(a) is a schematic diagram of an original picture provided by an embodiment of the present invention, and FIG8(b) is a schematic diagram of template matching of the original picture;

Step S230: construct a contact network bird nest detection model based on a cascade neural network, and train the contact network bird nest detection model. The contact network bird nest detection model includes a first-level YOLO detector and a second-level YOLO detector. The second-level YOLO detector is trained using the template library, and the first-level YOLO detector is trained using the domain of interest image dataset.

The first-level YOLO detector uses 3900 images after template matching in the template library. The label file of each image is obtained during the template matching process and trained using the yolov3-spp neural network.

The image of the second-level YOLO detector is the interest domain obtained by the reverse reasoning algorithm (which contains the bird's nest). The position of the bird's nest is the relative position relative to the interest domain. The label.txt can be obtained in the reverse reasoning algorithm process. The yolov3-spp neural network is used for training.

Step S240: input the image to be detected into the trained first-level YOLO detector, the first-level YOLO detector outputs the domain of interest image, input the domain of interest image into the trained second-level YOLO detector, the second-level YOLO detector outputs the bird nest detection result of the image to be detected.

The size of the bird's nest in the contact network is small and lacks significant shape and texture features. It is difficult to obtain ideal results by using existing artificially designed features to classify stripe images. In this regard, deep learning provides a feasible solution. As a common target detection network, the YOLO neural network has powerful detection performance. The present invention uses a detection network with a two-level prediction network cascade to detect the bird's nest in the contact network, and uses the YOLOv3-SPP network structure as a hierarchical detector.

The YOLO neural network takes the entire image as input and outputs the predicted bounding box and the category it belongs to.

The algorithm starts by dividing an image into S×S grids, as shown in Figure 9. The grid where the center of the labeled object is located is responsible for predicting the corresponding labeled object. Each grid needs to predict B bounding boxes, and each bounding box needs to regress its own position (x, y, w, h) and confidence.

YOLOv3 changes the size of the propagation tensor in the network by changing the step size of the convolution kernel. YOLOv3 improves the prediction speed of the model by calculating the bounding box in advance. YOLOv3 determines the position of the bounding box by predicting the relative offset between the center point of the bounding box and the upper left corner of the corresponding grid. And normalize t _x and _ty so that the bounding box prediction value is between 0 and 1, which can ensure that the center point of the bounding box is definitely in the divided grid.

b _x =σ(t _x )+c _x

_by =σ( _ty )+ _cy

_tx , _ty , _tw , and _th are the predicted outputs of the model. _cx and _cy represent the coordinates of the grid, _pw and _ph represent the size of the bounding box before prediction, and _bx , by, _{bw ,} and _bh are the center coordinates and size of the predicted bounding box.

The confidence is whether the grid correctly predicts the object to be detected and the deviation between the bounding box and the actual position of the object. The confidence can be expressed as follows:

Where IOU is the intersection-over-union ratio of the bounding box and the object annotation box, and its calculation method is as follows:

Where (tx, ty, tw, th) represents the truth position attributes (x, y, w, h) of the annotation box.

(px, py, pw, ph) represents the bounding box pred position attributes (x, y, w, h).

Pr(Object) represents the probability that the object to be predicted exists in the grid. During the training process, if no object to be predicted falls into the grid, then Pr(Object) is 0, otherwise it is 1.

During the training process, each grid corresponds to a pixel matrix, and each pixel matrix is used as the input of the neural network. For each grid, the network outputs (x, y, w, h, conf, c ₁ ... c _n ) for each bounding box, where (x, y, w, h) gives the position of the bounding box, conf is the confidence of the bounding box, and c ₁ ... c _n represents the class probability of the object.

YOLOv3 uses multi-scale features to detect targets. The feature map obtained after 32 times downsampling has a relatively large receptive field, which is suitable for detecting larger targets in the image; the feature map obtained after 16 times downsampling has a medium-scale receptive field, which is suitable for detecting medium-sized targets in the image; the feature map obtained after 8 times downsampling has a relatively small receptive field, which is suitable for detecting smaller targets in the image.

YOLOv3 uses the K-means clustering method to obtain the size of the prior frame, and sets 3 prior frames of different sizes for each downsampling scale, clustering a total of 9 prior frames of different sizes. When the resolution of the input image is 416*416, the distribution of the 9 prior frames on feature maps of different sizes is shown in Table 1.

Table 1 Feature maps and prior boxes

Directly using the YOLO network to detect bird nests on the contact network is not ideal, because the bird nests account for a very small proportion of the image, resulting in invalid calculations on a large number of grids and a waste of computing resources.

Moreover, the YoloV3 neural network uses the prior anchor boxes obtained by clustering on the dataset.

For small-scale objects, the 10U value of large-size anchor boxes and medium-size anchor boxes will not be very high, and the overall accuracy is not high.

FIG10 is a schematic diagram of a single-stage network prediction provided by an embodiment of the present invention. As shown in FIG10 , a grid in the upper left corner performs effective calculation.

The prediction expectation of a single network is:

remember: is the average IOU of the valid prediction box

In the formula The proportion of bird nest pictures.

In the overhead line environment, the environmental distribution of bird nests is continuous due to physical factors, that is, the location of bird nests in the overhead line is bounded. The domain of interest obtained above is the total possible area where bird nests may appear. It can be expressed as follows:

P(distribution) = P(the bird's nest exists in the interest domain of the image | the image contains the bird's nest) = 1

Based on the prior condition of bird nest distribution, a cascaded YOLO neural network can be used to first identify the region of interest, and then identify the location of the bird nest from the predicted sub-image of the region of interest given by the network.

(1) First level prediction:

Figure 11 is a schematic diagram of the first-level network prediction of a cascade network provided by an embodiment of the present invention. The segmentation effect is shown in Figure 11, the black box is the domain of interest given by template matching, and the black segmentation line is the grid divided by the yolo network. Only the four grids in the upper left corner are responsible for predicting the domain of interest and have confidence, and the confidence of other grids is 0.

The confidence and expectations for the first level of network detection are as follows:

Where Pr(Zone) is the probability that the current grid has the object to be tested (domain of interest). During the training process, if the grid to be tested contains the domain of interest, Pr(Zone) is 1, otherwise it is 0. is the intersection-over-union ratio of the grid predicted annotation box and the rectangular box where the interest area actually is. B is the number of annotation boxes predicted for each grid, S ² is the total number of grids divided into the image, The average IOU of all predicted boxes for all grids where the object is located. i(Zone) is the size of the region of interest, and I(image) is the size of the original image. E(Zone) is the sum of the total IOU given by the image, which reflects the overall degree of accuracy of the object prediction.

(2) Second level prediction:

Figure 12 is a schematic diagram of the second-level network prediction of a cascade network provided by an embodiment of the present invention. The second-level Yolo network takes the sub-image set of the domain of interest predicted by the first-level network as input, and performs YOLO cascade detection on the sub-image set, which is essentially a secondary division of the grid, and increases the IOU of the bounding box as much as possible during the training process to improve the accuracy of the training and perform as many effective calculations as possible.

The confidence of the second-level network is:

Where Pr(Birdnest) is the probability that the current grid has the object to be tested (bird nest). During the training process, if the grid to be tested contains a bird nest, Pr(Birdnest) is 1, otherwise it is 0. It is the intersection-over-union ratio of the grid predicted annotation box and the rectangular box where the bird’s nest actually is. P(distribution) is the probability that the bird’s nest in the image exists in the domain of interest. Obviously, all bird’s nests exist in the domain of interest, so this item is 1.

The prediction expectation of bird nests in the domain of interest is:

Where Pr _i (birdnest) is the probability that the i-th grid in the sub-image contains a bird nest in the sub-image of the interest domain. is the intersection-over-union ratio of the jth rectangular box predicted by the i-th grid of the sub-image and the bird's nest image, confidence(Zone) is the confidence of the interest region marked by a rectangular box predicted by the original image grid, A prediction box representing the divided grid of the original image can mark the degree of certainty of the bird's nest.

The expectation of the cascade prediction is:

In the formula The average intersection-over-union ratio of the predicted annotation box of the grid contained in the bird's nest in the sub-image of the interest domain and the rectangle where the bird's nest is located. (The predicted annotation box is given by the prior clustering algorithm, which is proportional to the size of the sub-image, so the sub-image prediction box is more suitable for the shape of the bird's nest than the prediction box size in the original large image, so its average IOU value is larger). The meanings of the other parameters are the same as above.

In the formula represents the average IOU value of the grid prediction anchor box, because the size of the anchor box is given by the dataset clustering prior, and it is positively correlated with the size of the input image. The closer the ratio of the object to be detected to the anchor box, the larger the average IOU value, so the given formula is and Both are greater than the original The cascade prediction accuracy is greater than the original prediction accuracy.

On the other hand, the average prediction accuracy of a single-stage neural network is positively correlated with the number of training sample data sets. Suppose the average prediction accuracy of the neural network when the training samples are labeled with Object in the Base picture and the number is n is F(Object, Base, n).

From the above discussion, the expected accuracy of the cascade is higher than that of the single-stage network:

F(birdnest, Zone, n)*F(Zone, image, n)>F(birdnest, image, n)

In the dataset of bird nests in the contact network, there are few sample datasets with bird nest annotations, and most images do not contain bird nests. However, the number of samples in the interest domain dataset is extremely large. The interest domain that is not related to the bird nest to be detected has information gain on the distribution of bird nests. The training gain of the interest domain can increase the accuracy of the overall detector. Let the number of dataset samples be M, and the number of samples containing bird nests be N.

Then the overall accuracy of the trainer is:

P＝F(birdnest，Zone，N)*F(Zone，image，M)＞F(birdnest，image，N)

The interest domain dataset is relatively easy to obtain, M>>N. When there are enough samples, the first-level detector can distinguish all interest domains of the image to be tested.

F(Zone, image, M)→1

After the extreme amplification of the first-stage detector, the accuracy of the detector reaches its extreme value:

P _max =F (birdnest, Zone, N)

The accuracy of the detector is determined by the accuracy of the second-stage detector. The first-stage detector plays the role of object magnification. The second-stage detector detects the bird's nest in the domain of interest. The average IoU value of the object is significantly greater than that of the single-stage detector, and the detection accuracy of the detector is significantly improved.

Embodiment 2

Experimental environment and data

The experimental environment and system configuration of the present invention are as follows:

(1) Hardware configuration: Core ^TM i9-10900K@3.70GHz+NVIDIA Geforce RTX 3090+64GB memory

(2) Operating system: Windows 10

(3) Deep learning framework: CUDA 11.0+Pytorch1.7.0

Reverse reasoning

The experimental data comes from the overhead line inspection video collected by the inspection vehicle on a heavy-load railway. The video frames were manually processed to obtain a picture set, and a total of 400 pictures containing bird nests were selected. Reverse reasoning was used to obtain 130 bird nest pictures in the domain of interest for the second-level deep learning detector training. After manual verification, 58 pictures of the domain of interest were selected as the template library to perform template matching on the 3,900 pictures without bird nests in the video set. The matched data set was manually verified and corrected for the first-level deep learning detector training. Other deep learning models were used for comparative training. 240 of the original 400 pictures were used as the deep learning training set, 80 as the validation set, and the remaining 80 pictures were subjected to data enhancement methods to obtain 126 pictures containing bird nests for testing. The organizational structure of the experimental data is shown in Table 1:

Table 1 Experimental data organization structure

The experimental data set contains 130 pictures of bird nests to be tested. The pictures are selected by the reverse reasoning algorithm. They are the interest area of the contact network where the bird nests are located. The contact network area structure between the pictures in the picture set is similar, which can effectively reflect the spatial characteristics of the bird nests. The 130 pictures obtained are used as the data set for the second-level detector training. However, some pictures have more environmental background factors and large environmental noise, which have strong interference on the template matching process and need to be manually removed. The pictures after manual removal constitute the template library for template matching.

Template Matching

The dataset contains 3900 images of overhead lines. Template matching is performed according to the template matching algorithm to mark the domain of interest. Due to the environmental error of the template library, some samples have annotation deviations and need to be manually checked. The detection frames with incorrect matching are removed, and the images with a small number of matches are supplemented with annotations.

Due to the limitations of template matching, it is very likely that template matching of unknown samples cannot accurately mark all interest areas. The template matching results after manual verification are used as the data set for the first-level detector training. The deep learning algorithm is used to learn the image features of the interest area to identify the interest area of unknown samples. It has strong generalization ability and can accurately identify all contact network areas along the railway where bird damage may exist.

Some mainstream deep learning model detection

The accuracy of the prediction is reflected by the ratio of the number of correctly predicted samples to the total number of predicted samples. The higher the accuracy, the more accurate the detection model. The detection rate refers to the ratio of the number of correctly predicted samples to the total number of true samples. The higher the detection rate, the more reliable the detection model.

In order to achieve real-time detection, while ensuring that the model has a relatively high detection accuracy, the detection speed of the model should be increased as much as possible. Generally, the number of images processed per second (frame rate per second FPS) is used to reflect the speed of detection. For a single network detector, its detection speed is the speed at which the detection network processes images; for a cascade network, the first-level detector and the second-level detector are calculated in series, and the calculation method of parallel detection of each domain of interest is adopted.

When the dataset is small, both two-stage algorithms such as the YOLO series and one-stage algorithms such as Faster R-CNN have limited recognition capabilities for small target objects. In addition, the shape features of contact wire bird nests are relatively simple and difficult to distinguish from environmental samples. They are very difficult to train and it is difficult to perform well in the test set.

When the training sample size is small, Faster R-CNN has difficulty learning the overall features of bird nest samples due to their small size. It has poor expression ability and is prone to missing labels, which affects the detection rate of the detector.

The YOLO series of networks can identify objects at multiple scales and have stronger detection capabilities than the Faster R-CNN network. However, the shape features of bird nests are relatively simple and can be easily confused with environmental samples, which can easily lead to mislabeling and affect the accuracy of the detector.

The results of the recognition detection models such as YOLOv3-spp, YOLOv4 and Faster R-CNN are as follows:

It can be seen from the table that whether it is a two-stage network such as the YOLO series or a one-stage network such as Faster R-CNN, when the bird nest data set is small, the volume is small, and the shape features are single, the network learning effect is poor, it is difficult to learn the overall characteristics of the bird nest, and it is impossible to accurately mark the bird nests on the contact network along the railway.

Cascade detection

(1) First-stage detector

The first-level detector uses Faster R-CNN, yolov3-spp and yolov4 detection models for training domain of interest recognition tasks.

Use different network structures to detect the domain of interest.

YOLOv3-SPP has 225 layers and 62.5 million parameters, while YOLOv4 has 327 layers and 64 million parameters. The backbone network structures of YOLOv3 and YOLOv4 are roughly the same, and YOLOv4 has made optimizations and improvements in the training process: YOLOv3's prediction box regression uses GIOU_Loss, which cannot distinguish the relative position relationship of objects, while YOLOv4 uses CIOU_Loss, which takes into account scale information such as the distance between the center points of the bounding box and the aspect ratio of the bounding box on the basis of GIOU_Loss, greatly improving the accuracy of the prediction box regression.

A schematic diagram of an IOU curve in a Yolov3-GIO training process provided by an embodiment of the present invention is shown in Figure 13, and a schematic diagram of an IOU curve in a Yolov4-CIOU training process provided by an embodiment of the present invention is shown in Figure 14. It can be seen from Figures 13 and 14 that when the training progress is the same, the CIOU of YOLOv4 is much larger than the GIOU of YOLOv3. It contains more of the center position and area size of the domain of interest, and can better learn the regional characteristics of the domain of interest.

Faster R-CNN uses two-stage detection compared to the YOLO series. It first generates candidate regions and generates more prediction boxes than the YOLO series network, and then classifies and regresses the candidate boxes. The one-stage algorithm of the YOLO series directly classifies and regresses the input image without generating candidate regions. Therefore, Faster R-CNN has a lower error rate and missed recognition rate, but the recognition speed of the one-stage algorithm is slower.

A total of 126 images of the railway overhead contact network test data set were tested, and the prediction results of different network models are as follows:

Comparison of the number of interest area detections in the first-level network model

(2) Second stage detector

The model of the second-level detector is trained on a total of 130 bird nest images in the template library.

The detection task of the second-level detector is to identify the bird's nest in the domain of interest. The pictures in the data set are enlarged by bilinear interpolation. Even a bird's nest with a small volume can occupy a large area in the picture, and the interference factor of environmental noise is effectively eliminated, and it has a strong anti-interference ability. Since the bird's nest is enlarged by bilinear interpolation with the original picture, its training IOU value is also improved. Compared with the same detection model, the IOU value is compared as follows under the same training progress: Figure 15 is a YOLOv3-SPP second-level detection and direct detection IOU comparison schematic diagram provided by an embodiment of the present invention, and Figure 16 is a YOLOv4 second-level detection and direct detection IOU comparison schematic diagram provided by an embodiment of the present invention. It can be seen from Figures 15 and 16 that the detection model under the same YOLO series has a higher IOU value for the image enlarged by bilinear interpolation of the bird's nest, and can more accurately locate the position of the bird's nest.

A total of 222 bird nests in 126 pictures of the test set were detected. Different first-level detectors were used to detect the domain of interest, and the detection performance of the second-level detector was tested. The calculation method of serial calculation of the first and second-level detectors and parallel detection of the second-level detector was adopted.

The test results are as follows:

Performance comparison of different object detection algorithms

As can be seen from the results in the table, the worst detection rate of the cascade results of the three detection models is 84.68%, which is still higher than the best result of direct detection, 77.48%. Cascade network recognition, through the first-level network to induce recognition of the prior interest domain, and the second level to identify the object to be tested, the bird's nest, in the interest domain. The distribution of the bird's nest in the interest domain is relatively simple, which can reduce the interference of other environmental samples, improve the distinction of the bird's nest, and reduce the difficulty of training. Therefore, the cascade network is less dependent on the size of the data set of the second-level object to be tested, and only a few pictures are needed to learn the distribution characteristics of the bird's nest in the interest domain.

Since the first-level detector uses Faster-RCNN to detect the largest number of interest domains, it can reach the extreme value condition of the cascade network, so the detector with the best detection rate is the Faster-RCNN cascade YOLOv3-SPP network, but the detection rate of Faster-RCNN is low. The detection rate of the overall detector is limited by the first-level detector and cannot meet the performance of real-time detection. The YOLOv4 network has stronger expression ability than YOLOv3. It performs well in interest domain recognition training with a large number of samples and easy-to-distinguish features of the objects to be tested, but the accuracy is not high in training with a small number of samples and unclear sample features. The cascaded YOLOv3-SPP network and the YOLOv4 cascaded YOLOv3-SPP network both have high detection rates and accuracy rates, and have high FPS. It can perform real-time detection of videos along the railway.

In summary, the method for automatic identification and rapid tracking of bird nests on high-speed railway contact networks proposed in the embodiments of the present invention can effectively solve the problem of accurate and rapid identification and tracking of bird nests on high-speed railway contact networks; can effectively solve the identification difficulties caused by the small amount of bird nest information in the contact network and the lack of significant shape or texture features; can narrow the search range when the target is much smaller than the scene range, thereby converting it to a large target tracking situation, which can significantly improve the accuracy rate. Thus, effective automatic identification and detection of bird nests on railway contact networks can be performed.

The present invention proposes a tracking method based on a cascaded neural network, in which the first-level neural network represents the interest domain where the bird's nest appears and then uses it for training the second-level neural network, thereby maintaining a high tracking speed while pursuing accuracy.

Existing deep learning models are very difficult to detect bird nests directly, and the effect is not good. Based on this, we have invented a detection method that can complete complex detection tasks with excellent detection capabilities by combining existing algorithms to process the data set and designing a cascaded detection network when there are few data sets and the detection targets are small. This article focuses on the advantages of the cascade detection structure compared to traditional network detection, and the principle of its excellent detection capabilities.

Those skilled in the art can understand that the accompanying drawings are only schematic diagrams of one embodiment, and the modules or processes in the accompanying drawings are not necessarily required to implement the present invention.

It can be known from the description of the above implementation methods that those skilled in the art can clearly understand that the present invention can be implemented by means of software plus a necessary general hardware platform. Based on such an understanding, the technical solution of the present invention can be essentially or partly contributed to the prior art in the form of a software product, which can be stored in a storage medium such as ROM/RAM, a magnetic disk, an optical disk, etc., and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in the various embodiments of the present invention or certain parts of the embodiments.

Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can refer to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device or system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can refer to the partial description of the method embodiment. The device and system embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

The above is only a preferred specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed by the present invention should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims (4)

1. The method for detecting the bird nest of the railway contact net is characterized by comprising the following steps of:

obtaining an interest domain picture containing a bird nest region through reverse reasoning according to a picture containing a bird nest in a railway contact net picture data set, taking the interest domain picture as a template picture, forming a template library according to all the template pictures, and training a second-level YOLO detector by using the template library;

sequentially matching the pictures which do not contain bird nest in the railway contact net picture data set with each template picture in the template library to obtain an interest domain picture data set, and training a first-stage YOLO detector by using the interest domain picture data set;

Inputting a picture to be detected into a trained first-stage YOLO detector, outputting a region-of-interest picture by the first-stage YOLO detector, inputting the region-of-interest picture into a trained second-stage YOLO detector, and outputting a bird nest detection result of the picture to be detected by the second-stage YOLO detector;

The method comprises the steps that a picture containing a bird nest in a railway contact net picture data set is subjected to reverse reasoning to obtain a region-of-interest picture containing a bird nest region, the region-of-interest picture is used as a template picture, and a template library is formed according to all the template pictures, and the method comprises the following steps:

Performing preliminary segmentation on a picture containing a bird nest in a railway contact net picture dataset to obtain a basic region with set similarity, performing preliminary merging on the basic region according to the difference between the regions to obtain a series of preliminary candidate regions, surrounding the preliminary candidate regions by rectangular frames, merging the preliminary candidate regions according to the similarity between the rectangular frames to obtain a final candidate region, manually marking the bird nest position in the final candidate region, using a rectangle to represent the marked bird nest region attribute, using the final candidate region containing the bird nest region as an interest region, using the interest region picture as a template picture, and forming a template library according to all the template pictures;

The method for primarily dividing the pictures containing the bird nest in the railway contact net picture data set to obtain basic areas with set similarity, primarily combining the basic areas according to the difference between the areas to obtain a series of primary candidate areas, and comprises the following steps:

A picture containing bird's nest is represented by an undirected graph G= < V, E >, wherein the vertex of the undirected graph represents a pixel point of the picture, the weight of the edge e= (V _i,v_j) represents the dissimilarity of the adjacent vertex pair i, j, and the color distance of the pixel is used Representing dissimilarity w (e) between two pixels, one basic region being a set of points with minimum dissimilarity;

the intra-class differences of the base region are defined as:

The inter-class difference between the two base areas C ₁、C₂ defines the minimum connecting edge of the two base areas:

If two base regions are not edge-connected, diff (C ₁,C₂) = infinity

When the condition Diff (C ₁,C₂)≤min(Int(C₁)+τ(C₁),Int(C₂)+τ(C₂) is satisfied), it is judged that the two basic areas C ₁、C₂ can be merged;

Where τ (C) is a threshold function that weights the region of isolated points:

τ(C)＝k/‖C‖

Preliminary merging is carried out on each basic region to obtain a series of preliminary candidate regions;

The preliminary candidate areas are surrounded by rectangular frames, the preliminary candidate areas are combined according to the similarity between the rectangular frames, and a final candidate area is obtained, and the method comprises the following steps:

Surrounding the preliminary candidate region by a rectangular frame, wherein the position of the rectangular frame C is represented by a quadruple of (x, y, w, h), wherein x, y represents the coordinate of the upper left corner of the rectangular frame, and w, h represents the width and the height of the rectangular frame;

Rectangular box of preliminary candidate region r _i And a rectangular box/>, of the preliminary candidate region r _j The color distance between the two is as follows:

In the middle of Representing the pixel proportion of the kth bin of the color histogram;

Rectangular box of preliminary candidate region r _i And a rectangular box/>, of the preliminary candidate region r _j Rectangular box betweenAndThe texture distance between the two is as follows:

In the middle of Representing the k-th dimension pixel point proportion of the texture histogram;

for the preliminary candidate regions r _i and r _j:

Wherein size (r _i) represents the size of a rectangular frame corresponding to the region r _i, size (r _j) represents the size of a rectangular frame corresponding to the region r _j, size (im) represents the size of an original picture to be segmented, and S _size(r_i,r_j) is the size similarity score of the rectangular regions r _i and r _j;

for the preliminary candidate regions r _i and r _j:

Wherein size (BB _ij) represents the size of the circumscribed rectangle of the regions r _i and r _j, S _fill(r_i,r_j) is the similarity score filled in the rectangular regions r _i and r _j, and the intersection degree of different regions is measured;

The total difference between the preliminary candidate regions r _i and r _j is:

S(r_i,r_j)＝a₁S_colour(r_i,r_j)+a₂S_texture(r_i,r_j)+a₃S_size(r_i,r_j)+a₄S_fill(r_i,r_j)

a ₁,a₂,a₃,a₄ is a corresponding weight value;

When the total difference S (r _i,r_j) between the preliminary candidate areas r _i and r _j is larger than a set merging threshold, merging the preliminary candidate areas r _i and r _j to obtain a final candidate area;

The step of sequentially matching the pictures which do not contain bird nest in the railway contact net picture data set with each template picture in the template library to obtain an interest domain picture data set comprises the following steps:

Sequentially matching a picture to be matched, which does not contain a bird nest, in a railway contact net picture data set with each template picture in a template library, wherein the template picture is set to be T, the picture to be matched, which does not contain a bird nest, is set to be I, the width of the template picture is set to be w, the height of the template picture is set to be h, and R represents a matching result, and the matching method is expressed by the following general expression:

Wherein:

The larger the R value is, the higher the similarity between a rectangular area with the size (w, h) at the (x, y) position of the picture to be matched and the template is, the maximum value of the similarity of the template is taken as a template matching result, and the template matching value is required to be higher than a threshold parameter;

Recording device

Each template picture corresponds to an optimal matching value R, the positions of rectangular matching frames corresponding to the R are (x, y, w and h), and a template primary matching result forms a result set S:

Wherein c is a matched threshold parameter;

the result set S is arranged in descending order of Rs value, and the intersecting condition of the two rectangular matching frames S and t rectangular is as follows:

max(x(s),x(t))≤min(x(s)+w(s),x(t)+w(t))

max(y(s),y(t))≤min(y(s)+h(s),y(t)+h(t))

And traversing the result set S in sequence, discarding the labeling if the current rectangular matching frame is intersected with the labeled rectangular matching frame, otherwise, labeling the current rectangular matching frame in the VOC format, and forming the interest domain data set by all the labeled rectangular matching frames.

2. The method according to claim 1, wherein the manually labeling the bird's nest position in the final candidate region, representing the labeled bird's nest region attribute with a rectangle, using the final candidate region including the bird's nest region as the interest region, using the interest region picture as the template picture, and constructing the template library according to all the template pictures, includes:

the final candidate region C is represented by a rectangle, and the position attribute of the rectangle is represented by a quadruple (x, y, w, h);

Labeling the nest position in the final candidate region, and representing the labeled nest region attribute by using a rectangle, wherein the rectangular position attribute is (bx, by, bw, bh), the interest region is a candidate region containing the nest region, and the position coordinates of the interest region satisfy the following conditions:

and satisfies a threshold condition:

And taking the interest domain picture as a template picture, and forming a template library according to all the template pictures.

3. The method according to any one of claims 1 to 2, wherein the first stage YOLO detector and the second stage YOLO detector comprise: YOLOv3-spp, YOLOv4 and Faster R-CNN.

4. The method of claim 3, wherein the confidence and expectations of the first-stage YOLO detector are as follows:

Wherein Pr (Zone) is the probability that the current grid has an object to be tested (interest domain), in the training process, the grid to be tested contains the interest domain, pr (Zone) is 1, otherwise is 0, For the intersection ratio of the marking frame of grid prediction and the rectangular frame where the interest domain is actually located, B is the marking frame number of each grid prediction, S ² is the total division grid number of the picture, and/(A)The method comprises the steps that an average value of all prediction frames IOU of prediction is made for all grids where objects are located, I (Zone) is the size of a region of interest, I (image) is the size of an original picture, and E (Zone) is the sum of total IOUs given by the picture;

The confidence level of the second-stage YOLO detector is:

where Pr (Birdnest) is the probability that the current grid has a bird nest, the grid to be measured contains a bird nest, pr (Birdnest) is 1, otherwise is 0, For the intersection ratio of the labeling frame of grid prediction and the rectangular frame where the bird nest is truly located, P (distribution) is the probability that the bird nest of the picture exists in the interest domain, and obviously all bird nests exist in the interest domain, and the probability is 1;

the bird nest predictions in the interest domain are expected to be:

Where Pr _i (birdnest) is the probability that the ith grid in the sub-image has a bird's nest, For the intersection ratio of the jth rectangular frame predicted by the ith grid of the sub-image and the nest picture, confidence (Zone) marks the confidence of the interest domain for one rectangular frame predicted by the original image grid, and the confidence of the interest domain is marked by the element of the matrixA prediction frame representing the prediction made by the divided grids of the original image can mark the grasping degree of the bird nest;

The expectations of cascading predictions are:

In the middle of Average cross-ratios of prediction annotation frames made for grids contained in bird's nest in interest domain sub-images and rectangles in which bird's nest is located, whereRepresenting an average IOU value of the grid prediction anchor frame;

the precision of the cascade prediction is:

P＝F(birdnest,Zone,N)*F(Zone,image,M)>F(biednest,inage,N)。