CN116453133B - Banner Text Detection Method and System Based on Bezier Curve and Key Points - Google Patents

Abstract

The invention discloses a banner text detection method and system based on Bezier curves and key points. First, an initial text box of a text area is generated according to an image label, and then a fixed threshold is used to simplify the long side coordinates of the initial text box. Based on the simplified The long side coordinate points of the Bezier curve are generated, and the two Bezier curves are connected end to end to form a new text box, and the label of the text box is changed from the text box boundary coordinate point to the key point coordinate and the width of the key point, and then Build and train the banner text detection network model, and finally use the trained banner text detection network model to detect the text in the banner image. The invention solves the problem that the prior art cannot accurately frame the banner text, can improve the detection speed and use less data while completing the text detection.

Description

Banner Text Detection Method and System Based on Bezier Curve and Key Points

technical field

The invention belongs to the technical field of natural scene text positioning, and in particular relates to a banner text detection method and system based on Bezier curves and key points.

Background technique

With the continuous development of computer vision, target detection and semantic segmentation are also being updated iteratively. Use technologies such as target detection and semantic segmentation to divide the text area and non-text area in the banner, realize the detection of the banner text, and then identify the content of the text area.

By studying the text area of the banner image, it is found that the banner text has problems such as large text aspect ratio and text distortion. Although object detection methods have made some progress in dealing with this type of text, there are still problems such as false detection, missed detection, and many non-text areas in the detected text. At the same time, there are still some problems in using semantic segmentation to detect this kind of text, such as complex post-processing, slow detection speed and high requirements for hardware equipment. Therefore, to solve these problems, it is urgent to propose a new text detection method.

Contents of the invention

Aiming at the problem that the existing technology cannot accurately frame the banner text, the present invention provides a banner text detection method and system based on Bezier curves and key points.

In order to achieve the above object, the present invention provides a kind of banner text detection method based on Bezier curve and key point, comprises the following steps:

Step 1, generate the initial text box of the text area according to the label of the public text data set, reduce the number of long side coordinates of the text box through a fixed threshold, generate a Bezier curve based on the simplified long side coordinate points, and combine the two Bezier curves Connect the end to end to form a new text box, and change the label of the text box from the coordinate point of the border of the text box to the coordinates of the key point and the width of the key point;

Step 1.1, select images with special text such as long text and distorted text in the public text image data set as the data set, and generate the initial text box of the text area according to the label of the public text data set;

Step 1.2, using a fixed threshold method to determine the degree of curvature of the long side of the text box;

Step 1.3, according to the degree of curvature of the two long sides of the text box, selectively simplify the coordinate points of the two long sides of the text box;

Step 1.4, use the simplified coordinate points on the two long sides as the control points of the Bezier curve, generate two corresponding Bezier curves, and connect the two Bezier curves end to end to get the real bounding box of the text ;

Step 1.5, transform the label of the public data set from the coordinate point of the border of the text box into the coordinate of the key point and the width of the key point;

Step 2, build a banner text detection network model;

Step 3, using the key point data set obtained in step 1 to train the banner text detection network model constructed in step 2;

Step 4, use the trained banner text detection network model to detect the text in the banner image.

Moreover, the label of the public text data set in the step 1.1 is a plurality of groups of coordinates arranged clockwise, each group of coordinates is the coordinates of the text frame boundary points of the framed text, and each group of coordinates is connected clockwise to form a closed polygon. The initial text box of the text is obtained, and the number of boundary points of the dataset image is , select the previous in order /> as the upper boundary point, then /> As the lower boundary point, the connection line of the upper boundary point and the connection line of the lower boundary point are used as the two long sides of the initial text box.

Moreover, in the step 1.2, the distance between the first and last coordinates of the long side of the text box in the data set is compared with the distance from other coordinate points on the long side to the line by a fixed threshold to determine the degree of curvature of the two long sides of the text box. Right now:

(1)

In the formula, Indicates the curvature of the long side of the text box, /> Indicates the ratio of the furthest distance from the coordinate point on the long side of the text box in the image data set to the line between the first and last coordinates of the long side and the distance between the first and last coordinates. When the ratio is greater than or equal to 0 and less than /> , it is judged that the long side is a straight line, when the ratio is greater than or equal to /> , and less than /> , it is judged that the long side part is bent, when the ratio is greater than or equal to /> , it is judged that the long side is completely bent, /> , /> is the set threshold.

And, set the coordinate point on the long side to the distance of the line connecting the first and last coordinate points in the step 1.3 as , the first and last coordinate points are respectively /> , /> , the specific simplification process is as follows: when it is judged that the long side is a straight line, only the first and last coordinate points of the long side are reserved; Set the threshold when the edge is fully curved /> It is 0.1 times the length of the line connecting the first and last coordinates, when /> greater than /> , keep the corresponding coordinate points and discard other coordinate points; set /> The largest coordinate point is /> , use /> divide the curve into /> , /> Two parts, repeat the above operation until the distance from the point without coordinates to the line is greater than /> until.

Moreover, in the step 1.4, the coordinate points on the long side after simplification are used as the control points of the Bezier curve, and the Bezier curve is represented by a parameter curve based on the Bernstein polynomial, and the specific definition is as shown in the following formula:

(2)

(3)

In the formula, Represents the coordinate set of points on the Bezier curve, /> Indicates the order of the Bezier curve, /> Indicates the first coordinates of control points, /> Indicates the first /> Bernstein polynomials for control points, /> represents the binomial coefficient, /> Indicates time, when corresponding to the coordinates of all points on the Bezier curve, because /> or 1, /> has a value of 0, so when /> , select the first coordinate point on the long side as the position coordinate of the Bezier curve at time 0, when /> , select the last coordinate point on the long side as the position coordinate of the Bezier curve at time 1.

Two Bezier curves are generated by formula (2), and the closed polygon formed by connecting the two Bezier curves end to end is used as the real text box of the text instance.

Moreover, in the step 1.5, the boundary points on the two long sides are converted into a set of key points to represent the text box, and before being converted into key points, an upwardly compatible method is adopted to ensure the boundary points of the upper and lower two long sides of the text box The numbers are the same, and the specific steps are as follows: when the upper and lower sides are straight and partially curved, extract the midpoint of the straight side as one of the boundary points, so that there are three boundary points on the upper and lower sides; when the upper and lower sides are straight and partially curved, respectively When fully bent, divide the straight line equally according to the number of coordinate points on the fully bent side, and extract the equally divided coordinate points so that the number of boundary points on the upper and lower sides is the same; when the upper and lower sides are partially bent and fully bent, follow the complete bending The number of edge coordinate points minus the number of partially curved edge coordinate points divides the two curves of the partially curved edge equally, and extracts the equally divided coordinate points so that the number of boundary points on the upper and lower sides is the same. After the above operations make the number of upper and lower boundary points uniform, then transform the boundary points, and correspond the coordinates of the upper and lower sides from the beginning to the end one by one, take the midpoint coordinates of the corresponding coordinate points as the key point coordinates, and the two corresponding distances One-half is used as the width of the key point, so far the label in the public image text dataset is transformed from the coordinate point of the bounding box into a set of key point coordinates and corresponding width.

Moreover, the banner text detection network model in step 2 includes a feature extraction module, a feature fusion module, a regression module and a text box generation module. The feature extraction module is used to extract feature information at different levels, and obtain feature images containing semantic information from low-level to high-level. The feature fusion module is used for merging feature images of different levels to obtain a fused feature image for subsequent detection of banner text. The regression module is used to regress the shape of the text instance, as well as the key point coordinates and the width of the key point of the text instance. The text box generation module is used to generate a banner image text box based on the key point coordinates and width information in the current image.

The backbone network of the feature extraction module adopts the ResNet-50 model. After inputting the image into the ResNet-50 model, four feature images are sequentially obtained through channel increase and downsampling. , /> , /> , /> , the number of channels of the four feature images of different scales obtained in the backbone network is uniformly processed to obtain /> , /> , /> , /> , then from the lowest scale feature map /> Start the upsampling process, and the feature map with the same scale as the input of the FPN structure /> Perform an addition operation to obtain the fused lower-scale feature image/> , right /> After upsampling with /> Add to get the fused low-scale feature image/> , same for /> After upsampling with /> Add to get the fused feature image /> , and finally the fused feature image /> , /> , /> , /> as the output of the FPN.

The feature fusion module is to merge the fusion feature images of different scales to obtain the merged fusion feature image , the specific calculation process is as follows:

(4)

In the formula, Indicates a channel connection, /> and /> 2x, 4x, and 8x upsampling respectively, /> , , /> , /> is the fused feature image.

will fuse feature images Perform upsampling so that /> Same size as original image.

The regression module includes two parts: shape regression and key point regression. The shape regression will fuse the feature map through the convolution layer of the activation function Converted to a text shape feature map, by setting the threshold as /> Binarize the feature map above the threshold /> The area is used as the text area, below the threshold /> The region of is the background region, and the text shape binary image separated from the background is obtained. Compare the shape of the text outline in the binary image with the shape of the text box generated by the image key point label, and compare the IOU to the shape of the text outline in the binary image and the shape of the text box generated by the image key point label to match. The input for key point regression is the fused feature map /> , the output is the key point coordinates and width, including two branches, one of which outputs is /> heat map of key points, /> For the number of key points with the most key points in the detected image text instance, select the one with the highest score in the key point heat map The highlighted coordinate points are the key point coordinates in this key point heat map, which are also the key point coordinates of each text instance in this image corresponding to this type of key point, /> is the number of text instances in the detected image, if the number of key points of the text instance is insufficient, the number of highlighted coordinates will be reduced accordingly, and the output of another branch detection Width information, the width information corresponds to the key points one by one, if the number of key points of the text instance is insufficient, the remaining width information is set to 0.

The text box generation module takes the key point coordinates and width information output by the regression module as text instance information, and uses this information to generate a text box. The key point width is the distance from the key point to the corresponding long-side coordinate point. The line connecting two adjacent key points is used as the normal line connecting the key point and the long-side coordinate point. The key point is perpendicular to the normal line and extends the key up and down. The point corresponds to the width distance, and the end point coordinates are the long side coordinate points. According to the above operation, each coordinate point is processed to obtain two groups of long-side coordinate points with the same number of key points, and the long-side coordinate points are used as the control points of the Bezier curve to generate two Bezier curves, and the two Bezier curves are connected end to end to obtain a completely closed curved box, which is the text box of the text instance. Finally, the image of the framed text is output to realize the text detection of the banner image.

And, in described step 3, the key point dataset obtained in step 1 is divided into a training set and a test set, the training set is input into the banner text detection network model for iterative training, and the parameters of the banner text detection network model are updated, so that Minimize the loss function, record the accuracy of the test model in the test set, and save the optimal model. The training process is divided into shape detection training and key point detection training, and the corresponding loss function It is calculated as follows:

(5)

In the formula, is the shape loss function, /> is the key point loss function, /> is the weight factor of the loss function.

shape loss function is calculated as follows:

(6)

In the formula, Indicates the intersection and union ratio of the regressed text outline shape and the text box generated by the key point label, /> and /> represent the center point coordinates of the text box generated by the regressed text outline shape and the key point label, and the center point coordinates of the regressed text outline shape are the key point coordinates of the clockwise median of the key points of the text outline shape, When the number of key points is an even number, select the center point coordinates of the line connecting the two most middle key points, and the center point coordinates of the text box generated by the key point label is the key point of the clockwise median of the key points of the generated text box Coordinates, when the number of key points is an even number, select the coordinates of the center point of the line connecting the two most middle key points, /> Indicates the Euclidean distance between two center points, /> Indicates the diagonal length of the minimum closure area that can contain both the regressed text outline shape and the text box generated by the key point label, /> As an adjustment factor to balance the weight between overlapping area and aspect ratio similarity, /> is a measure of aspect ratio similarity.

key point loss function Including key point coordinates and width, the specific calculation formula is as follows:

(7)

(8)

(9)

In the formula, is the key point coordinate loss function, /> is the key point width loss function, /> is the weight factor, /> is the number of text instances in the image, /> Indicates the number of channels of the regression keypoint heatmap, /> and /> Indicate the height and width of the regression key point heat map, respectively, /> is the key point in the key point heatmap of the regression module regression /> score, Indicates the coordinate point score of the real key point heat map calculated by the Gaussian function of the image with the key point label, and /> is a hyperparameter that controls the contribution of each keypoint, via /> To reduce the penalty for points around the key point coordinates, /> Indicates to return the absolute value of the number enclosed in parentheses.

In order to speed up the model convergence, when the key point coordinates are regressed, the non-text area coordinate points are not considered to reduce the number of negative samples. After using the training set data to train the banner text detection network model, put the test set into the model, compare the accuracy and detection speed of text detection, and extract the optimal detection model.

The present invention also provides a banner text detection system based on Bezier curves and key points, which is used to realize the above-mentioned banner text detection method based on Bezier curves and key points.

Moreover, it includes a processor and a memory, the memory is used to store program instructions, and the processor is used to call the stored instructions in the memory to execute the above-mentioned method for detecting banner text based on Bezier curves and key points.

Compared with prior art, the present invention has following advantage:

1) Use a set of key points instead of a rectangular frame for regression, and use the key points to generate a text box instead of a fixed-shape anchor box to represent the text instance, and replace the long side of the text box with a Bezier curve, using Bezier The shape variability of the curve adapts to different text shapes, avoiding the problem that the shape of the text instance cannot be accurately represented due to the fixed shape of the anchor box.

2) In order to reduce the calculation pressure of the text box coordinate regression, in the key point label production stage, the number of long side coordinate points is reduced in an adaptive way to reduce the calculation pressure of the text box coordinate regression; the text box is replaced by the key point coordinates and width Coordinates are regressed, which significantly reduces the computational cost of regressing labels, thereby improving detection speed and using less data while completing text detection.

Description of drawings

Fig. 1 is a flowchart of an embodiment of the present invention.

FIG. 2 is a structural diagram of a banner text detection network according to an embodiment of the present invention.

Detailed ways

The present invention provides a banner text detection method and system based on Bezier curves and key points. The technical solutions of the present invention will be further described below in conjunction with the accompanying drawings and embodiments.

Embodiment one

As shown in Figure 1, the present invention provides a kind of banner text detection method based on Bezier curve and key point, comprises the following steps:

Step 1, generate the initial text box of the text area according to the label of the public text data set, reduce the number of long side coordinates of the text box through a fixed threshold, generate a Bezier curve based on the simplified long side coordinate points, and combine the two Bezier curves Connect the end to end to form a new text box, and change the label of the text box from the coordinate point of the text box boundary to the coordinate of the key point and the width of the key point.

Step 1.1, select images with special text such as long text and distorted text in the public text image data set as the data set, and generate the initial text box of the text area according to the labels of the public text data set.

The labels of the public text data set are multiple sets of coordinates arranged clockwise. Each set of coordinates is the coordinates of the boundary points of the text box that frame the text. Connect each set of coordinates clockwise to form a closed polygon, and the text box of the text is obtained. , which is the initial text box. In this example, the number of boundary points in the ctw-1500 data set is 14, and the first 7 are selected as the upper boundary points in order, and the last 7 are used as the lower boundary points. The connecting line of points (8~14) is used as the long side of the initial text box.

In step 1.2, a fixed threshold method is used to determine the degree of curvature of the long side of the text box.

Compare the distance between the first and last coordinates of the long side of the text box in the data set and the distance from other coordinate points on the long side to the line through a fixed threshold to determine the curvature of the two long sides of the text box, namely:

(1)

In the formula, Indicates the curvature of the long side of the text box, /> Indicates the ratio of the furthest distance from the coordinate point on the long side of the text box in the image data set to the line between the first and last coordinates of the long side and the distance between the first and last coordinates. When the ratio is greater than or equal to 0 and less than 0.1, the long side is determined to be a straight line , when the ratio is greater than or equal to 0.1 and less than 0.7, it is determined that the long side is partially bent, and when the ratio is greater than or equal to 0.7, it is determined that the long side is completely bent.

In step 1.3, according to the degree of curvature of the two long sides of the text box, the coordinate points of the two long sides of the text box are selectively simplified.

Let the distance from the coordinate point on the long side to the line connecting the first and last coordinate points be , the first and last coordinate points are respectively /> , /> , the specific simplification process is as follows: when it is judged that the long side is a straight line, only the first and last coordinate points of the long side are reserved; When the edge is completely curved, inspired by the Douglas-Peucker algorithm, set the threshold /> It is 0.1 times the length of the line connecting the first and last coordinates, when /> greater than /> , keep the corresponding coordinate points and discard other coordinate points; set /> The largest coordinate point is /> , use /> divide the curve into /> , Two parts, repeat the above operation until the distance from the point without coordinates to the line is greater than /> until.

Through the above operations, the simplification of the coordinate points on the long side is realized, the number of labels is reduced, and the calculation amount of the regression module of the banner image text detection model is reduced, and the detection speed is improved.

Step 1.4, use the simplified coordinate points on the two long sides as the control points of the Bezier curve, generate two corresponding Bezier curves, and connect the two Bezier curves end to end to get the real bounding box of the text .

The coordinate points on the long side after simplification are used as the control points of the Bezier curve. The Bezier curve is represented by a parametric curve based on the Bernstein polynomial. The specific definition is shown in the following formula:

(2)

(3)

In the formula, Represents the coordinate set of points on the Bezier curve, /> Indicates the order of the Bezier curve, /> Indicates the first coordinates of control points, /> Indicates the first /> Bernstein polynomials for control points, /> represents the binomial coefficient, /> Indicates time, when corresponding to the coordinates of all points on the Bezier curve, because /> or 1, /> has a value of 0, so when /> , select the first coordinate point on the long side as the position coordinate of the Bezier curve at time 0, when /> , select the last coordinate point on the long side as the position coordinate of the Bezier curve at time 1.

Two Bezier curves are generated by formula (2), and the closed polygon formed by connecting the two Bezier curves end to end is used as the real text box of the text instance.

In step 1.5, the label of the public data set is changed from the coordinate point of the border of the text box to the coordinate of the key point and the width of the key point.

In order to further reduce the number of labels in the data set and improve the detection efficiency, the boundary points on the two long sides are converted into a set of key points to represent the text box. Before converting it into a key point, first ensure that the number of boundary points on the upper and lower long sides of the text box is the same. Since the two long sides have different degrees of curvature, an upwardly compatible method is adopted to ensure that the number of boundary points is consistent. The specific steps are as follows :

When the upper and lower sides are straight and partially curved, extract the midpoint of the straight side as one of the boundary points, so that there are three boundary points on the upper and lower sides; when the upper and lower sides are straight and completely curved, press the full bending The number of side coordinate points divides the straight line equally, and extracts the coordinate points of the equal division so that the number of boundary points on the upper and lower sides is the same; when the upper and lower sides are partially curved and completely curved, subtract the part according to the number of coordinate points on the completely curved side The number of coordinate points of the curved edge divides the two curves of the partially curved edge equally, and extracts the coordinate points of the equal division, so that the number of boundary points on the upper and lower sides is consistent.

After the above operations make the number of the upper and lower boundary points uniform, then transform the boundary points. Correspond the coordinates of the upper and lower sides from the beginning to the end, take the midpoint coordinates of the corresponding coordinate points as the key point coordinates, and take half of the distance from the corresponding coordinate points as the width of the key point. So far, the labels in the public image text dataset are represented by The coordinate points of the bounding box are converted into a set of key point coordinates and corresponding widths, which realizes labeling based on key points.

Step 2, build a banner text detection network model.

The banner text detection network model includes a feature extraction module, a feature fusion module, a regression module and a text box generation module. The feature extraction module is used to extract feature information at different levels, and obtain feature images containing semantic and semantic information from low-level to high-level. The feature fusion module is used to superimpose and merge feature images of different levels to obtain a fused feature image for subsequent detection of banner text. The regression module is used to regress the shape of the text instance, as well as the key point coordinates and the width of the key point of the text instance. The text box generation module is used to generate a banner image text box based on the key point coordinates and width information vector information in the current image.

The banner text detection network model first uses Resnet50 as the backbone network to extract four feature images of different scales, and uses FPN (Feature Pyramid Network) to sequentially merge the feature images of different scales to obtain four fusion feature images of different scales; The fused feature images are up-sampled by corresponding multiples to obtain four feature images of the same scale, and then the four images are superimposed to obtain the fused feature image, and then the fused feature image is up-sampled four times to obtain the same size as the original image The same fusion feature image; perform regression operation on the fusion feature image to obtain two parts of regression data, compare the shape of the regressed text outline with the shape of the real text box, judge the similarity between the two, and send the key point regression data into the text box The generation module uses the coordinates and width information of the key points to obtain the coordinates of two sets of long-side control points, converts the obtained control points into two Bezier curves, and connects the Bezier curves to obtain the final text box of the text instance.

The backbone network of the feature extraction module adopts the ResNet-50 model. After the image is input to the ResNet-50 model, the image is first down-sampled so that the length and width of the image are reduced to 1/4 of the original image, and the number of channels is increased from 3 to 64. Then, a 1×1 convolution kernel is used to increase the number of channels from 64 to 256 while the length and width of the image remain unchanged, and the first feature map is obtained. , and then the feature map is channel-increased and down-sampled, so that the feature map reduces the length and width of the image to 1/2 and the number of channels is doubled to obtain the second feature map /> , repeat this operation to get four feature images in sequence /> , /> , , /> . The ResNet-50 with the fully connected layer removed is combined with the FPN structure, and the four feature images of different scales obtained in the backbone network are used as the input of the FPN structure. Before the fusion of feature images of different scales, the number of channels of the feature images needs to be processed uniformly, so a 1×1 convolution kernel is added to the input of the FPN structure, so that the number of feature image channels is reduced to 256, and /> , /> , /> , /> . Feature map from the lowest scale /> At the beginning, the nearest neighbor interpolation method is used for double upsampling, and the feature map of the same scale as the input of the FPN structure /> Perform an addition operation to obtain the fused lower-scale feature image/> , and again using nearest neighbor interpolation for /> After double upsampling with /> Add to get the fused low-scale feature image/> , same for /> After double upsampling with /> Add to get the fused feature image /> , and finally the fused feature image /> , /> , /> , /> as the output of the FPN.

The feature fusion module is to merge the fusion feature images of different scales to obtain the merged fusion feature image , the specific calculation process is as follows:

(4)

In the formula, Indicates a channel connection, /> and /> 2x, 4x, and 8x upsampling respectively, /> , , /> , /> is the fused feature image.

Using a 3×3 convolutional layer (with BN layer and ReLU layer to speed up model convergence and reduce model parameters) will The number of channels is reduced to 256, followed by the feature image /> 4x upsampling such that /> Same size as original image.

The regression module includes two parts: shape regression and key point regression. The shape regression will fuse the feature map through the 3×3 convolutional layer of the Sigmoid activation function Converted to a text shape feature map, by setting the threshold /> The feature map is binarized at 0.5, the area above the threshold of 0.5 is used as the text area, and the area below the threshold of 0.5 is used as the background area, and the text shape binary image separated from the background is obtained. Compare the shape of the text outline in the binary image with the shape of the text box generated by the image key point label, and compare the IOU to the shape of the text outline in the binary image and the shape of the text box generated by the image key point label to match. Since an image has multiple texts, there may be cases where key points match other texts. The function of shape regression is to ensure that key points are within the shape of the corresponding text outline to avoid false detection. The input for key point regression is the fused feature map /> , will fuse feature maps /> Input to two different detection branches, one of which detects the coordinates of key points, and the detection branch outputs through a 3×3 convolutional layer and a 1×1 convolutional layer/> heat map of key points, /> For the number of key points with the most key points in the detected image text instance, select the one with the highest score in the key point heat map The highlighted coordinate points are the key point coordinates in this key point heat map, which are also the key point coordinates of each text instance in this image corresponding to this type of key point,/> is the number of text instances in the detected image, if the number of key points of the text instance is insufficient, the number of highlighted coordinates will be reduced accordingly. The other branch detects the key point width, which is output through a 3×3 convolutional layer and a 1×1 convolutional layer/> width information, /> is the number of text instances in the detected image, and the width information corresponds to the key points one by one. If the number of key points of the text instance is insufficient, the remaining width information is set to 0.

The text box generation module takes the key point coordinates and width information output by the regression module as text instance information, and uses this information to generate a text box. Specifically, the width of the key point is the distance from the key point to the corresponding long-side coordinate point, and the line connecting two adjacent key points is used as the normal line connecting the key point and the long-side coordinate point, and the key point is perpendicular to the normal line up and down Extend the key point to correspond to the width distance, and the coordinates of the end point are the coordinates of the long side. Each coordinate point is processed according to the above method, and two sets of long-side coordinate points with the same number as key points are obtained. Take the coordinate point of the long side as the control point of the Bezier curve, obtain two Bezier curves according to the formula (2), connect the two Bezier curves end to end, and obtain a completely closed curve frame, and use the curve frame as the A text box for a text instance. Finally, the image of the framed text is output to realize the text detection of the banner image.

Step 3, use the key point data set obtained in step 1 to train the banner text detection network model built in step 2.

Divide the key point data set obtained in step 1 into a training set and a test set, input the training set into the banner text detection network model for iterative training, update the parameters of the banner text detection network model, minimize the loss function, and record the test Set the accuracy of the test model and save the optimal model. The training process is divided into shape detection training and key point detection training, and the corresponding loss function It is calculated as follows:

(5)

In the formula, is the shape loss function, /> is the key point loss function, /> is the weight factor of the loss function, this embodiment sets /> .

Considering that the shape of the banner text is arbitrary and there is a problem of large aspect ratio, the CIOU loss function is used to define , the specific formula is as follows:

(6)

In the formula, Indicates the intersection and union ratio of the regressed text outline shape and the text box generated by the key point label, /> and /> represent the center point coordinates of the text box generated by the regressed text outline shape and the key point label, and the center point coordinates of the regressed text outline shape are the key point coordinates of the clockwise median of the key points of the text outline shape, When the number of key points is an even number, select the center point coordinates of the line connecting the two most middle key points, and the center point coordinates of the text box generated by the key point label is the key point of the clockwise median of the key points of the generated text box Coordinates, when the number of key points is an even number, select the coordinates of the center point of the line connecting the two most middle key points, /> Indicates the Euclidean distance between two center points, /> Indicates the diagonal length of the minimum closure area that can contain both the regressed text outline shape and the text box generated by the key point label, /> As an adjustment factor to balance the weight between overlapping area and aspect ratio similarity, /> is a measure of aspect ratio similarity.

The key point contains two parts: the key point coordinate and the width, so the key point loss function Calculated as follows:

(7)

In the formula, is the key point coordinate loss function, /> is the key point width loss function, /> is a weighting factor, which is set to 0.2 in this embodiment.

Considering that in the process of training, the number of negative samples of key point coordinates is much larger than the number of positive samples, in order to solve the problem of imbalance between positive and negative samples, a variant of the focal loss function is used as ,Right now:

(8)

In the formula, is the key point in the key point heatmap of the regression module regression /> score, /> Indicates the coordinate point score of the real key point heat map calculated by the Gaussian function of the image with the key point label, /> is the number of text instances in the image, /> Indicates the number of channels of the regression keypoint heatmap, /> and /> Indicate the height and width of the regression key point heat map, respectively, /> and /> is a hyperparameter that controls the contribution of each key point, this embodiment sets /> , /> ,pass to reduce the penalty for points around the keypoint coordinates.

Since the width generated by each key point is random, the L ₁ loss function is used as :

(9)

In the formula, is the number of text instances in the image, /> Indicates to return the absolute value of the number in brackets, /> Indicates the coordinate point score of the real key point heat map calculated by the Gaussian function of the image with the key point label, is the key point in the key point heatmap of the regression module regression /> score.

In order to speed up the model convergence, when the key point coordinates are regressed, the non-text area coordinate points are not considered to reduce the number of negative samples.

After using the training set data to train the banner text detection network model, put the test set into the model, compare the accuracy and detection speed of text detection, and extract the optimal detection model.

Step 4, use the trained banner text detection network model to detect the text in the banner image.

Input an image with banner text into the banner text detection network model trained in step 3, after feature extraction, feature fusion, regression and text box generation, a banner text image with text box is obtained. The specific process includes: input a banner text image into the banner text detection network model, obtain four feature images of different scales through the feature extraction network of resnet50+FPN, and then perform up-sampling on the four images with different multiples, so that the image The scales are exactly the same, and then perform feature fusion on the four feature images to obtain a fused fusion feature image, quadruple upsample the fusion feature image to make it the same size as the original image, and then perform activation mapping on the fusion feature image, Get the key point heat map, get the key point coordinates and width through the key point heat map, and finally calculate two sets of long-side coordinate points according to the key point coordinates and width information, and use the long-side coordinate points as the control of the Bezier curve Point to generate two Bezier curves, connect the two Bezier curves end to end to get a closed curve frame as a text box, and output a banner text image with text box annotations.

Embodiment two

Based on the same inventive concept, the present invention also provides a banner text detection system based on Bezier curves and key points, including a processor and a memory, the memory is used to store program instructions, and the processor is used to call the program instructions in the memory for execution as described above A Banner Text Detection Method Based on Bezier Curves and Keypoints.

During specific implementation, the method proposed by the technical solution of the present invention can be implemented by those skilled in the art using computer software technology to realize the automatic operation process. The system device for realizing the method is, for example, a computer-readable storage medium that stores the corresponding computer program of the technical solution of the present invention and includes a computer that runs the corresponding computer program. The computer equipment of the program should also be within the protection scope of the present invention.

The specific embodiments described herein are merely illustrative of the spirit of the invention. Those skilled in the technical field to which the present invention belongs may make various modifications or supplements or replace them in similar ways to the described specific implementation cases, but they will not deviate from the spirit of the present invention or go beyond what is described in the appended claims. defined range.

Claims (10)

1. a banner text detection method based on Bezier curve and key points, is characterized in that, comprises the following steps: Step 1, generate the initial text box of the text area according to the label of the public text data set, reduce the number of long side coordinates of the text box through a fixed threshold, generate a Bezier curve based on the simplified long side coordinate points, and combine the two Bezier curves Connect the end to end to form a new text box, and change the label of the text box from the coordinate point of the border of the text box to the coordinates of the key point and the width of the key point; Step 1.1, select the image with long text and distorted text in the public text image data set as the data set, and generate the initial text box of the text area according to the label of the public text data set; Step 1.2, using a fixed threshold method to determine the degree of curvature of the long side of the text box; Step 1.3, according to the degree of curvature of the two long sides of the text box, selectively simplify the coordinate points of the two long sides of the text box; Step 1.4, use the simplified coordinate points on the two long sides as the control points of the Bezier curve, generate two corresponding Bezier curves, and connect the two Bezier curves end to end to get the real bounding box of the text ; Step 1.5, transform the label of the public data set from the coordinate point of the border of the text box into the coordinate of the key point and the width of the key point; Step 2, build a banner text detection network model; The banner text detection network model includes a feature extraction module, a feature fusion module, a regression module, and a text box generation module; the feature extraction module is used to extract feature information at different levels of the input image, and obtain feature images containing semantic information from low-level to high-level; The feature fusion module is used to merge feature images of different levels to obtain a fused feature image for subsequent detection of banner text; the regression module includes shape regression and key point regression, and shape regression obtains text and background based on the fusion feature map The separated text shape binary image, the key point regression is based on the fusion feature map to obtain the key point coordinates and width; the text box generation module is used to generate the banner image text box based on the key point coordinates and width information in the current image output by the regression module ; Step 3, using the key point data set in the image and label obtained in step 1 to train the banner text detection network model constructed in step 2; Divide the key point data set in the image and label obtained in step 1 into a training set and a test set, input the training set into the banner text detection network model for iterative training, and update the parameters of the banner text detection network model to minimize the loss function , record the accuracy of the test model in the test set, and save the optimal model; the training process is divided into shape detection training and key point detection training, and the corresponding loss function It is calculated as follows: (5) In the formula, is the shape loss function, /> is the key point loss function, /> is the weight factor of the loss function; shape loss function is calculated as follows: (6) In the formula, Indicates the intersection and union ratio of the regressed text outline shape and the text box generated by the key point label, /> and /> represent the center point coordinates of the text box generated by the regressed text outline shape and the key point label, and the center point coordinates of the regressed text outline shape are the key point coordinates of the clockwise median of the key points of the text outline shape, When the number of key points is an even number, select the center point coordinates of the line connecting the two most middle key points, and the center point coordinates of the text box generated by the key point label is the key point of the clockwise median of the key points of the generated text box Coordinates, when the number of key points is an even number, select the coordinates of the center point of the line connecting the two most middle key points, /> Indicates the Euclidean distance between two center points, /> Indicates the diagonal length of the minimum closure area that can contain both the regressed text outline shape and the text box generated by the key point label, /> As an adjustment factor to balance the weight between overlapping area and aspect ratio similarity, /> is a measure of aspect ratio similarity; Step 4, use the trained banner text detection network model to detect the text in the banner image. 2. a kind of banner text detection method based on Bezier curve and key point as claimed in claim 1, is characterized in that: the label of public text data set is a plurality of groups of coordinates arranged clockwise in step 1.1, each group The coordinates are the coordinates of the boundary points of the text box that frame the text, and each group of coordinates is connected clockwise to form a closed polygon to obtain the initial text box of the text, and the number of boundary points of the dataset image is set to , select the previous in order /> as the upper boundary point, then /> As the lower boundary point, the connection line of the upper boundary point and the connection line of the lower boundary point are used as the two long sides of the initial text box. 3. a kind of banner text detection method based on Bezier curve and key point as claimed in claim 1, it is characterized in that: in step 1.2, by fixed threshold value, the first and last coordinate line distance and length of text box long side in data set Compare the distance from other coordinate points on the side to the line to judge the curvature of the two long sides of the text box, namely: (1) In the formula, Indicates the curvature of the long side of the text box, /> Indicates the ratio of the farthest distance from the coordinate point on the long side of the text box to the line between the first and last coordinates of the long side and the distance between the first and last coordinates in the image dataset. When the ratio is greater than or equal to 0 and less than , it is judged that the long side is a straight line, when the ratio is greater than or equal to /> , and less than /> , it is judged that the long side part is bent, when the ratio is greater than or equal to /> , it is judged that the long side is completely bent, /> , /> is the set threshold. 4. a kind of banner text detection method based on Bezier curve and key point as claimed in claim 3, is characterized in that: set the coordinate point on the long side to the distance of the first and last coordinate point connecting line in step 1.3 as , the first and last coordinate points are respectively /> , /> , the specific simplification process is as follows: when it is judged that the long side is a straight line, only the first and last coordinate points of the long side are reserved; Set the threshold when the edge is fully curved /> It is 0.1 times the length of the line connecting the first and last coordinates, when /> greater than /> , keep the corresponding coordinate points and discard other coordinate points; set /> The largest coordinate point is /> , use /> divide the curve into /> , /> Two parts, repeat the above operation until the distance from the point without coordinates to the line is greater than /> until. 5. a kind of banner text detection method based on Bezier curve and key point as claimed in claim 1, is characterized in that: in step 1.4, the coordinate point on the long side after streamlining is used as the control point of Bezier curve, The Bezier curve is represented by a parametric curve based on the Bernstein polynomial, and the specific definition is shown in the following formula: (2) (3) In the formula, Represents the coordinate set of points on the Bezier curve, /> Indicates the order of the Bezier curve, /> Indicates the first /> coordinates of control points, /> Indicates the first /> Bernstein polynomials for control points, /> represents the binomial coefficient, /> Indicates time, when corresponding to the coordinates of all points on the Bezier curve, because /> or 1, /> The value of is 0, so when , select the first coordinate point on the long side as the position coordinate of the Bezier curve at time 0, when /> , select the last coordinate point on the long side as the position coordinate of the Bezier curve at time 1; Two Bezier curves are generated by formula (2), and the closed polygon formed by connecting the two Bezier curves end to end is used as the real text box of the text instance. 6. a kind of banner text detection method based on Bezier curve and key point as claimed in claim 3, is characterized in that: in step 1.5, the boundary point on the two long sides is converted into a group of key points to represent text box, before converting it into a key point, adopt an upwardly compatible method to ensure that the number of boundary points on the upper and lower two long sides of the text box is the same, the specific steps are as follows: when the upper and lower sides are straight and partially curved, extract the middle point as one of the boundary points, so that there are three boundary points on the upper and lower sides; when the upper and lower sides are straight and completely curved, the straight line is equally divided according to the number of coordinate points on the completely curved side, and the equally divided coordinate points are extracted. Make the number of boundary points on the upper and lower sides the same; when the upper and lower sides are partially curved and fully curved, divide the two curves of the partially curved side equally according to the number of coordinate points on the fully curved side minus the number of coordinate points on the partially curved side, and extract, etc. The coordinate points of the points are divided so that the number of boundary points on the upper and lower sides is the same; after the above operations make the number of the upper and lower boundary points uniform, then the boundary points are transformed, and the coordinates of the upper and lower sides are corresponding from the beginning to the end one by one, and the middle of the corresponding coordinate points is taken. The point coordinates are used as the key point coordinates, and half of the distance between the corresponding coordinate points is used as the width of the key point. So far, the labels in the public image text dataset have been transformed from the coordinate points of the bounding box into a set of key point coordinates and corresponding widths. 7. a kind of banner text detection method based on Bezier curve and key point as claimed in claim 1, it is characterized in that: in step 2, feature extraction module backbone network adopts ResNet-50 model, image is input to ResNet-50 After the model, four feature images are sequentially obtained through channel increase and downsampling processing , /> , /> , /> , the number of channels of the four feature images of different scales obtained in the backbone network is uniformly processed to obtain /> , /> , /> , /> , then from the lowest scale feature map /> Start the upsampling process, and feature maps with the same scale as the input of the FPN structure /> Perform an addition operation to obtain the fused lower-scale feature image/> , right /> After upsampling with /> Add to get the fused low-scale feature image/> , same for /> After upsampling with /> Add to get the fused feature image /> , and finally the fused feature image /> , /> , /> , /> as the output of the FPN; The feature fusion module is to merge the fusion feature images of different scales to obtain the merged fusion feature image , the specific calculation process is as follows: (4) In the formula, Indicates a channel connection, /> and /> 2x, 4x, and 8x upsampling respectively, /> , , /> , /> is the fused feature image; will fuse feature images Perform upsampling so that /> Same size as original image. 8. A kind of banner text detection method based on Bezier curve and key point as claimed in claim 7, it is characterized in that: in step 2, regression module comprises shape regression and key point regression two parts, and shape regression passes activation function The convolutional layer will fuse the feature maps Converted to a text shape feature map, by setting the threshold as /> Binarize the feature map above the threshold /> The area is used as the text area, below the threshold /> The area is the background area, and the text shape binary image separated from the text and the background is obtained; the text outline shape in the binary image is compared with the text box shape generated by the image key point label, and the IOU is compared by comparing the two. The shape of the text outline in the binary image is matched with the shape of the text box generated by the key point label of the image; the input of the key point regression is the fusion feature map /> , the output is the key point coordinates and width, including two branches, one of which outputs is /> heat map of key points, /> For the number of key points with the most key points in the detected image text instance, select the one with the highest score in the key point heat map The highlighted coordinate points are the key point coordinates in this key point heat map, which are also the key point coordinates of each text instance in this image corresponding to this type of key point, /> is the number of text instances in the detected image, if the number of key points of the text instance is insufficient, the number of highlighted coordinates will be reduced accordingly, and the output of another branch detection The width information corresponds to the key points one by one. If the number of key points in the text instance is insufficient, the remaining width information is 0; the text box generation module takes the key point coordinates and width information output by the regression module as the text instance information, and uses this information To generate a text box; the width of the key point is the distance from the key point to the corresponding long-side coordinate point, and the line connecting two adjacent key points is used as the normal line between the key point and the long-side coordinate point, and the key point is perpendicular to the normal Extend the key point up and down corresponding to the width distance, and the coordinates of the end point are the long-side coordinate points; process each coordinate point according to the above operation, and obtain two sets of long-side coordinate points with the same number as the key points, and use the long-side coordinate points as the shell The control points of the Zier curve are generated to obtain two Bezier curves, and the two Bezier curves are connected end to end to obtain a completely closed curve frame, which is the text box of the text instance; finally, the framed text image Output, implementing text detection for banner images. 9. a kind of banner text detection method based on Bezier curve and key point as claimed in claim 1, is characterized in that: key point loss function in step 3 Including key point coordinates and width, the specific calculation formula is as follows: (7) (8) (9) In the formula, is the key point coordinate loss function, /> is the key point width loss function, /> is the weight factor, is the number of text instances in the image, /> Indicates the number of channels of the regression keypoint heatmap, /> and /> Indicate the height and width of the regression key point heat map, respectively, /> is the key point in the key point heatmap of the regression module regression /> score, /> Indicates the coordinate point score of the real key point heat map calculated by the Gaussian function of the image with the key point label, /> and /> is a hyperparameter that controls the contribution of each keypoint, via /> To reduce the penalty for points around the key point coordinates, /> Indicates to return the absolute value of the number in brackets; In the regression of key point coordinates, non-text area coordinate points are not considered to reduce the number of negative samples; after using the training set data to train the banner text detection network model, put the test set into the model, and compare the accuracy of text detection and Detection speed, extract the optimal detection model. 10. A banner text detection system based on Bezier curves and key points, characterized in that it includes a processor and a memory, the memory is used to store program instructions, and the processor is used to call stored instructions in the memory to perform as claimed in claim 1- A banner text detection method based on Bezier curves and key points described in any one of 9.