CN110458158B - Text detection and identification method for assisting reading of blind people - Google Patents

Abstract

The invention discloses a text detection and recognition method for blind reading assistance. The method comprises the following steps: Step 1: scene detection, this step mainly detects whether the image captured by the camera is a scene where a finger is placed on the reading text; Step 2 : finger positioning, this step realizes the positioning of the fingertip, and uses the fingertip as the cursor for subsequent text detection; step 3: text extraction, this step mainly includes the extraction of text lines and the extraction of each word in the text line; step 4: Word tracking, this step mainly uses the template matching method to track the word frame of the correctly recognized word. The method of the invention has fast running speed and good effect, can not only accurately identify the word pointed by the user's fingertip, but also has low cost and high versatility, and can be widely used in smart products such as wearable blind auxiliary reading rings and the like .

Description

A Text Detection and Recognition Method for Blind Aided Reading

technical field

The invention belongs to the application field of computer vision, and particularly relates to a text detection and recognition method for blind reading aids.

Background technique

Today, there are 314 million people suffering from eye diseases worldwide. Of these, 269 million suffer from low vision and 45 million are blind. Currently, there are 8.77 million visually impaired people in China, accounting for about 19.5% of the total number of blind people in the world and about 0.7% of the total population in my country. According to the analysis of relevant authorities, the number of blind people in my country will exceed 75 million in six years. Therefore, how to help blind people overcome the difficulties of daily study and life, especially the most basic reading problems, has great research value and social significance, and also has broad application prospects.

Currently, there are a number of assisted reading products for the blind on the market, such as a Touch Reader worn on a finger. The built-in scanner of the product automatically scans and recognizes the passing text, and then converts the text into raised and recessed Braille through a dot matrix. Because the dot matrix is distributed on the inner layer of the finger cuff, the finger can sense its shape change, so that blind friends can recognize the braille. Similarly, the bottom of another touch-type blind reader can read ordinary text through the internal column array to output braille information, and then protruding columns appear on the top panel to form braille that can be recognized by touch. The EyeRing ring captures the text content in the book by the embedded miniature image scanning collector, and changes the blind spot combination in real time with the help of the "Braille dot display" set on the back of the collector near the finger, so as to allow blind friends to recognize the text through their fingers. But these products make it difficult for people who have not learned Braille to use these devices. In addition, other products like OrCam, a wearable device, consist of a small camera strapped to glasses and a processing system. The product can run computer vision algorithms, parse what you see, and then tell blind people and people with low vision what they see and information through bone conduction voice. However, the product is expensive, and the blind person may not be able to correctly align the scanning glasses with the reading material, so it is inconvenient to use. It can be seen that the above existing products either require users to learn Braille, or are expensive and inconvenient to use.

Regarding the patent for the blind reading method, Qiu Hong et al. (patent publication number CN108492682A) provided a blind reader, which sends the picture information obtained by the camera to the image recognition processor for recognition, and uses the recognition result as The level signal is fed back to the driving circuit to drive the Braille dot matrix component to output the corresponding Braille characters. A ring reader for the blind invented by Wang Lu (Patent Publication No. CN106601081A) can recognize the printed fonts on normal books and convert them into Braille through a camera arranged on the ring. But the main problem with these patented methods is that they still require users to be able to use Braille. In addition, Li Chongzhou et al. (Patent Publication No. CN103077625A) proposed an electronic reader for the blind and a reading method for the blind. The blind reading method first converts paper text into electronic picture format data by scanning or taking pictures, then recognizes it as an electronic text document through OCR recognition technology, and finally uses TTS speech synthesis technology to convert the electronic text document into voice data for playback. However, this patented method reads the text in the entire image aloud at one time, and fails to provide a convenient personalization function for users to read where they want to.

Under this background, it is particularly important to study a method with strong robustness, high accuracy, low cost, and automatic detection and recognition of the text pointed to by the fingers of blind or low-sighted users.

SUMMARY OF THE INVENTION

The technical problem to be solved by the present invention is to provide a text detection and recognition method for assisted reading for blind people, so that blind friends or people with low vision can also read ordinary books, and solve the problem of difficulty in reading for blind friends or people with low vision.

The technical scheme adopted in the present invention is as follows:

A text detection and recognition method for blind reading aids, comprising the following steps:

Step 1: For the image sequence captured by the camera, determine whether the scene in the current image is that the finger is placed on the reading text, if so, go to step 2, otherwise skip this frame of image, and use the next frame of image as the current image to perform the above judgment and processing;

Step 2: Position the user's fingertip in the current image;

Step 3: Determine the text line indicated by the user according to the position of the user's fingertip;

Step 4: Extract the words on the text line indicated by the user and convert it to speech output.

Further, in the step 1, the method for judging whether the scene in the current image is that the finger is placed on the reading text is as follows:

Step 11. Pre-shoot some typical images including the user's finger and the text area where it is located by the camera, and save it in the database;

Step 12, using the current image and multiple images taken in front of the image as sample images;

Step 13, normalize the RGB color space of the image in the database and the sample image respectively;

Step 14. For each sample, calculate the Euclidean distance between the normalized red channel image and the normalized red channel image of each image in the database, and use the minimum value in the result as the matching of the sample. Score; obtain the mean μ _Im and variance σ _Im of all sample matching scores, if μ _Im +σ _Im <Th, the scene in the current image is determined to be a finger on the reading text, where Th is the threshold, which is an empirical parameter.

Further, in the step 14, all images are reduced to the set size, and then the Euclidean distance is calculated.

Further, the step 2 includes the following steps:

Step 21. Use K-means to find the candidate area of the user's fingertip;

First, use a Gaussian filter to filter the current image;

Then, three two-dimensional matrices are generated according to the images of the three channels in the filtered image, and the element value in each two-dimensional matrix is the pixel value of the corresponding point on the image of the corresponding channel;

For each two-dimensional matrix, sum all its columns and take the average to get a row×1 column vector m _{c_ave} ; sum all its rows and take the average to get a 1×col row vector m _{r_ave} ;Therefore, convert the current image into three column vectors and three row vectors, where col represents the total number of columns of the two-dimensional matrix, and row represents the total number of rows of the two-dimensional matrix;

Each dimension of the column vector is regarded as a longitudinal data point, and the components of the same dimension of the three column vectors are regarded as the three features of the corresponding longitudinal data point to form the feature vector of the longitudinal data point, and the number of longitudinal data points is equal to the number of columns. The dimension of the vector, namely row; take each dimension of the row vector as a horizontal data point, and take the components of the same dimension of the three row vectors as the three features of the corresponding horizontal data point to form the feature vector of the horizontal data point, the horizontal The number of data points is equal to the dimension of the row vector, that is, col;

Secondly, K-means is used to cluster the longitudinal data points and the horizontal data points respectively, and the number of clusters is 2;

Again, the result of the clustering of longitudinal data points is expressed as a longitudinal label vector, which is a row×1 column vector, and the components of each dimension are expressed as the labels of the longitudinal data points of the corresponding dimension, and the value is 0 or 1 ; Express the result of the clustering of the horizontal data points as a horizontal label vector, which is a 1×col row vector, and the components of each dimension are expressed as the labels of the horizontal data points of the corresponding dimension, taking the value 0 or 1;

Perform mean filtering on the vertical label vector and horizontal label vector respectively, and then perform threshold processing. If the value of a dimension element is greater than or equal to the set threshold, set it to 1, otherwise set it to 0 to obtain the final vertical label vector and horizontal label vector;

Set the intersection point of the corresponding horizontal line in the current image with the demarcation point of element values 0 and 1 in the vertical label vector and the corresponding vertical line in the current image of the left demarcation point of element values 0 and 1 in the horizontal label vector as the upper left Vertex, delineate a rectangular area as a candidate area for the user's fingertip;

Step 22. Position the fingertip by calculating the curvature;

First, the canny operator is used to obtain the edge in the candidate area of the user's fingertip, and the edge is connected to obtain the contour; if multiple contours are obtained, only the contour containing the number of pixels not less than the set threshold is retained;

Then, smooth the remaining contours;

Finally, for the smoothed contour, the curvature of each pixel on the contour is calculated, and the point with zero curvature is the position of the user's fingertip.

Further, in the process of clustering vertical data points/horizontal data points using K-means, two groups of cluster centers are randomly initialized, two clusters are performed, and the compactness of the two clustering results is evaluated, and a good compactness is selected. The clustering result is used as the final clustering result.

Further, the step 3 includes the following steps:

Step 31, the extraction of the text area;

First, the current image is converted into a grayscale image;

Then, the grayscale image is binarized to obtain the foreground area and the background area of the image;

Then exclude the non-text areas in the background area as follows:

First extract all the connected regions in the background area to form a set _CR ; then each connected region in the set _CR is obtained its rotation rectangle, denoted as ζ(ο, θ, w, h), wherein o represents The center of the rotating rectangle, θ represents the angle deflected by the rotating rectangle, which is the counterclockwise rotation of the horizontal axis, and the angle between the first side of the rotating rectangle encountered, w and h respectively represent the two adjacent sides of the rotating rectangle; Then filter out the connected areas in which the area of the rotated rectangle and the deflected angle in the _CR do not meet the constraints. For the remaining connected areas, further filtering is performed based on the relationship between the text areas. The specific implementation includes the following steps:

3.1) Take the upper left vertex of the current image as the origin O, the length direction of the current image is the y-axis, the right direction is positive, the width direction of the current image is the x-axis, and the removal direction is positive; As a point of interest, each line passing through the point of interest in the current image is represented as follows:

xcosθ _SL +ysinθ _SL =ρ _SL ;

Among them, θ _SL is the angle between the straight line and the x-axis, ρ _SL is the distance from the origin O to the straight line, the value range of θ _SL is (-π/2, π/2), and the value range of ρ _SL is (-D , D), D is the diagonal length of the original image captured by the camera;

3.2) Subdivide the (ρ _SL , θ _SL ) parameter space into multiple accumulator units, and denote the value of the accumulator unit at the coordinates (ρ _k , θ _k ) as A(ρ _k , θ _k ); All accumulator cells are set to zero, and the distance d from each point of interest (x _i , y _i ) to the line xcosθ _k +ysinθ _k =ρ _k is calculated separately:

d=|x _i cosθ _k +y _i sinθ _k -ρ _k |;

The distances from all attention points to the straight line xcosθ _k +ysinθ _k =ρ _k are judged in turn. For each concern point corresponding to a distance smaller than the threshold, the value of A(ρ _k ,θ _k ) is increased by 1. After the judgment, the final result is obtained. The A(ρ _k , θ _k ) value of ; if the A(ρ _k , θ _k ) value is higher than the threshold value, the corresponding straight line is considered as a reference straight line, and the number of reference straight lines is N;

3.3) Find the text area by the method of unsupervised line clustering, the specific process is as follows:

3.31) Input the set of attention points, and initialize the reference straight line set _CL , which includes N reference straight lines, which are respectively the N reference straight lines obtained in step 3.2);

3.32) Calculate the distance from all points of interest in the point of interest set to each reference line in the set _CL ; for each point of interest, take the minimum distance value; filter out the points whose minimum distance value is less than the set threshold; Marking the category of attention points, and assigning the attention points whose minimum distance value corresponds to the same reference line to the same category;

3.33) Perform straight line fitting on the points of interest of the same category, and judge whether the difference between the slope and intercept of the newly fitted straight line and the reference straight line corresponding to the category is less than the set threshold, if it is less than the set _CL The reference line corresponding to this category remains unchanged, otherwise, update the reference line corresponding to this category in the set _CL to the newly fitted line; if all the reference lines in the set _CL remain unchanged in this step, Then output the set of attention points C _S with category marks, and the area contained in the rotation rectangle corresponding to these attention points is the extracted text area, otherwise return to step 3.31);

Step 32, determine the text line;

According to the fingertip position obtained in step 2, determine a rectangular area of interest, and the fingertip is located on the bottom edge of the area of interest;

To the text area that step 31 is extracted, seek the outermost contour of each character wherein; Select the bottom point of each contour as the reference point, and filter out the reference point located in the region of interest;

For the selected datum points, select three adjacent datum points each time, and perform straight line fitting to obtain multiple straight lines;

All the fitted straight lines are scored separately. The scoring formula is as follows:

Among them, d(i) _distance is the distance from the i-th benchmark point screened to the straight line, n is the total number of benchmark points screened out, and μ _score is the score;

Select the straight line with the lowest score as the judgment line; filter out the datum points whose distance to the judgment line is less than the set threshold, and then perform straight line fitting based on all the remaining datum points, and the fitted line is Line of text indicated for the user.

Further, the specific processing process of the step 4 is as follows:

Step 41, identify the first word;

For the text area extracted in step 31, extract the part located in the area of interest as the target text area; respectively obtain the minimum circumscribed rectangle of each character in the target text area as a letter box; The difference between the distance between the centers of two adjacent letter boxes and the distance between the centers of two adjacent letter boxes between words, cluster the letter boxes, and synthesize words; find the smallest circumscribed rectangle belonging to all letter boxes in a word as the word box ;

Perform angle compensation on the image according to the angle between the text line indicated by the user and the horizontal direction in the image, so that the text line is rotated to the horizontal direction;

Select the first word box along the text line, and then use OCR recognition technology to identify, the results returned by the recognition include: word, word confidence and word box; when the returned word confidence is greater than the threshold, the word is considered to be Correct recognition, voice output for correctly recognized words;

Step 42, adopt the template matching method to track the position of the word frame of the correctly identified word on the follow-up image frame, to determine the new word area on the follow-up image frame, and identify the new word in the new word area;

vi. Binarize each frame of images;

Initialize s=1, initialize the fingertip speed V _fingertip ; take the word frame corresponding to the jth word identified in the lth frame image and all the letter frames it contains as the word frame and letter frame that need to be tracked; take the mth frame The region of interest in the image is used as the search region; initialize m=l+1;

vii. For the word box/letter box to be tracked, use template matching to track its position in the search area;

If the current word frame to be tracked is successfully tracked, go to step iii; otherwise, determine whether there is a successfully tracked word frame before, if so, first determine the matching position on the m-th frame image with the matching position of the latest tracked successful word frame. A cutoff line, the cutoff line coincides with the right side of the matching position; then judge whether the width of the cutoff line from the left side of the image is less than the set threshold, if so, take the target text area to the right of the cutoff line as the new word area, and continue to identify new words in the new word area. Otherwise, end the new word recognition; if there is no word frame successfully tracked before, then take the mth frame image as the current image, and perform steps 1 to 41 on it to recognize the first word;

viii. Update fingertip speed V _fingertip :

Among them, V _word,j represents the horizontal distance difference between the position of the successfully tracked jth word box in the lth frame image and the mth frame image, V _letter,k represents the successfully tracked kth letter box in the lth frame image The horizontal distance difference between the frame image and the position in the mth frame image, N ₁ represents the number of successfully matched word boxes, and N ₂ represents the number of successfully matched letter boxes;

ix. let s=s+1;

For the word box corresponding to the sth word and each letter box it contains in the lth frame image, judge whether the word box and each letter box will move out of the mth frame image according to the fingertip speed. If the lth frame image If the abscissa of the upper left vertex of the upper word box/letter box minus V _fingertip ×(ml) is less than zero, it is determined that the word box/letter box will move out of the mth frame image;

The word box/letter box that is judged not to be moved out of the current frame image is used as the word box/letter box that needs to be tracked; and a rectangular area is defined as a new search area, the abscissa of the upper left corner of the rectangular area = the previous tracked area The abscissa of the upper left corner of the word box - fingertip speed - set the offset value, the length of the rectangular area = the length of the last word box to be tracked + the speed of the fingertip, the width of the rectangular area = the length of the last word box to be tracked width + set offset value;

Calculate the ratio of black pixels to white pixels in the new search area, if this ratio is less than the set threshold, discard the frame image, and let m=m+1, and repeat the above judgment and processing until the black pixels in the new search area If the ratio to the white pixel is not less than the set threshold, then enter step v; otherwise, directly enter step v;

x. Return to step ii.

Beneficial effects:

The invention discloses a text detection and recognition method for blind reading aids. The method includes the following steps: step 1: scene detection, which mainly detects whether the image captured by the camera is a scene where the finger is placed on the reading text; step 2: the finger Positioning, this step realizes the positioning of the fingertip, and uses the fingertip as the cursor for subsequent text detection; Step 3: text extraction, this step mainly includes the extraction of text lines and the extraction of each word in the text line; Step 4: Word tracking, this step mainly uses template matching method to track the word frame of the correctly recognized word. Using this method, combined with voice output, it can be applied to related auxiliary reading products for the blind, so that even if the user does not understand Braille, he can easily and quickly know the text content pointed to.

Compared with other reading aids for the blind, more commonly used is to convert the captured image text into Braille form that can be perceived by blind friends, which requires users to learn Braille. Relatively speaking, this solution adopts the method of directly outputting the word pointed by the user's finger by voice. Therefore, even if users do not understand Braille, they can easily and quickly enjoy reading anytime, anywhere. At the same time, the method of this scheme runs fast and has good effect. It can not only accurately identify the word pointed by the user's fingertip, but also has low cost and strong versatility, and can be widely used in wearable auxiliary reading rings for the blind, etc. Smart products.

Description of drawings

Fig. 1 is the overall implementation flow schematic diagram of the method of the present invention;

Fig. 2 is the scene detection process of the method of the present invention; wherein Fig. 2(a) is the finger database image established in advance, Fig. 2(b) is the normalization of the finger image, Fig. 2(c) is the image taken by the camera and the Schematic diagram of the matching process of finger database image template;

Fig. 3 is the fingertip detection process of the method of the present invention; wherein Fig. 3(a) is the original input picture of the camera, Fig. 3(b) is the RGB channel of the input picture, Fig. 3(c) is the column converted from the RGB channel vector, Fig. 3(d) is the possible area of the fingertip, Fig. 3(e) is the possible area of the enlarged fingertip, and Fig. 3(f) is the found fingertip position;

Fig. 4 is the schematic diagram of the rotation rectangle definition of the method of the present invention;

Fig. 5 is the text extraction process of the method of the present invention; wherein Fig. 5(a) is a grayscale image obtained by graying the color image of the fingertip position, and Fig. 5(b) is a comparison of the image shown in Fig.5(a). The binary image obtained after the grayscale image is binarized, Fig. 5(c) is the result of filtering Fig. 5(b) according to the area condition, Fig. 5(d) is the further analysis of Fig. 5(c) according to the angle condition ) filtered result;

Fig. 6 is the text region extraction process of the method of the present invention; Fig. 6(a) is the image obtained by angle filtering, Fig. 6(b) is the coordinate representation of all ζ(o), Fig. 6(c) is (ρ) _SL , θ _SL ) three-dimensional representation of the matrix, the z-axis is the value of the matrix, Figure 6(d) is the obtained initialization line, and Figure 6(e) is the text area obtained by line clustering;

Fig. 7 is the text line determination process of the method of the present invention; Fig. 7 (a) is the text area and fingertip position obtained through line clustering, Fig. 7 (b) is the schematic diagram of the key area determined by fingertip position; Fig. 7(c) is a schematic diagram of the extracted attention area, and the text line and abnormal straight line are marked on it.

Fig. 8 is the word recognition process of the method of the present invention; wherein Fig. 8 (a) is a schematic diagram of determining the word frame; Fig. 8 (b) is to determine the rotation angle according to the text line; Fig. 8 (c) is the image after the angle compensation is performed result;

Fig. 9 is a schematic diagram of word tracking in the method of the present invention; Fig. 9(a) is a schematic diagram of tracking reading; Fig. 9(b) is a schematic diagram of fuzzy block detection.

FIG. 10 is an actual effect diagram of the method of the present invention during operation.

Detailed ways

The present invention will be further described below in conjunction with the accompanying drawings:

This embodiment is aimed at the specific application of reading assistance for the blind, and the detection and recognition of the text in the captured image is carried out according to the following steps, and the overall implementation process is shown in FIG. 1 . As can be seen from the figure, the detailed implementation process mainly includes the following main steps:

Step 1: Determine whether the scene in the current image is a finger placed on the reading text through scene detection, if so, go to Step 2, otherwise do not go to the subsequent steps; the specific processing process of this step is as follows:

Step 11. Pre-shoot some typical images including the user's finger and the text area where it is located by the camera, and save it in the database, as shown in Figure 2(a);

Step 12, take the current image and the 19 images shot in front of the image (that is, the 20 images shot continuously by the camera recently) as samples;

Step 13, color normalization; as shown in Figure 2(b), normalize the RGB color space of the image in the database and the sample image respectively, in order to reduce the influence of light and shadow:

Among them, (r, g, b) represents the pixel value of a point in the original image, and (R, G, B) represents the pixel value of the point in the normalized image;

Since the finger color is skin color, extract the red channel image of each image after normalization processing, and perform subsequent image matching processing;

Step 14: Image matching; in order to reduce the matching time, first reduce the size of all images to a size of 50×50 pixels; for each sample, calculate the corresponding red channel image and the red channel image corresponding to each image in the database. The Euclidean distance of , the minimum value in the result is taken as the matching score of the sample; the mean μ _Im and the variance σ _Im of the matching scores of 20 samples are obtained, and in this embodiment, the two are calculated to be 35.08 and 10.01 respectively; if μ _Im +σ _Im <Th, it is determined that the scene in the current image is that the finger is placed on the reading text, and subsequent fingertip positioning operations can be performed, where Th is the threshold, which is set to 150 in this embodiment.

Step 2: Position the user's fingertip in the current image, as shown in Figure 2, as the cursor for subsequent text detection; the specific processing process is as follows:

Step 21. Use K-means to find the candidate area of the user's fingertip;

First, the current image (as shown in Figure 3(a)) is filtered with a Gaussian filter to reduce the interference of outliers and improve the accuracy of clustering. The size of the Gaussian kernel matrix H used in Gaussian filtering is (2k _G +1)×(2k _G +1). The calculation formula of the Gaussian kernel matrix H is:

In the above formula, H(i,j) is the element value of the ith row and the jth column of the Gaussian kernel matrix H, i,j=1,2,...,2k _G +1; σ _G is the width parameter of the Gaussian kernel function, The radial action range of the function is controlled, and is set to 3 in this embodiment; k _G is a parameter that controls the size of the Gaussian kernel, and is set to 15 in this embodiment;

Then, three two-dimensional matrices are generated according to the images of the three channels in the filtered image (RGB image), and the element value in each two-dimensional matrix is the pixel value of the corresponding point on the image of the corresponding channel;

Calculate each two-dimensional matrix according to the following formula to obtain a column vector and a row vector:

Among them, m _{c_ave} is the column vector obtained by summing all the columns of the two-dimensional matrix and then taking the average value; M(:,j) is the jth column of the two-dimensional matrix, and col represents the total number of columns of the two-dimensional matrix; m _{r_ave} is the row vector obtained by summing and averaging all rows of the two-dimensional matrix; M(i,:) is the i-th row of the two-dimensional matrix, and row represents the total number of rows of the two-dimensional matrix;

Thus, the current image is converted into three row×1 column vectors (as shown in Figure 3(b), (c)) and three 1×col row vectors; each dimension of the column vector is used as a longitudinal data point, the components of the same dimension of the three column vectors are used as the three features of the corresponding longitudinal data point to form the feature vector of the longitudinal data point, and the number of longitudinal data points is equal to the dimension of the column vector, namely row; Each dimension is used as a horizontal data point, and the components of the same dimension of the three row vectors are used as the three features of the corresponding horizontal data point to form the feature vector of the horizontal data point. The number of horizontal data points is equal to the dimension of the row vector, i.e. col;

Second, K-means is used to cluster the longitudinal and transverse data points separately. In this process, in order to prevent the local optimal solution from being obtained, the clustering process uses two random initial clustering centers to obtain two clustering results, evaluate the compactness of the two clustering results, and select the best compactness. The clustering result is used as the final clustering result.

The following formulas are used to evaluate the compactness of the two clustering results:

In the above formula, K represents the number of clusters, the value is 2, N represents the number of samples of a certain class, _xi represents the feature vector of the ith data point of the jth class, and _mj represents the cluster center of the jth class. The μ _score obtained by this formula is the sum of the squared distances from each point in the clustering result to its corresponding cluster center, which can reflect the compactness of the clustering result. The larger the μ _score , the worse the compactness; The clustering result with lower μ _score is selected as the final clustering result.

The result of the clustering of longitudinal data points is expressed as a longitudinal label vector, which is a row × 1 column vector, and the components of each dimension are expressed as the labels of the longitudinal data points of the corresponding dimension, which are 0 or 1; since In the clustering process, data points related to fingers are more likely to be classified into one category, that is, 0 and 1 in the clustering result are clustered together, and because fingers generally appear in the middle and lower parts of the picture, the elements of the label vector are generally valued. The upper part is basically all 0, and the lower part is basically all 1; in order to remove the labels of abnormal data points, one-dimensional mean filtering is used to filter the label vector, that is, the target data is given a template on the label vector, The size of the template is usually an odd number, and the template includes the adjacent data around it (for example, the size of the template is 5, the data included in the template are the two adjacent data on the right and the two adjacent data on the left of the target data, excluding the adjacent data. itself), and then replace the target data with the average value of all the data in the template; perform threshold processing on the filtered label vector, if the value of a dimension element is greater than or equal to the set threshold, set it to 1, otherwise set it to 1 Set 0 (the threshold is set according to experience, and is set to 0.4 in this embodiment) to obtain the final label vector. The horizontal line corresponding to the demarcation point of element values 0 and 1 in the label vector in the current image is the vertical cut-off position of the user's finger in the image;

The result of the horizontal data point clustering is represented as a horizontal label vector, which is a 1×col row vector, and the components of each dimension are represented as the labels of the horizontal data points of the corresponding dimension, which take the value of 0 or 1; since The finger generally appears in the middle of the image, so the value distribution of the elements of the label vector is 1 in the middle and 0 on both sides, so select the corresponding left dividing point of element values 0 and 1 in the label vector in the current image. The vertical line is the horizontal cut-off position of the user's finger in the image;

A large number of experimental results show that the intersection of the vertical cut-off position and the horizontal cut-off position is at the upper left of the finger; due to the mean filtering of the label vector, the left boundary point of 0 and 1 will be shifted to the 0 part, so that the intersection is at The upper left of the finger, so taking the intersection point as the upper left vertex, a large enough rectangular area can be delineated (the specific size of the rectangular area is related to the size of the input image. In this embodiment, the size of the input image is 480×640, and the length of the rectangular area is set is 320, and the width is 160) as the candidate area of the user's fingertip (ie, the area where the user's fingertip may exist), as shown in Figures 3(d) and (e).

Step 22. Position the fingertip by calculating the curvature;

First, the canny operator is used to obtain the edge in the candidate area of the user's fingertip, and the edges are connected to obtain the contour; if multiple contours are obtained, the set contour size threshold (experimental results show that the threshold is set to 100 is the best), Eliminate the contours with the number of pixels less than the set threshold to eliminate the interference of isolated points, and retain the contours with the number of pixels not less than the set threshold;

According to the parametric equation of the curve:

Γ(t)=(x(t),y(t));

Among them, t is the parameter, x(t) is the equation of the abscissa of the curve with respect to t, y(t) is the equation of the ordinate of the curve with respect to t, and Γ(t) is the equation of the curve with respect to t, then the curvature calculation formula of the curve can be known. for:

和

分别为x(t)和y(t)的一阶导数，

和

分别为为x(t)和y(t)的二阶导数。in,

and

are the first derivatives of x(t) and y(t), respectively,

and

are the second derivatives of x(t) and y(t), respectively.

In the present invention, since the contour is a collection of pixel points, in order to obtain the curvature of each pixel point in the contour, the calculation is performed as follows.

First, in order to reduce the influence of noise on the curvature measurement, the curve needs to be smoothed. Generate a one-dimensional Gaussian kernel according to the one-dimensional Gaussian function; define the size of the one-dimensional Gaussian kernel as M, where M is the smaller of the odd numbers closest to 10σ, and σ is the width parameter of the one-dimensional Gaussian function, which controls the diameter of the function. In the direction of the action range, the value in the experiment is 3, and M is 31. The specific operation is to convolve the coordinates of each pixel on the contour with a one-dimensional Gaussian kernel, which can be defined as follows:

Wherein L=(M-1)/2=15, X(n _point ), Y(n _point ) are the horizontal and vertical coordinates of the nth _point pixel on the contour (the coordinate system is established as: the origin O is the upper left in the image Vertex, the horizontal axis is the y-axis, the right direction is positive, the vertical axis is the x-axis, and the downward direction is positive), g(k, σ) is the k-th weight value of the one-dimensional Gaussian kernel, X(n, σ ) and Y(n,σ) are the coordinates of the nth pixel on the contour after smoothing, and the value of n _point is determined by the following formula:

where n _size is the number of pixels included in the contour, n,n _point =1,2,...,n _size . According to the properties of convolution, the first and second derivatives of X(n,σ) and Y(n,σ) can be calculated as follows:

和

分别表示根据一维高斯函数的一阶和二阶导数生成的一维高斯核的第k个权重值；in,

and

respectively represent the kth weight value of the one-dimensional Gaussian kernel generated according to the first-order and second-order derivatives of the one-dimensional Gaussian function;

Since the curvature of the contour at the nth pixel is calculated as follows:

Therefore, for the retained contour, the curvature of each pixel of the contour can be calculated, and the result is shown in Figure 3(f), where the fingertip corresponds to a point with zero curvature, and thus the position of the user's fingertip can be obtained.

Step 3: Determine the text line indicated by the user according to the position of the user's fingertip; the specific processing process is as follows:

Step 31, the extraction of the text area;

First, convert the color image taken by the camera into a grayscale image, as shown in Figure 5(a), and then use the Ostu adaptive threshold method to binarize the grayscale image, as shown in Figure 5(b), to obtain The foreground area (finger) and background area (text document) of this image. Since the background area may contain non-text areas, the constraints need to be further strengthened to exclude non-text areas in the background area. To this end, first, extract all the connected regions in the background area of the image to form a set _CR ; then, obtain its rotation rectangle (minimum circumscribed rectangle) for each connected region in the set _CR , denoted as ζ(o ,θ,w,h); where o represents the center of the rotating rectangle, θ represents the angle (rotation angle) deflected by the rotating rectangle, which is the counterclockwise rotation of the horizontal axis (x-axis), and the first side of the rectangle that encounters The included angle of , whose range is (-π/2,0), w and h respectively represent the two adjacent sides of the rotated rectangle, as shown in Figure 4. Then, the non-text regions in the set _CR are filtered according to the following constraints.

1) Area filtering. Through a large number of experiments, it can be found that the area size of the text area is within a certain range, that is, it satisfies:

t _min <ζ _Area <t _max ;

Wherein, ζ _Area =wh, t _min and t _max are the upper and lower area thresholds of the text area, respectively, and the values are determined according to the area of the text area measured by multiple experiments, and are respectively set to 100 and 1500 in this embodiment;

The area of the non-text area is randomly sized, so the first filtering of the non-text area can be performed according to the area area. That is, for each connected area in the set C, if its rotating rectangle does not satisfy the constraint condition t _min <ζ _Area <t _max , the rotating rectangle can be considered as a non-text area, and it is deleted from the set _CR , and the set _CR The remaining connected regions in the middle form a set C _R1 ; the result is shown in Figure 5(c);

2) Angle filtering. Since the scene concerned by the present invention is plain text and text with illustrations (such as books, etc.) rather than text in complex images (such as posters, etc.), in the set C _R1 , the real text area is more than the non-text area There is an absolute quantitative advantage. And note that in the 26 English letters, except o is an exception (the angle deflected by the rotating rectangle is always zero), the angle deflected by the rotating rectangle of the text area of other letters is not much different, that is, the rotating rectangle of most of the text areas. The deflected angle θ all satisfy the following constraints:

|θ-μ _θ |<σ _θ ;

where μ _θ and σ _θ are the mean and variance of the angles deflected by the rotating rectangles of all connected regions in the set C processed in step 1).

Therefore, for each connected region in the set C1, if its rotated rectangle does not satisfy the constraint |θ-μ _θ |<σ _θ , then the rotated rectangle can be considered as a non-text region, and it is deleted from the set C _R1 , and the set The remaining connected regions in C _R1 constitute a set C _R2 . Although the constraint |ζ( _θ )-μθ|< _σθ may clear the text area at the edge of the image, since the text detection and recognition processing is mainly aimed at the middle area of the image, this processing will not affect the final text. The recognition accuracy has a great impact, and the results are shown in Figure 5(d).

3) Filter based on the relationship between text areas. In the scene concerned by the present invention, the text areas do not appear independently but form text lines. Therefore, the centers of the rotated rectangles of all the text areas belonging to the same text line will have a linear relationship. The reference line is fitted to the centers of the rotated rectangles of all text areas. Finally, the text area is determined according to the distance difference between the center of the rotation rectangle of the non-text area and the center of the rotation rectangle of the text area from the reference line. The key of this problem becomes how to determine Which text regions in the image belong to the same text line, and after determining which text regions belong to the same text line, a reference straight line can be obtained by fitting according to the centers of the rotated rectangles of these text regions. The specific implementation includes the following steps:

3.1) Taking the center of the rotated rectangle of each connected region R in the set C _R2 as a point of interest,

For a straight line passing through the point of interest, rewrite its straight line equation y=kx+b into the following form:

若把直线的参数(ρ_SL,θ_SL)当成未知量，则在(ρ_SL,θ_SL)参数空间该直线将对应为一条正弦曲线。其中，θ_SL的取值范围(-π/2,π/2)(逆时针旋转为正，顺时针旋转为负)，ρ_SL的取值范围为(-D,D)，D为相机拍摄的原始图像对角线的长；Among them, θ _SL is the angle between the straight line and the x-axis, ρ _SL is the distance from the origin O to the straight line (the origin O is the upper left vertex in the image, the horizontal axis is the y-axis, the right direction is positive, and the vertical axis is the x-axis, remove direction is positive),

If the parameters of the straight line (ρ _SL , θ _SL ) are regarded as unknown quantities, the straight line will correspond to a sine curve in the (ρ _SL , θ _SL ) parameter space. Among them, the value range of θ _SL (-π/2, π/2) (counterclockwise rotation is positive, clockwise rotation is negative), the value range of ρ _SL is (-D, D), D is the camera shooting The length of the original image diagonal;

3.2) Subdivide the (ρ _SL , θ _SL ) parameter space into multiple accumulator units, and denote the value of the accumulator unit at the coordinates (ρ _k , θ _k ) as A(ρ _k , θ _k ); in this In the embodiment, θ _k is taken as an integer degree of (-90, 90); first, all accumulator units are set to zero, and then each focus point (x _i , y _i ) is calculated separately to the straight line xcosθ _k +ysinθ _k = distance d of ρ _k :

d=|x _i cosθ _k +y _i sinθ _k -ρ _k |;

The distances from all attention points to the straight line xcosθ _k +ysinθ _k =ρ _k are judged in turn. For each concern point corresponding to a distance smaller than the threshold, the value of A(ρ _k ,θ _k ) is increased by 1. After the judgment, the final result is obtained. The A(ρ _k , θ _k ) value of , the final result of this experiment is shown in Figure 6(c). The final A(ρ _k , θ _k ) value represents the number of points of interest contained in the long strip with the straight line xcosθ _k +ysinθ _k =ρ _k as the axis, so the higher the A(ρ _k , θ _k ) value is , which means that the probability that this line is the reference line is greater. Thus, a threshold value can be set, which is set to 7 in this embodiment. Once the value of A(ρ _k , θ _k ) is higher than the threshold value, the corresponding straight line is considered to be a reference straight line, and the number of reference straight lines remembered is N;

3.3) Find the most likely text region by unsupervised line clustering, the specific process is:

3.31) Input the set of attention points, and initialize the reference line set _CL , which includes N reference lines, which are respectively the N reference lines obtained in step 3.2;

3.32) Calculate the distance from all points of interest in the point of interest set to each reference line in the set _CL ; for each point of interest, take the minimum distance value; filter out the points whose minimum distance value is less than the set threshold; Marking the category of attention points, and assigning the attention points whose minimum distance value corresponds to the same reference line to the same category;

3.33) Perform straight line fitting on the points of interest of the same category, and judge whether the difference between the slope and intercept of the newly fitted straight line and the reference straight line corresponding to the category is less than the set threshold, if it is less than the set _CL The reference line corresponding to this category remains unchanged, otherwise, update the reference line corresponding to this category in the set _CL to the newly fitted line; if all the reference lines in the set _CL remain unchanged in this step, Then output a set of attention points C _S with category labels, and the area contained in the rotation rectangle corresponding to these attention points is the most likely text area, otherwise return to step 3.31); the final reference line obtained by clustering is shown in Figure 6 ( d), the resulting point set is shown in Figure 6(e).

Step 32, determine the text line;

According to the fingertip position obtained in step 2, determine a rectangular area of interest, the length of the area of interest is equal to the length of the original image, and the width of the area of interest is set to a fixed value (set to one-sixth of the width of the original image) ), the fingertip is located on the bottom edge of the region of interest;

For the text area obtained in step 31, as shown in Figure 8(a), the outermost contour of each character is obtained, and the adjacent points between the contours are 8 connected areas. Since for most English letters, the bottommost points of the letters are almost in a straight line, even when there is rotation. Therefore, the bottommost point of each contour can be selected as the reference point, and then according to whether the reference point is in the region of interest, if not, these reference points are filtered out, as shown in Figure 7(b), and then only the region of interest will be filtered. operate on the datum point. It is worth noting that there are some special letters, such as g, q, y, j, p and other letters whose reference points are below the ideal text line, and the reference points of these letters are regarded as abnormal reference points. Three adjacent reference points are selected each time, with the goal of minimizing the sum of the distances from these three adjacent reference points to the straight line, and a straight line is fitted to obtain multiple straight lines. In order to remove the straight lines fitted by the abnormal reference points, all the fitted straight lines are scored separately. The scoring formula is as follows:

Among them, d(i) _distance is the distance from the i-th benchmark point to the straight line, n is the total number of benchmark points selected, and μ _score is the score. The lower the score, the better the result. Therefore, the straight line with the lowest score can be selected as the judgment line, and then the abnormal reference points can be filtered out according to whether the distance between the selected reference points and the judgment line is less than the set threshold, and then based on all the remaining reference points, these reference points can be minimized. The sum of the distances from the point to the straight line is the target, and a straight line is fitted, and the fitted straight line is the text line indicated by the user, as shown in Figure 7(c).

Step 4: Extract the words on the text line indicated by the user and convert it to speech output. The specific processing process is as follows:

Step 41, identify the first word;

For the text region extracted in step 31, extract its part in the region of interest as the target text region; respectively obtain the minimum circumscribed rectangle (rotated rectangle) of each character in the target text region as a letter box; The difference between the distance between the centers of two adjacent letter boxes within a word and the distance between the centers of two adjacent letter boxes between words is used to cluster the letter boxes to synthesize words. Along the text line, if the distance between the current letter box and the next letter box is less than the threshold, it is considered that these two letter boxes belong to one word. In this embodiment, the threshold is set to 20 pixels, and the above process is repeated until all the text lines are The letter boxes are all judged by this, and the smallest circumscribed rectangle belonging to all letter boxes in a word is obtained as the word box;

Considering that the input image is often not in the horizontal direction, and because the rotation of the letters will have a greater impact on the accuracy of the final recognition, according to the angle between the text line in the image and the horizontal direction (determined according to the slope of the text line), the image Perform angle compensation so that the text line is rotated to the horizontal direction before subsequent processing, as shown in Figure 8(b) and Figure (c).

Select the first word box along the text line, and then use the tesseract OCR recognition engine to recognize it. The results returned by the recognition include: word, word confidence, and word box. When the confidence of the returned word is greater than the threshold (set to 80 in the experiment), the word is considered to be correctly recognized, and the correctly recognized word is output by voice (the word will be read aloud).

It is worth noting that the domain of interest of the above recognition operation is the middle area of the image, and the homography between the image and the paper caused by the shooting angle during the recognition process has little effect on the recognition (because the shooting angle is not vertical shooting. , but shoot along the direction of the fingertip, the text in the captured image will be deformed relative to the real text, but the region of interest is the middle of the image, and the deformation is small, so it has little effect on the accuracy) , so the interference factor can be ignored.

Step 42, adopt the template matching method to track the position of the word frame of the correctly identified word on the follow-up image frame, to determine the new word area on the follow-up image frame, and identify the new word in the new word area;

i. Due to the fact that the image is not high-definition due to motion blur and other reasons during actual recognition, this will affect the correct tracking of words, so first perform binarization processing on each frame of image;

Initialize s=1, initialize the fingertip speed V _fingertip ; take the word frame corresponding to the jth word identified in the lth frame image and all the letter frames it contains as the word frame and letter frame that need to be tracked; take the mth frame The region of interest in the image is used as the search region; initialize m=l+1;

ii. For the word box/letter box that needs to be tracked, use template matching to track its position in the search area;

The present invention adopts the standard squared difference matching algorithm, that is, the following function is minimized:

In the above formula, T(x', y') represents the pixel value at the coordinate (x', y') on the word frame/letter frame to be tracked in the lth frame of image, I(x+x', y+y ') represents the pixel value at the coordinates (x+x', y+y') in the search area, and (x, y) represents the coordinates of the upper left vertex of the search area. The smaller the R _{sq_diff} , the more successful the match. Therefore, in the actual operation, when the index value is less than the given threshold, it is considered that the matching is successful, that is, the word box/letter box is tracked successfully;

If the current word frame to be tracked is successfully tracked, go to step iii; otherwise, determine whether there is a successfully tracked word frame before, if so, first determine the matching position on the m-th frame image with the matching position of the latest tracked successful word frame. A cutoff line, the cutoff line coincides with the right side of the matching position; then determine whether the width of the cutoff line from the left side of the image is less than the set threshold (0.6 times the horizontal width of the image), if so, the target text area to the right of the cutoff line is used as the new word area, Continue to recognize new words in the new word area to synchronize the word reading with the fingers; otherwise, end the new word recognition, so that the tracking and recognition of the word can be completed; The image is used as the current image, and steps 1 to 41 are performed on it to identify the first word;

If no word frame is successfully tracked for several consecutive frames, it is considered that the tracking fails, which is often caused by moving too fast;

iii. Update fingertip velocity V _fingertip :

Among them, V _word,j represents the horizontal distance difference between the position of the successfully tracked jth word box in the lth frame image and the mth frame image, V _letter,k represents the successfully tracked kth letter box in the lth frame image The horizontal distance difference between the frame image and the position in the mth frame image, N ₁ represents the number of successfully matched word boxes, and N ₂ represents the number of successfully matched letter boxes;

There are two functions for calculating the fingertip speed here. One is to judge which frame of image the word frame will move out of the image (if the abscissa of the upper left vertex of the word frame/letter frame on the first frame image is subtracted from the abscissa) If V _fingertip ×(ml) is less than zero, it can be judged that the word box/letter box will move out of the mth frame image). When it is judged that the word box will move out of the mth frame image, the whole word will not be discarded immediately, but the remaining letters of the word box still in the mth frame image will be retained, and the letter boxes of these letters will be retained, because according to the speed of the fingertips The definition formula shows that they will affect the calculation of fingertip speed. When it is judged that the letter box will move out of the mth frame of image, the corresponding letters are discarded, and these letter boxes do not participate in the calculation of the fingertip speed. The second is to improve the efficiency of tracking, that is, the entire image is not searched for the next word tracking, but a rectangular area is delineated as a new search area according to the speed of the fingertip, and tracking is only performed in this search area;

iv. Let s=s+1; for the word frame corresponding to the s-th word and each letter frame it contains in the l-th frame image, first judge whether the word frame and each letter frame will move out according to the fingertip speed. The mth frame image; the word box/letter box that is judged not to be moved out of the current frame image is used as the word box/letter box to be tracked; and a rectangular area is designated as a new search area, the abscissa of the upper left corner of the rectangular area = The abscissa of the upper left corner of the last tracked word box-fingertip speed-set offset value (set to 15 pixels in the embodiment of the present invention), the length of the rectangular area=the length of the last tracked word box + fingertip speed, the width of the rectangular area = the width of the last word box to be tracked + the set offset value (set to 30 pixels in this embodiment), because the actual finger movement is not a parallel movement, but a possible Along with moving down and up, add a certain deviation value, and the deviation value is obtained according to the experiment; calculate the ratio of black pixels and white pixels in the new search area, if this ratio is less than the set threshold (set to 20 in this embodiment) %), then discard the frame image, as shown in Figure 9(b), that is, do not continue to track and identify the word frame in this frame image, and let m=m+1, and re-do the aforementioned judgment and processing until a new The ratio of black pixels and white pixels in the search area of is not less than the set threshold, then enter step 5);

v. Return to step ii.

It should be noted that the above disclosures are only specific examples of the present invention, and changes that can be conceived by those skilled in the art according to the ideas provided by the present invention should fall within the protection scope of the present invention.

Claims (6)

1. a kind of text detection and identification method for the blind auxiliary reading, is characterized in that, comprises the following steps: Step 1: For the image sequence captured by the camera, determine whether the scene in the current image is that the finger is placed on the reading text, if so, go to Step 2, otherwise skip the current image of the frame, and take the next frame of the image as the current image, and perform the above steps. judgment and processing; Step 2: Position the user's fingertip in the current image; Step 3: Determine the text line indicated by the user according to the position of the user's fingertip; Step 4: Extract the words on the text line indicated by the user and convert them into speech output; The step 2 includes the following steps: Step 21. Use K-means to find the candidate area of the user's fingertip; First, use a Gaussian filter to filter the current image; Then, three two-dimensional matrices are generated according to the images of the three channels in the filtered image, and the element value in each two-dimensional matrix is the pixel value of the corresponding point on the image of the corresponding channel; For each two-dimensional matrix, sum all its columns and take the average to get a row×1 column vector m _{c_ave} ; sum all its rows and take the average to get a 1×col row vector m _{r_ave} ;Therefore, convert the current image into three column vectors and three row vectors, where col represents the total number of columns of the two-dimensional matrix, and row represents the total number of rows of the two-dimensional matrix; Each dimension of the column vector is regarded as a longitudinal data point, and the components of the same dimension of the three column vectors are regarded as the three features of the corresponding longitudinal data point to form the feature vector of the longitudinal data point, and the number of longitudinal data points is equal to the number of columns. The dimension of the vector, namely row; take each dimension of the row vector as a horizontal data point, and take the components of the same dimension of the three row vectors as the three features of the corresponding horizontal data point to form the feature vector of the horizontal data point, the horizontal The number of data points is equal to the dimension of the row vector, that is, col; Secondly, K-means is used to cluster the longitudinal data points and the horizontal data points respectively, and the number of clusters is 2; Again, the result of the clustering of longitudinal data points is expressed as a longitudinal label vector, which is a row×1 column vector, and the components of each dimension are expressed as the labels of the longitudinal data points of the corresponding dimension, and the value is 0 or 1 ; Express the result of the clustering of the horizontal data points as a horizontal label vector, which is a 1×col row vector, and the components of each dimension are expressed as the labels of the horizontal data points of the corresponding dimension, taking the value 0 or 1; Perform mean filtering on the vertical label vector and horizontal label vector respectively, and then perform threshold processing. If the value of a dimension element is greater than or equal to the set threshold, set it to 1, otherwise set it to 0 to obtain the final vertical label vector and horizontal label vector; Set the intersection point of the corresponding horizontal line in the current image with the demarcation point of element values 0 and 1 in the vertical label vector and the corresponding vertical line in the current image of the left demarcation point of element values 0 and 1 in the horizontal label vector as the upper left Vertex, delineate a rectangular area as a candidate area for the user's fingertip; Step 22. Position the fingertip by calculating the curvature; First, the canny operator is used to obtain the edge in the candidate area of the user's fingertip, and the edge is connected to obtain the contour; if multiple contours are obtained, only the contour containing the number of pixels not less than the set threshold is retained; Then, smooth the remaining contours; Finally, for the smoothed contour, the curvature of each pixel on the contour is calculated, and the point with zero curvature is the position of the user's fingertip. 2. the text detection and identification method for the blind reading assistance according to claim 1, is characterized in that, whether the scene in the described step 1 judges whether the scene in the current image is that the finger is placed on the method for reading the text as follows: Step 11. Pre-shoot some typical images containing the user's fingers and the text area where they are located by the camera, and save them in the database; Step 12, using the current image and multiple images taken in front of the image as sample images; Step 13, normalize the RGB color space of the image in the database and the sample image respectively; Step 14. For each sample, calculate the Euclidean distance between the normalized red channel image and the normalized red channel image of each image in the database, and use the minimum value in the result as the matching of the sample. Score; obtain the mean μ _Im and variance σ _Im of all sample matching scores, if μ _Im +σ _Im <Th, the scene in the current image is determined to be a finger on the reading text, where Th is the threshold, which is an empirical parameter. 3 . The text detection and recognition method for assisted reading for the blind according to claim 2 , wherein, in the step 14, all images are reduced to a set size, and then the Euclidean distance is calculated. 4 . 4. the text detection and identification method for blind reading aids according to claim 1, is characterized in that, in the process of using K-means to carry out clustering to vertical data point/horizontal data point, initialize two groups of cluster centers at random , perform two clusterings, evaluate the compactness of the two clustering results, and select the clustering result with good compactness as the final clustering result. 5. the text detection and identification method for the blind reading assistance according to claim 1, is characterized in that, described step 3 comprises the following steps: Step 31, the extraction of the text area; First, the current image is converted into a grayscale image; Then, the grayscale image is binarized to obtain the foreground area and the background area of the image; Then exclude the non-text areas in the background area as follows: First extract all the connected regions in the background area to form a set _CR ; then each connected region in the set _CR is obtained its rotation rectangle, denoted as ζ(ο, θ, w, h), wherein o represents The center of the rotating rectangle, θ represents the angle deflected by the rotating rectangle, which is the counterclockwise rotation of the horizontal axis, and the angle between the first side of the rotating rectangle encountered, w and h respectively represent the two adjacent sides of the rotating rectangle; Then filter out the connected areas in which the area of the rotated rectangle and the deflected angle in the _CR do not meet the constraints. For the remaining connected areas, further filtering is performed based on the relationship between the text areas. The specific implementation includes the following steps: 3.1) Take the upper left vertex of the current image as the origin O, the length direction of the current image is the y-axis, the right direction is positive, the width direction of the current image is the x-axis, and the removal direction is positive; As a point of interest, each line passing through the point of interest in the current image is represented as follows: xcosθ _SL +ysinθ _SL =ρ _SL ; Among them, θ _SL is the angle between the straight line and the x-axis, ρ _SL is the distance from the origin O to the straight line, the value range of θ _SL is (-π/2, π/2), and the value range of ρ _SL is (-D , D), D is the diagonal length of the original image captured by the camera; 3.2) Subdivide the (ρ _SL , θ _SL ) parameter space into multiple accumulator units, and denote the value of the accumulator unit at the coordinates (ρ _k , θ _k ) as A(ρ _k , θ _k ); All accumulator cells are set to zero, and the distance d from each point of interest (x _i , y _i ) to the line xcosθ _k +ysinθ _k =ρ _k is calculated separately: d=|x _i cosθ _k +y _i sinθ _k -ρ _k |; The distances from all attention points to the straight line xcosθ _k +ysinθ _k =ρ _k are judged in turn. For each concern point corresponding to a distance smaller than the threshold, the value of A(ρ _k ,θ _k ) is increased by 1. After the judgment, the final result is obtained. The A(ρ _k , θ _k ) value of ; if the A(ρ _k , θ _k ) value is higher than the threshold value, the corresponding straight line is considered as a reference straight line, and the number of reference straight lines is N; 3.3) Find the text area by the method of unsupervised line clustering, the specific process is as follows: 3.31) Input the set of attention points, and initialize the reference straight line set _CL , which includes N reference straight lines, which are respectively the N reference straight lines obtained in step 3.2); 3.32) Calculate the distance from all points of interest in the point of interest set to each reference line in the set _CL ; for each point of interest, take the minimum distance value; filter out the points whose minimum distance value is less than the set threshold; Marking the category of attention points, and assigning the attention points whose minimum distance value corresponds to the same reference line to the same category; 3.33) Perform straight line fitting on the points of interest of the same category, and judge whether the difference between the slope and intercept of the newly fitted straight line and the reference straight line corresponding to the category is less than the set threshold, if it is less than the set _CL The reference line corresponding to this category remains unchanged, otherwise, update the reference line corresponding to this category in the set _CL to the newly fitted line; if all the reference lines in the set _CL remain unchanged in this step, Then output the set of attention points C _S with category marks, and the area contained in the rotation rectangle corresponding to these attention points is the extracted text area, otherwise return to step 3.31); Step 32, determine the text line; According to the fingertip position obtained in step 2, determine a rectangular region of interest, and the fingertip is located on the bottom edge of the region of interest; To the text area that step 31 is extracted, seek the outermost contour of each character wherein; Select the bottom point of each contour as the reference point, and filter out the reference point located in the region of interest; For the selected datum points, select three adjacent datum points each time, and perform straight line fitting to obtain multiple straight lines; All the fitted straight lines are scored separately. The scoring formula is as follows:

Among them, d(i) _distance is the distance from the i-th benchmark point screened to the straight line, n is the total number of benchmark points screened out, and μ _score is the score; Select the straight line with the lowest score as the judgment line; filter out the datum points whose distance to the judgment line is less than the set threshold, and then perform straight line fitting based on all the remaining datum points, and the fitted line is Line of text indicated for the user. 6. the text detection and identification method for blind reading aids according to claim 5, is characterized in that, the concrete processing process of described step 4 is as follows: Step 41, identify the first word; For the text area extracted in step 31, extract the part located in the area of interest as the target text area; respectively obtain the minimum circumscribed rectangle of each character in the target text area as a letter box; The difference between the distance between the centers of two adjacent letter boxes and the distance between the centers of two adjacent letter boxes between words, cluster the letter boxes, and synthesize words; find the smallest circumscribed rectangle belonging to all letter boxes in a word as the word box ; Perform angle compensation on the image according to the angle between the text line indicated by the user and the horizontal direction in the image, so that the text line is rotated to the horizontal direction; Select the first word box along the text line, and then use OCR recognition technology to identify, the results returned by the recognition include: word, word confidence and word box; when the returned word confidence is greater than the threshold, the word is considered to be Correct recognition, voice output for correctly recognized words; Step 42, adopt the template matching method to track the position of the word frame of the correctly identified word on the follow-up image frame, to determine the new word area on the follow-up image frame, and identify the new word in the new word area; i. Binarize each frame of image; Initialize s=1, initialize the fingertip speed V _fingertip ; take the word frame corresponding to the jth word identified in the lth frame image and all the letter frames it contains as the word frame and letter frame that need to be tracked; take the mth frame The region of interest in the image is used as the search region; initialize m=l+1; ii. For the word box/letter box that needs to be tracked, use template matching to track its position in the search area; If the current word frame to be tracked is successfully tracked, go to step iii; otherwise, determine whether there is a successfully tracked word frame before, if so, first determine the matching position on the m-th frame image with the matching position of the latest tracked successful word frame. A cutoff line, the cutoff line coincides with the right side of the matching position; then judge whether the width of the cutoff line from the left side of the image is less than the set threshold, if so, take the target text area to the right of the cutoff line as the new word area, and continue to identify new words in the new word area. Otherwise, end the new word recognition; if there is no word frame successfully tracked before, then take the mth frame image as the current image, and perform steps 1 to 41 on it to recognize the first word; iii. Update fingertip velocity V _fingertip :

Among them, V _word,j represents the horizontal distance difference between the position of the successfully tracked jth word box in the lth frame image and the mth frame image, and V _letter,k represents the successfully tracked kth letter box in the lth frame The horizontal distance difference between the image and the position in the mth frame image, N ₁ represents the number of successfully matched word boxes, and N ₂ represents the number of successfully matched letter boxes; iv. Let s=s+1; For the word box corresponding to the sth word and each letter box it contains in the lth frame image, judge whether the word box and each letter box will move out of the mth frame image according to the fingertip speed. If the lth frame image If the abscissa of the upper left vertex of the upper word box/letter box minus V _fingertip ×(ml) is less than zero, it is determined that the word box/letter box will move out of the mth frame image; The word box/letter box that is judged not to be moved out of the current frame image is used as the word box/letter box that needs to be tracked; and a rectangular area is defined as a new search area, the abscissa of the upper left corner of the rectangular area = the previous tracked area The abscissa of the upper left corner of the word box - fingertip speed - set the offset value, the length of the rectangular area = the length of the last word box to be tracked + the speed of the fingertip, the width of the rectangular area = the length of the last word box to be tracked width + set offset value; Calculate the ratio of black pixels to white pixels in the new search area, if this ratio is less than the set threshold, discard the frame image, and let m=m+1, and repeat the above judgment and processing until the black pixels in the new search area If the ratio to the white pixel is not less than the set threshold, then enter step v; otherwise, directly enter step v; v. Return to step ii.

Images