WO2024244760A1 - Mathematical formula recognition method and apparatus, electronic device, and readable storage medium - Google Patents

Abstract

The present disclosure relates to a mathematical formula recognition method and apparatus, an electronic device, and a readable storage medium. The method comprises: acquiring an original image comprising a mathematical formula; inputting the original image into a formula recognition model to obtain a predicted character set outputted by the formula recognition model, wherein the predicted character set comprises character data and structural data; and, according to a preset formula format and the structural data, restoring the position of the character data in the mathematical formula to obtain the mathematical formula in the original image. In the present embodiments, a relative positional relationship between the character data can be represented by means of the provision of the structural data, thereby eliminating the problem of missing characters in a complex formula recognition process, improving the recognition accuracy.

Description

Mathematical formula recognition method, device, electronic device and readable storage medium Technical Field

The present disclosure relates to the field of data processing technology, and in particular to a mathematical formula recognition method, device, electronic device and readable storage medium.

Background Art

At present, handwritten formula recognition is usually implemented using an encoding and decoding architecture. For example, a handwritten formula image is input into an encoder, which extracts an image feature map; then, the image feature map is input into a decoder, which recognizes the handwritten formula character by character and finally recognizes the handwritten formula.

The encoding-decoding architecture in the existing solution is based on latex tags for character-by-character prediction. However, when there are complex structures such as fractions, radicals, exponents, logarithms, etc. that are nested in the formula to be recognized, it is very easy to cause "{" or "}" to be lost, especially when the formula to be recognized is long, it is easy to cause some characters to be lost during the recognition process, resulting in recognition errors and reducing the recognition accuracy.

Summary of the invention

The present disclosure provides a mathematical formula recognition method, device, electronic device and readable storage medium to address the deficiencies of the related art.

According to a first aspect of an embodiment of the present disclosure, a mathematical formula recognition method is provided, the method comprising: acquiring an original image containing a mathematical formula; inputting the original image into a formula recognition model to obtain a predicted character set output by the formula recognition model; the predicted character set comprising character data and structural data; restoring the position of the character data in the mathematical formula according to a preset formula format and structural data to obtain the mathematical formula in the original image.

Optionally, the formula recognition model includes an encoder and a decoder; the encoder is used to obtain an image feature map corresponding to the original image; and the decoder is used to determine a predicted character set corresponding to the original image based on the image feature map.

Optionally, the encoder includes a DenseNet network or a Transformer network.

Optionally, the decoder includes a multi-scale counting module, a cyclic decoding module, a feature fusion module and a character prediction module; the multi-scale counting module is used to convert the image feature map into a plurality of counting vectors of preset scales; the cyclic decoding module is used to obtain character latents according to the image feature map; the feature fusion module is used to fuse the counting vector, the character latent vector and the previous predicted character vector to obtain a target vector; the character prediction module is used to predict character data and structure data respectively according to the target vector.

Optionally, the circular decoding module is also used to obtain a context vector based on the image feature map; the feature fusion module is also used to fuse the count vector, the context vector, the character latent vector and the previous predicted character vector to obtain a target vector.

Optionally, the feature fusion module performs linear transformation processing on the count vector, the context vector, the character latent vector and the previous predicted character vector, respectively, to obtain a first linear vector, a second linear vector, a third linear vector and a fourth linear vector; and obtains the sum vector of the first linear vector, the second linear vector, the third linear vector and the fourth linear vector to obtain a target vector.

Optionally, the multi-scale counting module includes at least two sub-counting modules and a vector averaging sub-module; the at least two sub-counting modules contain convolution kernels of different sizes and are used to output sub-vectors of the same size, and each sub-vector is used to represent the number of times character data is predicted at different scales; the vector averaging sub-module is used to obtain the average vector of at least two sub-vectors to obtain a counting vector.

Optionally, the multi-scale counting module includes a first sub-counting module and a second sub-counting module; the first sub-counting module is used to identify feature information of a first preset scale in the image feature map to obtain a first sub-vector; the first sub-vector is used to represent the number of character data recognized under the convolution kernel of the first preset scale; the second sub-counting module is used to identify feature information of a second preset scale in the image feature map to obtain a second sub-vector; the second sub-vector is used to represent the number of character data recognized under the convolution kernel of the first preset scale.

Optionally, the first sub-counting module includes a first convolution unit, a channel unit, a conversion unit and a pooling unit; the first convolution unit is used to convert the image feature map into a first feature map using a convolution kernel of a first preset size; the channel unit is used to assign corresponding weights to different channels of the first feature map to obtain an attention feature map; and the attention feature map and the first feature map are multiplied to obtain a channel feature map; the conversion unit is used to scale the channel feature map to obtain a counting feature map; the pooling unit is used to pool the counting feature map to obtain a first sub-vector.

Optionally, the second sub-counting module includes a second convolution unit, a channel unit, a conversion unit and a pooling unit; the second convolution unit is used to convert the image feature map into a second feature map using a convolution kernel of a third preset size; the channel unit is used to assign corresponding weights to different channels of the second feature map to obtain an attention feature map; and the attention feature map and the second feature map are multiplied to obtain a channel feature map; the conversion unit is used to scale the channel feature map to obtain a counting feature map; the pooling unit is used to pool the counting feature map to obtain a second sub-vector.

Optionally, the multi-scale counting module further includes a third sub-counting module, and the third sub-counting module is used to identify feature information of a third preset scale in the image feature map to obtain a third sub-vector.

Optionally, the third sub-counting module includes a third convolution unit, a channel unit, a conversion unit and a pooling unit; the third convolution unit is used to convert the image feature map into a third feature map using a convolution kernel of a third preset size; the channel unit is used to assign corresponding weights to different channels of the third feature map to obtain an attention feature map; and the attention feature map and the third feature map are multiplied to obtain a channel feature map; the conversion unit is used to scale the channel feature map to obtain a counting feature map; the pooling unit is used to pool the counting feature map to obtain a third sub-vector.

Optionally, the cyclic decoding module includes a gated cyclic unit; the gated cyclic unit is used to generate the character latent vector according to the latent vector and a previous predicted character vector.

Optionally, the loop decoding module includes a first gated loop unit and a second gated loop unit; the first gated loop unit is used to generate a first gated vector based on a latent vector and a previous predicted character vector; the second gated loop unit is used to generate the character latent vector based on the first gated vector.

Optionally, the cyclic decoding module includes an attention submodule; the attention submodule is used to generate a context vector based on the image feature map, the all-zero tensor and the first gating vector; the second gated cyclic unit is used to generate the character latent vector based on the first gating vector and the context vector.

Optionally, the attention submodule includes a weight acquisition submodule and a context acquisition submodule; the weight acquisition submodule is used to convolve and linearly process the all-zero tensor to obtain a first eigenvector, and to obtain the image The feature map is convolved to obtain a second feature vector, and the first gating vector is linearly processed to obtain a third feature vector; and the first feature vector, the second convolution vector and the third feature vector are superimposed to obtain a fourth feature vector; the context acquisition submodule is used to perform a first activation process, a linear conversion process and a second activation process on the fourth feature vector to obtain an attention weight; and the image feature map and the attention weight are multiplied, and then summed to obtain a context vector.

Optionally, the character prediction module includes a character prediction submodule and a structure prediction submodule; the character prediction submodule is used to process the target vector to obtain predicted character data; the structure prediction submodule is used to process the target vector to obtain predicted structure data.

Optionally, the character prediction submodule includes a linear conversion unit and an activation unit; the linear conversion unit is used to perform linear conversion on the target vector; the activation unit is used to perform activation processing on the vector after linear conversion to obtain predicted character data.

Optionally, the structure prediction submodule includes a linear conversion unit and an activation unit; the linear conversion unit is used to perform linear conversion on the target vector; the activation unit is used to perform activation processing on the vector after linear conversion to obtain predicted structure data.

Optionally, the formula recognition model is trained through the following steps, including: obtaining training sample data; the training sample data includes a data set of multiple mathematical formulas; each data set includes character data and structural data obtained after expansion according to a preset formula format; the structural data is used to represent the relative position relationship of some characters in the mathematical formula; using the training sample data to train the formula recognition model to be trained to obtain a predicted character set output by the formula recognition model; the predicted character set includes character data and structural data; when it is determined that the character data meets the preset conditions, the training is stopped to obtain a trained formula recognition model.

Optionally, determining whether the character data meets a preset condition includes: obtaining the frequency of occurrence of each character data in the predicted character set to obtain the predicted frequency corresponding to each character data; obtaining the difference between the predicted frequency corresponding to each character data and the marked frequency in the label of the training sample data to obtain the prediction error value corresponding to each character data; obtaining the average value of the prediction error values corresponding to the character data in the predicted character set to obtain an error average value; when it is determined that the error average value is less than or equal to a preset average value threshold, determining that the character data meets the preset condition.

Optionally, the training sample data is acquired through the following steps, including: acquiring multiple original images containing mathematical formulas; acquiring character data corresponding to each original image; creating structural data according to a preset formula format and character data and arranging the character data to obtain training sample data corresponding to each original image.

According to a second aspect of an embodiment of the present disclosure, a mathematical formula recognition device is provided, the device comprising: an original image acquisition module, used to acquire an original image containing a mathematical formula; a prediction set acquisition module, used to input the original image into a formula recognition model to obtain a predicted character set output by the formula recognition model; the predicted character set comprises character data and structural data; and a mathematical formula acquisition module, used to restore the position of the character data in the mathematical formula according to a preset formula format and structural data to obtain the mathematical formula in the original image.

According to a third aspect of an embodiment of the present disclosure, there is provided an electronic device, comprising a processor; and a memory for storing a computer program executable by the processor; wherein the processor is configured to execute the computer program in the memory to implement the method described in the first aspect.

According to a fourth aspect of an embodiment of the present disclosure, a computer-readable storage medium is provided. When an executable computer program in the storage medium is executed by a processor, the method described in the first aspect can be implemented.

The technical solution provided by the embodiments of the present disclosure may have the following beneficial effects:

As can be seen from the above embodiments, the scheme provided by the embodiments of the present disclosure can obtain an original image containing a mathematical formula; then, the original image is input into a formula recognition model to obtain a predicted character set output by the formula recognition model; the predicted character set includes character data and structural data; then, the position of the character data in the mathematical formula is restored according to the preset formula format and structural data to obtain the mathematical formula in the original image. In this way, by setting structural data in this embodiment, the relative position relationship of the character data can be represented, and the problem of missing characters in the complex formula recognition process can be eliminated, which is conducive to improving the recognition accuracy.

It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure.

BRIEF DESCRIPTION OF THE DRAWINGS

The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

Fig. 1 is a flow chart showing a method for identifying a mathematical formula according to an exemplary embodiment.

Fig. 2 is a block diagram of a formula recognition model according to an exemplary embodiment.

Fig. 3 is a block diagram of a decoder according to an exemplary embodiment.

Fig. 4 is a block diagram of a multi-scale counting module according to an exemplary embodiment.

Fig. 5 is a block diagram of a first sub-counting module according to an exemplary embodiment.

Fig. 6 is a block diagram showing a cyclic decoding module according to an exemplary embodiment.

Fig. 7 is a block diagram of an attention submodule according to an exemplary embodiment.

Fig. 8 is a block diagram of a character prediction module according to an exemplary embodiment.

Fig. 9 is a block diagram of a formula recognition model according to an exemplary embodiment.

Fig. 10 is a flowchart showing a training formula recognition model according to an exemplary embodiment.

Fig. 11 is a schematic diagram showing the relationship between a mathematical formula, a preset formula format and a latex tag according to an exemplary embodiment.

Fig. 12 is a flowchart showing a training formula recognition model according to an exemplary embodiment.

Fig. 13 is a block diagram of a mathematical formula recognition device according to an exemplary embodiment.

DETAILED DESCRIPTION

Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described exemplarily below do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices consistent with some aspects of the present disclosure as detailed in the attached claims. It should be noted that the features in the following embodiments and implementations may be combined with each other without conflict.

Considering that the mathematical formulas with complex structures in the related art may lose "{" or "}" during the recognition process, Especially when the formula to be recognized is long, some characters are easily lost during the recognition process, which may cause the problem of mathematical formula recognition errors.

In order to solve the above technical problems, the present disclosure provides a mathematical formula recognition method, and the inventive concept thereof includes:

First, structural information is added to the training sample data. This structural information can represent the relative position relationship of some characters in the mathematical formula, making the mathematical formula clearer and more accurate. When training the formula recognition model, it can enable the formula recognition model to more accurately predict the position of each character data in the mathematical formula, thereby improving the accuracy of the formula recognition model.

Second, a multi-scale counting module is set in the decoder of the formula recognition model. The multi-scale counting module can obtain the frequency of occurrence of character data in the predicted character set. The above frequency can be used to construct a loss function to realize the training process of the supervised formula recognition model, which is beneficial to improve the accuracy of character recognition by the formula recognition model, and then help improve the accuracy of the overall recognition of mathematical formulas.

The present disclosure provides a mathematical formula recognition method, which can be applied to electronic devices, including but not limited to smart phones, tablet computers, smart display screens, electronic whiteboards, etc. FIG1 is a flowchart of a mathematical formula recognition method according to an exemplary embodiment.

Referring to FIG. 1 , a mathematical formula recognition method includes steps 11 to 13:

In step 11, an original image containing a mathematical formula is obtained.

In this embodiment, the electronic device can obtain an original image containing a mathematical formula. In one example, the electronic device can be provided with an image acquisition module, such as a camera or an image sensor. When there is a need to acquire the original image, for example, when there is an object in the preview area of the image acquisition module, for example, the object can be a paper with a mathematical formula written on it, the electronic device can control the image acquisition module to shoot the object, thereby obtaining the original image. In another example, the electronic device can be provided with a communication module, such as a WiFi module, an infrared module, a USB module, etc., and the electronic device can communicate with the peer communication module of other devices through the above communication module, thereby reading the original image containing the mathematical formula from other devices.

It should be noted that the electronic device does not determine whether the image contains a mathematical formula when acquiring the image. In the subsequent embodiments, it is assumed that the image contains a mathematical formula for the convenience of description.

It should also be noted that the original image in the subsequent embodiments may be a color image (ie, an RGB image) or a grayscale image. In one example, the original image is implemented as a grayscale image, thereby reducing the amount of data processing of the formula recognition model and improving recognition efficiency.

In step 12, the original image is input into a formula recognition model to obtain a predicted character set output by the formula recognition model; the predicted character set includes character data and structure data.

In this embodiment, the electronic device may store a formula recognition model. The formula recognition model may include an encoder and a decoder. Referring to FIG. 2 , the encoder 21 is used to obtain an image feature map corresponding to the original image; the decoder 22 is used to determine a predicted character set corresponding to the original image according to the image feature map.

In one example, the encoder 21 may be implemented by a DenseNet network or a Transformer network. Taking the encoder 21 implemented by a DenseNet network as an example, the input data of the DenseNet network is an original image of size H*W*1, and the output data thereof is an image feature map, and the size of the image feature map is H/16*W/16*684. Wherein, H represents the height of the original image, and W represents the width of the original image.

In one example, the decoder 22 includes a multi-scale counting module, a cyclic decoding module, a feature fusion module and a character prediction module. Referring to FIG. 3 , the multi-scale counting module (MSCM) 31 is used to The image feature map E(X) is converted into counting vectors (CV) of multiple preset scales.

The loop decoding module 32 obtains the character latent vector M and the context vector Ω corresponding to the image feature map;

The feature fusion module 33 is used to combine the count vector CV, the character latent vector M and the previous predicted character vector Perform linear transformation to obtain the target vector LB;

The character prediction module 34 is used to predict character data and structure data according to the target vector LB, respectively. The character data and structure data constitute a predicted character set.

It should be noted that, in one example, the target vector LB is the sum vector of the first linear vector, the third linear vector and the fourth linear vector, that is, the sum of the numbers of the same position of each linear vector is obtained as the value of the sum vector at the same position, so that after the summation process, the dimension of the target vector is the same as that of each linear vector. For example, if the first linear vector, the second linear vector, the third linear vector and the fourth linear vector are all linear vectors of 1*512 dimension, then the target vector is also a linear vector of 1*512 dimension.

In one example, the multi-scale counting module 31 includes at least two sub-counting modules and a vector averaging sub-module; the at least two sub-counting modules contain convolution kernels of different sizes and are used to output sub-vectors of the same size, and each sub-vector is used to represent the number of times character data is predicted at different scales; the vector averaging sub-module is used to obtain the average vector of at least two sub-vectors to obtain a counting vector.

In one example, the multi-scale counting module 31 includes a first sub-counting module and a second sub-counting module; the first sub-counting module is used to identify feature information of a first preset scale in the image feature map to obtain a first sub-vector; the second sub-counting module is used to identify feature information of a second preset scale in the image feature map to obtain a second sub-vector.

In another example, the multi-scale counting module further includes a third sub-counting module, and the third sub-counting module is used to identify feature information of a third preset scale in the image feature map to obtain a third sub-vector. In this example, the multi-scale counting module 31 includes a first sub-counting module, a second sub-counting module, a third sub-counting module and a vector average sub-module (Element Average). Referring to FIG. 4 , the first sub-counting module 41, the second sub-counting module 42, the third sub-counting module 43 and the vector average sub-module (Element Average) 44;

The first sub-counting module 41 is used to identify feature information of a first preset scale in the image feature map E(X) to obtain a first sub-vector.

The second sub-counting module 42 is used to identify feature information of a second preset scale in the image feature map E(X) to obtain a second sub-vector.

The third sub-counting module 43 is used to identify feature information of a third preset scale in the image feature map E(X) to obtain a third sub-vector.

The vector average submodule (Element Average) 44 is used to obtain the average vector of the first subvector, the second subvector and the third subvector to obtain a counting vector CV (Counting Vector). The counting vector CV is used to represent the number of each character.

In one embodiment, referring to FIG. 5 , the first sub-counting module 41 includes a first convolution unit 411 (Conv3*3+BN), a channel unit 412, a conversion unit 413 (Conv1*1+Sigmoid) and a pooling unit 414 (Sum Pooling).

The first convolution unit 411 is used to convert the image feature map E(X) into a first feature map using a convolution kernel of a first preset size. The first convolution unit also includes a BN (Batch Normalization) layer, which is used to convert the image feature map E(X) into a feature map of a preset dimension. In addition, the first convolution unit includes a convolution layer. Kernel Conv, in one example, the size of the convolution kernel, that is, the first preset size is 3*3*684; and the number of convolution kernels is 512, so the dimension of the first feature map output by the first convolution unit is H/16*W/16*512.

The channel unit 412 is used to assign corresponding weights to different channels of the first feature map to obtain an attention feature map; and multiply the attention feature map and the first feature map to obtain a channel feature map. In one example, the above-mentioned attention channel may include but is not limited to the following activation functions: ReLU and Sigmoid. Of course, the above-mentioned attention channel may also include functions such as GAP and Linear. Since different functions have different output value ranges, corresponding weights can be assigned to different channels of the first feature map to select the attention value range under different channels. Then, the channel unit 52 can multiply the weight value of each channel with the feature map of a preset dimension to obtain a channel feature map.

Taking the attention channel including the GAP function as an example, the GAP function can perform average pooling on the first feature map, that is, average the plane of H/16*W/16 to obtain the value corresponding to the plane; after 512 times, a channel feature map of H/16*W/16*512 dimensions can be obtained.

The conversion unit 413 is used to scale the channel feature map to obtain a counting feature map. The conversion unit 413 is used to convert the channel feature map of different input dimensions into a feature map of another preset dimension, which is subsequently referred to as a counting feature map (Counting Map). In one example, the conversion unit includes a convolution kernel and a Sigmoid activation function. The size of the convolution kernel is 1*1*512, and the number of convolution kernels is K, so that K characters can be recognized, that is, the counting feature map is a 1*K dimensional vector.

The pooling unit 414 is used to perform pooling processing on the count feature map to obtain a first sub-vector.

Continuing to refer to Figure 5, the first convolution unit 411 performs 3*3*684 convolution through the convolution kernel. After 512 convolution kernels, the number of channels of the image feature map can be adjusted from 684 dimensions to 512 dimensions. Then, the channel unit 412 can obtain the weight value of each of the 512 channels, and multiply the weight value of each channel with the first feature map output by the first convolution unit 411 for weighting to obtain a channel feature map of H/16*W/16*512 dimensions. Afterwards, the conversion unit 413 adjusts the number of channels of the first feature map from 512 dimensions to 208 dimensions through 1*1*208 convolution. It should be noted that the above-mentioned 208 dimensions are the character data of the recognition network, which can be adjusted according to the mode. At this time, the dimension of the counting feature map is H*W*208. Afterwards, the pooling unit 414 may sum each H*W plane to obtain a first sub-vector of 1*208 dimension, wherein the pixel value of each of the 208 channels of the first sub-vector corresponds to the number of times the character appears in the current formula.

Continuing to refer to FIG5, the second sub-counting module 42 includes a second convolution unit 421, a channel unit 422, a conversion unit 423 and a pooling unit 424. Among them, the second convolution unit 421 is used to convert the image feature map E(X) into a second feature map using a convolution kernel of a second preset size; the channel unit 422 is used to assign corresponding weights to different channels of the second feature map to obtain an attention feature map, and multiply the attention feature map and the second feature map to obtain a channel feature map; the conversion unit 423 is used to scale the channel feature map to obtain a count feature map; the pooling unit 424 is used to pool the count feature map to obtain a second sub-vector.

It should be noted that the structure of the second sub-counting module 42 is the same as that of the first sub-counting module 41, and the difference is that the convolution units of the two are different, that is, the second sub-counting module 42 converts the image feature map into a second preset size instead of the first preset size. In one example, the second convolution unit 421 includes a convolution kernel of size 5*5*684, that is, the second preset size is 5*5*684; and the number of convolution kernels is 512, so that the dimension of the second feature map output by the second convolution unit 421 is H/16*W/16*512. And the dimension of the second sub-vector is H*W*208.

5, the third sub-counting module 43 includes a third convolution unit 431, a channel unit 432, a conversion unit 433 and a pooling unit 434. The third convolution unit 431 is used to convert the image feature map E(X) into a third feature map using a convolution kernel of a third preset size; the channel unit 432 is used to convert different channels of the third feature map Assign corresponding weights to obtain an attention feature map; and multiply the attention feature map and the third feature map to obtain a channel feature map; a conversion unit 433 is used to scale the channel feature map to obtain a count feature map; a pooling unit 434 is used to pool the count feature map to obtain a third sub-vector.

It should be noted that the structure of the third sub-counting module 43 is the same as that of the first sub-counting module 41, and the difference is that the convolution units of the two are different. In one example, the third convolution unit 431 includes a convolution kernel of size 7*7*684, that is, the third preset size is 7*7*684. In other words, the dimension of the third sub-vector is H*W*208.

In one embodiment, the loop decoding module includes a gated loop unit, which is used to generate the character latent vector based on the latent vector and the previous predicted character vector. In one example, the loop decoding module includes a first gated loop unit and a second gated loop unit; the first gated loop unit is used to generate a first gated vector based on the latent vector and the previous predicted character vector; the second gated loop unit is used to generate the character latent vector based on the first gated vector. It should be noted that the number of gated loop units can be selected according to the specific scenario. When the recognition accuracy is met, the corresponding scheme falls within the protection scope of the present disclosure.

In another example, the loop decoding module further includes an attention submodule (attention model). Referring to FIG. 6 , the loop decoding module 32 includes an attention submodule 61, a first gated loop unit 62, and a second gated loop unit 63; the first gated loop unit 62 is used to control the hidden vector and the previous prediction character vector Generate the first gating vector The two input latent vectors of the first gated recurrent unit 62 and the previous prediction character vector The dimension is 256. The previous prediction character vector The initial value generated is a 256-dimensional vector representation of the starting symbol "<sos>", and the values after the first time are the results of the previous training process. Hidden vector The initial value of is the linear transformation (dimension 1*256) of the average value (dimension 1*684) of each channel pixel of the image feature map E(X) output by the encoder, that is, the 1*256-dimensional vector corresponding to the image feature map E(X); and when it is not the first cycle This M is the character latent vector M output by the second gated recurrent unit 63.

It should be noted that the previous predicted character vector The character data can be obtained by predicting the character based on the character latent vector M; then, a table lookup operation is performed based on the character data to find the vector corresponding to the above character data, and the found vector is used as the previous predicted character vector The value of the previous prediction character vector effect.

The attention submodule 61 is used to obtain the image feature map E(X), the all-zero tensor att _α (X) and the first gate vector Generate context vector Ω; the input of attention submodule 61 is E(X), whose size is H/16*W/16*684. The output of the first gated recurrent unit 62 is Its dimension is 1*256. The initial dimension of the all-zero tensor att _α (X) is 1*1*H/16*W/16, which is used to record the historical attention area.

It should be noted that in this example, the all-zero tensor att _α (X) sets the initialization value of the vector to all zeros, so as to facilitate the operation. Of course, its initialization value can also be set to other random numbers, and the corresponding results can also be obtained. The value of the all-zero tensor att _α (X) in the non-first cycle is att _α (X) = att _α (X) + α. Where α is the update vector of the attention submodule 61 illustrated in Figure 7 after each cycle.

The second gated loop unit 63 is used to control the first gate vector The character latent vector M is generated by summing up the context vector Ω. The output result of the second gated recurrent unit 63 is M, whose dimension is 1*256.

Referring to FIG. 7 , the attention submodule 61 includes a weight acquisition submodule 71 and a context acquisition submodule 72 .

The weight acquisition submodule 71 is used to perform convolution and linear processing (Conv+linear) on the all-zero tensor att _α (X) to obtain a first feature vector, and to perform convolution processing (Conv) on the image feature map E(X) to obtain a second feature vector, and to perform convolution processing (Conv) on the first gating vector Perform linear processing (linear) to obtain a third eigenvector; and perform superposition processing ("+") on the first eigenvector, the second convolution vector and the third eigenvector to obtain a fourth eigenvector. Superposition processing refers to adding the H/16*W/16*512-dimensional vector after the all-zero tensor att _α (X) and the H/16*W/16*512-dimensional vector after the image feature map E(X) is processed to obtain a H/16*W/16*512-dimensional sum vector; then, the first gate vector Linear processing is performed to obtain a vector of 1*512 dimension; then, the sum vector of H/16*W/16*512 dimension is further summed with the vector of 1*512 dimension to obtain the fourth eigenvector of H/16*W/16*512 dimension.

The context acquisition submodule 72 is used to obtain the attention weight α after performing the first activation processing (activation function is tanh), linear conversion processing (linear) and second activation processing (activation function is softmax) on the fourth eigenvector; and multiply the image feature map E(X) and the attention weight ("X"), and then sum them (sum) to obtain the context vector Ω.

In this embodiment, the attention submodule 61 adopts the Coverage mechanism to avoid repeated recognition, the context vector Ω = Σ _h Σ _w E(X) × α, and the input att _α (X) = att _α (X) + α in the next cycle. The first gate vector After linear processing, the number of channels is adjusted to 512 dimensions. att _α (X) passes through the convolution layer (convolution kernel size is 11*11, padding=5, step size is 1), and the channel is adjusted to 512 dimensions after linear transformation. The image feature map E(X) is adjusted to 512 channels after 1*1 convolution. Then the first sub-vector, the second sub-vector and the third sub-vector after the above three channel processing are added, and then activated by the activation function tanh, and then the number of channels is compressed to 1 through linear transformation, and finally the attention map weight α is obtained after activation by the activation function softmax.

Referring to FIG8 , the character prediction module includes a character prediction submodule 81 and a structure prediction submodule 82. The character prediction submodule 81 is used to process the target vector to obtain predicted character data; the structure prediction submodule 82 is used to process the target vector to obtain predicted structure data. The character data and the structure data constitute a predicted character set.

In one example, the character prediction submodule 81 includes a linear conversion unit (Linear) and an activation unit (sigmoid). The linear conversion unit 811 is used to perform linear conversion on the target vector; the activation unit 812 is used to perform activation processing on the vector after linear conversion to obtain predicted character data.

In one example, the structure prediction submodule 82 includes a linear conversion unit (Linear) and an activation unit (softmax). The linear conversion unit is used to perform linear conversion on the target vector; the activation unit is used to perform activation processing on the vector after linear conversion to obtain predicted character data.

In conjunction with FIG. 2 to FIG. 8 , in a possible embodiment, the structure of the formula recognition model may be as shown in FIG. 9 .

Referring to FIG. 9 , by setting a multi-scale counting module 31 and a cyclic decoding module 32 in the formula recognition model, features of regions of different sizes in the image feature map can be identified, which is beneficial to improving the detection accuracy of characters.

Referring to FIG. 10 , the training process of the formula recognition model includes steps 101 to 103 .

In step 101, training sample data is obtained; the training sample data includes a data set of multiple mathematical formulas; each data set includes character data and structure data obtained after expansion according to a preset formula format; the structure data is used to represent the relative position relationship of some characters in the mathematical formula.

In this step, the electronic device can obtain training sample data. It is understandable that the training sample data includes a data set of multiple mathematical formulas, each of which includes character data and structure data obtained after expansion according to a preset formula format; the structure data is used to represent the relative position relationship of some characters in the mathematical formula.

In this step, the mathematical formula includes at least one of the following structures: fraction, subscript, superscript, radical, overline, overarrow, triangle, limit, summation symbol, etc. The symbols of the above structures are respectively \frac, _, ^, \sqrt, \overline, \overrightarrow, \widehat, \limits, \sum. It is understandable that the following structures in the mathematical formula, such as log, ln, lg or exponent, are also applicable to the scheme of the present disclosure.

Table 1: Mathematical formulas of different structures and their latex representations

As shown in Table 1, there are 7 kinds of positional relationships in mathematical formulas, namely inside, right, above, below, left superscript (left_sup, such as square root), and right superscript (right, above, below, and left superscript (left_sup, such as square root). ), subscript (sub), superscript (sup).

Take a mathematical formula including a fraction as an example, see Figure 11 to illustrate the relationship between the character position in the mathematical formula and the latex tag. The character a is above, the character b is below, and the character c is to the right. When the formula format \frac{above}{below}right is preset, the above formula It can be written as \frac{a}{b}c.

In combination with the positional relationship of characters in the mathematical formula and the preset structure shown in FIG. 11 , training sample data can be formed in this embodiment.

1. The mathematical formula is The training sample data is shown in Table 2.

Table 2: Training sample data

2. The mathematical formula is The training sample data is shown in Table 3.

Table 3: Training sample data

It should be noted that "<eos>" in Table 2 and Table 3 is the end character. The last <eos> indicates the end of the entire formula recognition, and the middle part <eos> indicates the end of each structure.

It should also be noted that in Table 2 and Table 3, "struct" indicates the relative position relationship of the characters contained therein (7 types). If there is a position relationship, fill it in; if there is no position relationship, fill in "None".

Combining Table 2 and Table 3, the first column (from top to bottom) is the formula using latex tags and adding the structure "struct" and the relative position relationship (7 types), and the 2nd to 7th columns are the relative position relationship after adding the structure. The entire table constitutes the training sample data. It can be seen that in this embodiment, by adding structural data to the training sample data, the mathematical formula can be more accurately represented, so as to facilitate the subsequent accurate restoration of the mathematical formula.

Based on the above principle, a data set of several mathematical formulas can be obtained to obtain training sample data.

In one embodiment, several users may write handwritten mathematical formulas and take pictures to obtain several original images containing mathematical formulas. The electronic device stores a preset image recognition model, such as a convolutional neural network, and uses the image recognition model to recognize each original image to obtain character data corresponding to each original image. Alternatively, the character data corresponding to each original image is annotated manually, and the electronic device can obtain the input character data. Finally, the electronic device can create structure data and arrange the character data according to the preset formula format and character data to obtain training sample data corresponding to each original image. This embodiment can not only improve the diversity of mathematical formulas, but also improve the efficiency of obtaining training sample data.

In step 102, the training sample data is used to train the formula recognition model to be trained, and a predicted character set output by the formula recognition model is obtained; the predicted character set includes character data and structure data.

In this step, the electronic device can sequentially input each training sample data into the formula recognition model to be trained, and the formula recognition model can process the training sample data to obtain a predicted character set. It is understandable that the predicted character set includes character data and structure data.

In this step, the reasoning process of the formula recognition model is shown in Figure 12. Referring to Figure 12, it includes: first, the structure information pushed into the stack is [position information, parent_symbol], and the prediction result pre_string is initialized to '', which is used to store the recognized character data, and is the representation of the latex tag of the mathematical formula in Figure 11. Character by character recognition is performed to obtain the current character current_symbol, and parent_symbol = current_symbol, that is, the value of current_symbol is assigned to parent_symbol. Then, it is determined that the current character current_symbol is not equal to 'struct' and current_symbol is not the end character '<eos>'. When current_symbol is not 'struct' and '<eos>', pre_string + = current_symbol, that is, the current character is added to pre_string. Then, it is determined whether the current character current_symbol is 'struct'. When current_symbol is 'struct' or '<eos>', it is determined whether the current character current_symbol is 'struct'.

For judging whether the current character current_symbol is ‘struct’, when current_symbol is ‘struct’, predict the structure information and push the structure data into the stack, pop the top element of the stack, and adjust pre_string in combination with the top element of the stack and parent_symbol. When current_symbol is not ‘struct’, judge whether the stack is empty. When the stack is not empty, pop the top element of the stack, and adjust pre_string in combination with the top element of the stack and parent_symbol. When the stack is empty, judge whether the number of “{” and “}” in pre_string is equal. If the number of “{” and “}” in pre_string is equal, exit the loop and return the recognition result. If the number of “{” and “}” in pre_string is not equal, add a “}” to pre_string, that is, pre_string+＝“}”. At this time, if the number of “{” and “}” in pre_string is equal, exit the loop and return the recognition result.

After adjusting pre_string, it can be determined whether the number of loops exceeds 100. If so, the loop is exited and the recognition result is returned. Otherwise, the character is recognized and the current character current_symbol is updated, and the loop is restarted until the recognition result is directly returned.

In step 103, when it is determined that the character data meets the preset conditions, the training is stopped to obtain a trained formula recognition model.

In this step, in order to improve the accuracy of the formula recognition model, a loss function is set in this embodiment. The loss function is constructed based on the frequency of occurrence of each character in the label data of the training sample data and the frequency of occurrence of each character in the prediction data set. In one example, the loss function includes three sub-loss functions, which are: the first sub-loss function is a sub-loss function for comparing the counting feature map with the number of each character in the training sample data, and the sub-loss function is a SmoothL1Loss function; the second sub-loss function is used to judge the prediction result of the structural information, and the sub-loss function adopts the cross entropy function; the third sub-loss function is used to judge the prediction result of the character information, and the sub-loss function adopts the cross entropy function.

In this step, the electronic device can determine that the character data meets the preset conditions. For example, the electronic device can obtain the frequency of occurrence of each character data in the predicted character set to obtain the predicted frequency corresponding to each character data; then, the electronic device can obtain the difference between the predicted frequency corresponding to each character data and the marked frequency in the label of the training sample data to obtain the prediction error value corresponding to each character data; thereafter, the electronic device can obtain the average value of the prediction error value corresponding to the character data in the predicted character set to obtain the error average value; when it is determined that the error average value is less than or equal to the preset average value threshold (such as 1-3 times), the electronic device can determine that the character data meets the preset conditions and stop training the formula recognition model, that is, obtain a formula recognition model that has completed training. When it is determined that the above-mentioned error average value is greater than the preset average value threshold, the electronic device can determine that the character data does not meet the preset conditions, and can return to step 102 to continue training until the preset conditions are met.

In step 13, the position of the character data in the mathematical formula is restored according to the preset formula format and structure data to obtain the mathematical formula in the original image.

In this step, considering that the predicted character set identified in step 12 is embodied in the structure shown in Table 2 and Table 3, the electronic device can restore the position of the character data in the mathematical formula according to the preset formula format and structural data to obtain the mathematical formula in the original image.

In this step, the comparison of the recognition results of the mathematical formulas obtained using the above formula recognition model is shown in Table 4.

Table 4 Recognition accuracy before and after using the formula recognition model

In Table 4, the check mark "√" indicates that the function is used. As shown in Table 4, when only the counting module is used, the accuracy of the mathematical formula recognition result is improved by 45.49%; when only the structural information is used, the accuracy of the mathematical formula recognition result is improved by 50.25%; after using both the counting module and the structural information, the accuracy of the mathematical formula recognition result is improved by 51.95%.

At this point, the scheme provided by the embodiment of the present disclosure can obtain the original image containing the mathematical formula; then, the original image is input into the formula recognition model to obtain the predicted character set output by the formula recognition model; the predicted character set includes character data and structural data; thereafter, the position of the character data in the mathematical formula is restored according to the preset formula format and structural data to obtain the mathematical formula in the original image. In this way, by setting the structural data in this embodiment, the relative position relationship of the character data can be characterized, and the problem of missing characters in the complex formula recognition process can be eliminated, which is beneficial to improving the recognition accuracy. In other words, in this embodiment, the dual prediction method of structural information and character count is adopted to improve the model's recognition ability for complex formulas with multi-layer nested structures, reduce the formula misrecognition caused by the loss of "{" or "}" when multi-layer nesting, and split the complex formula into component recognition according to different structures, thereby improving the recognition accuracy; a multi-scale counting module is added in the decoding stage to construct a count based on the number of characters in the label. The loss function weakly supervises the training process, improves the detection rate of characters in each component, and improves the recognition accuracy.

On the basis of a mathematical formula recognition method provided in an embodiment of the present disclosure, this embodiment also provides a mathematical formula recognition device, see Figure 13, the device includes: an original image acquisition module 131, used to acquire an original image containing a mathematical formula; a prediction set acquisition module 132, used to input the original image into a formula recognition model to obtain a predicted character set output by the formula recognition model; the predicted character set includes character data and structural data; a mathematical formula acquisition module 133, used to restore the position of the character data in the mathematical formula according to a preset formula format and structural data, and obtain the mathematical formula in the original image.

It should be noted that the device embodiment shown in this embodiment matches the content of the above method embodiment, and the content of the above method embodiment can be referred to, which will not be repeated here.

In an exemplary embodiment, an electronic device is also provided, comprising: a display screen; a processor; and a memory for storing a computer program executable by the processor, wherein the processor is configured to execute the computer program in the memory to implement the above method.

In an exemplary embodiment, a computer-readable storage medium is also provided, such as a memory including an executable computer program, and the executable computer program can be executed by a processor to implement the method of the above embodiment. The readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the disclosure disclosed herein. The present disclosure is intended to cover any variations, uses, or adaptations that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The description and examples are to be considered exemplary only, and the true scope and spirit of the present disclosure are indicated by the following claims.

Claims (25)

A mathematical formula recognition method, characterized in that the method comprises: Get the original image containing the mathematical formula; Inputting the original image into a formula recognition model to obtain a predicted character set output by the formula recognition model; the predicted character set includes character data and structure data; The position of the character data in the mathematical formula is restored according to the preset formula format and structure data to obtain the mathematical formula in the original image. The method according to claim 1, characterized in that the formula recognition model includes an encoder and a decoder; The encoder is used to obtain an image feature map corresponding to the original image; The decoder is used to determine a predicted character set corresponding to the original image according to the image feature map. The method according to claim 2, characterized in that the encoder includes a DenseNet network or a Transformer network. The method according to claim 2, characterized in that the decoder comprises a multi-scale counting module, a cyclic decoding module, a feature fusion module and a character prediction module; The multi-scale counting module is used to convert the image feature map into counting vectors of multiple preset scales; The cyclic decoding module is used to obtain a character latent vector according to the image feature map; The feature fusion module is used to fuse the count vector, the character latent vector and the previous predicted character vector to obtain a target vector; The character prediction module is used to predict character data and structure data respectively according to the target vector. The method according to claim 4 is characterized in that the circular decoding module is also used to obtain a context vector based on the image feature map; the feature fusion module is also used to fuse the count vector, the context vector, the character latent vector and the previous predicted character vector to obtain a target vector. The method according to claim 5 is characterized in that the feature fusion module performs linear transformation processing on the count vector, the context vector, the character latent vector and the previous predicted character vector respectively to obtain a first linear vector, a second linear vector, a third linear vector and a fourth linear vector; and obtains the sum vector of the first linear vector, the second linear vector, the third linear vector and the fourth linear vector to obtain a target vector. The method according to claim 4 is characterized in that the multi-scale counting module includes at least two sub-counting modules and a vector averaging sub-module; the at least two sub-counting modules contain convolution kernels of different sizes and are used to output sub-vectors of the same size, and each sub-vector is used to represent the number of times character data is predicted at different scales; the vector averaging sub-module is used to obtain the average vector of at least two sub-vectors to obtain a counting vector. The method according to claim 7, characterized in that the multi-scale counting module comprises a first sub-counting module and a second sub-counting module; The first sub-counting module is used to identify feature information of a first preset scale in the image feature map to obtain a first sub-vector; the first sub-vector is used to represent the number of times character data is recognized under the convolution kernel of the first preset scale; The second sub-counting module is used to identify feature information of a second preset scale in the image feature map to obtain a second sub-vector; the second sub-vector is used to represent the number of times character data is recognized under a convolution kernel of a second preset scale. The method according to claim 8, characterized in that the first sub-counting module includes a first convolution unit, a channel unit, a conversion unit and a pooling unit; The first convolution unit is used to convert the image feature map into a first feature map using a convolution kernel of a first preset size; The channel unit is used to assign corresponding weights to different channels of the first feature map to obtain an attention feature map; and to multiply the attention feature map and the first feature map to obtain a channel feature map; The conversion unit is used to perform a scale transformation process on the channel feature map to obtain a count feature map; The pooling unit is used to perform pooling processing on the count feature map to obtain a first sub-vector. The method according to claim 8, characterized in that the second sub-counting module includes a second convolution unit, a channel unit, a conversion unit and a pooling unit; The second convolution unit is used to convert the image feature map into a second feature map using a convolution kernel of a second preset size; The channel unit is used to assign corresponding weights to different channels of the second feature map to obtain an attention feature map; and multiply the attention feature map and the second feature map to obtain a channel feature map; The conversion unit is used to perform a scale transformation process on the channel feature map to obtain a count feature map; The pooling unit is used to perform pooling processing on the count feature map to obtain a second sub-vector. The method according to claim 8 is characterized in that the multi-scale counting module also includes a third sub-counting module, and the third sub-counting module is used to identify feature information of a third preset scale in the image feature map to obtain a third sub-vector. The method according to claim 11, characterized in that the third sub-counting module includes a third convolution unit, a channel unit, a conversion unit and a pooling unit; The third convolution unit is used to convert the image feature map into a third feature map by using a convolution kernel of a third preset size; The channel unit is used to assign corresponding weights to different channels of the third feature map to obtain an attention feature map; and multiply the attention feature map and the third feature map to obtain a channel feature map; The conversion unit is used to perform a scale transformation process on the channel feature map to obtain a count feature map; The pooling unit is used to perform pooling processing on the count feature map to obtain a third sub-vector. The method according to claim 4, characterized in that the loop decoding module includes a gated loop unit; The gated recurrent unit is used to generate the character latent vector according to the latent vector and the previous predicted character vector. The method according to claim 13, characterized in that the loop decoding module includes a first gated loop unit and a second gated loop unit; The first gated recurrent unit is used to generate a first gated vector according to the latent vector and the previous predicted character vector; The second gated recurrent unit is configured to generate the character latent vector according to the first gating vector. The method according to claim 14, characterized in that the loop decoding module further includes an attention submodule; The attention submodule is used to generate a context vector according to the image feature map, the all-zero tensor and the first gating vector; The second gated recurrent unit is configured to generate the character latent vector according to the first gating vector and the context vector. The method according to claim 15, characterized in that the attention submodule includes a weight acquisition submodule and a context acquisition submodule; The weight acquisition submodule is used to perform convolution and linear processing on the all-zero tensor to obtain a first feature vector, and to perform convolution processing on the image feature map to obtain a second feature vector, and to perform convolution processing on the first gating vector Linear processing is performed to obtain a third eigenvector; and the first eigenvector, the second convolution vector and the third eigenvector are superimposed to obtain a fourth eigenvector; The context acquisition submodule is used to obtain an attention weight after performing a first activation process, a linear conversion process and a second activation process on the fourth eigenvector respectively; and to multiply the image feature map and the attention weight, and then perform a summation process to obtain a context vector. The method according to claim 4, characterized in that the character prediction module includes a character prediction submodule and a structure prediction submodule; The character prediction submodule is used to process the target vector to obtain predicted character data; The structure prediction submodule is used to process the target vector to obtain predicted structure data. The method according to claim 17, characterized in that the character prediction submodule includes a linear conversion unit and an activation unit; The linear transformation unit is used to perform linear transformation on the target vector; The activation unit is used to perform activation processing on the linearly transformed vector to obtain predicted character data. The method according to claim 17, characterized in that the structure prediction submodule includes a linear transformation unit and an activation unit; The linear transformation unit is used to perform linear transformation on the target vector; The activation unit is used to perform activation processing on the linearly transformed vector to obtain predicted structural data. The method according to claim 1, characterized in that the formula recognition model is trained by the following steps, including: Acquire training sample data; the training sample data includes a data set of multiple mathematical formulas; each data set includes character data and structure data obtained after expansion according to a preset formula format; the structure data is used to represent the relative position relationship of some characters in the mathematical formula; Using the training sample data to train the formula recognition model to be trained, to obtain a predicted character set output by the formula recognition model; the predicted character set includes character data and structure data; When it is determined that the character data meets the preset conditions, the training is stopped to obtain a trained formula recognition model. The method according to claim 20, characterized in that determining that the character data satisfies a preset condition comprises: Obtaining the frequency of occurrence of each character data in the predicted character set to obtain the predicted frequency corresponding to each character data; Obtaining the difference between the predicted frequency corresponding to each character data and the marked frequency in the label of the training sample data, and obtaining the prediction error value corresponding to each character data; Obtaining an average value of prediction error values corresponding to character data in the predicted character set to obtain an error average value; When it is determined that the error average value is less than or equal to the preset average value threshold, it is determined that the character data meets the preset condition. The method according to claim 20, characterized in that the training sample data is obtained through the following steps, including: Get multiple original images containing mathematical formulas; Obtain character data corresponding to each original image; Structural data is created according to a preset formula format and character data, and the character data is arranged to obtain training sample data corresponding to each original image. A mathematical formula recognition device, characterized in that the device comprises: An original image acquisition module is used to acquire the original image containing the mathematical formula; A prediction set acquisition module, used for inputting the original image into a formula recognition model to obtain a predicted character set output by the formula recognition model; the predicted character set includes character data and structure data; The mathematical formula acquisition module is used to restore the position of the character data in the mathematical formula according to the preset formula format and structure data to obtain the mathematical formula in the original image. An electronic device, comprising: processor; a memory for storing a computer program executable by the processor; The processor is configured to execute the computer program in the memory to implement the method according to any one of claims 1 to 22. A computer-readable storage medium, characterized in that when an executable computer program in the storage medium is executed by a processor, the method according to any one of claims 1 to 22 can be implemented.