JP2022019334A - Instructed value reading program, indicated value reading device and indicated value reading method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an instruction value reading program, an instruction value reading device and an instruction value reading method capable of quickly detecting an abnormality generated in a factory.
SOLUTION: A partial image data about the display is extracted from an inspection target image data including a display having a pointer, and the guideline included in the partial image data is based on the extracted partial image data. The value indicated by the pointer included in the partial image data based on the specified first orientation and the range of values that the pointer can indicate on the display. Is specified, and the value specified by the specified guideline is output.
[Selection diagram] FIG. 12

Description

The present invention relates to an indicated value reading program, an indicated value reading device, and an indicated value reading method.

For example, in a factory that manufactures semiconductors and the like, many analog meters for measuring the temperature, pressure, and the like of each device are generally arranged.

In such a factory, for example, the worker patrols the factory at regular timings and visually confirms the value indicated by each analog meter. Then, the worker performs necessary calculations using, for example, the values of each analog meter, and determines whether or not an abnormality (for example, a device failure) has occurred in the factory. As a result, when it is determined that an abnormality has occurred in the factory, the worker makes arrangements for replacing the equipment in which the abnormality has occurred (Patent Documents 1 and 2 and Non-Patent Documents 1 to 1). See 2).

Japanese Unexamined Patent Publication No. 2004-133560 Japanese Unexamined Patent Publication No. 2017-126187

Yixiao Fang, Yan Dai, Guoli He and Donglian Qi, "A Mask RCNN based Automatic Reading Method for Pointer Meter", 38th Chinese Control Conference, 8466-8471. HaoWen Lai, Qi Kang, Le pan and Can Cui, "A Novel Scale Recognition Method for Pointer Meters Adapted to Different Types and Shape", 2019 IEEE 15th International Conference on Automation Science and Engineering (CASE), 374-379.

Here, the calculation performed using the values of each analog meter as described above is, for example, manually performed by an operator, and may take a certain amount of time. Further, such a calculation may need to be performed after moving to a place other than the factory (worker's office, etc.), for example.

Therefore, the worker cannot quickly detect the abnormality occurring in the factory, and it may take time to eliminate the generated abnormality.

Therefore, an object of the present invention is to provide an instruction value reading program, an instruction value reading device, and an instruction value reading method that can quickly detect an abnormality that has occurred in a factory.

The indicated value reading program in the present invention for achieving the above object extracts partial image data about the display from inspection target image data including a display having a pointer, and is based on the extracted partial image data. The first orientation included in the partial image data and indicated by the pointer is specified, and the identified first orientation and the range of values that the pointer can indicate on the display are used. It is characterized in that a computer is made to execute a process of specifying a value specified by the guideline included in the partial image data and outputting the value specified by the specified guideline.

Further, the instruction value reading program in the present invention for achieving the above object is included in each learning image data for each of a plurality of learning image data including a display having a guideline in one embodiment. By adding the position information of the display, a plurality of training data are generated, and by performing machine learning using the generated plurality of training data, a learning model is generated and the inspection target image data is input. Along with this, a value output from the learning model is acquired, and the partial image data corresponding to the acquired value is extracted from the inspection target image data.

Further, the instruction value reading program in the present invention for achieving the above object is included in each learning image data for each of a plurality of learning image data for a display having a pointer in one embodiment. By adding information indicating the direction of the pointer, a plurality of learning data are generated, and by performing machine learning using the generated plurality of learning data, a learning model is generated and the partial image data is input. Along with this, a value output from the learning model is acquired, and the direction of the guideline is specified from the acquired value.

Further, the instruction value reading program in the present invention for achieving the above object calculates the length of the guideline from the value in one embodiment, and when the calculated length is within a predetermined range. It is characterized in that a process of specifying the direction of the pointer from the above value is performed.

Further, in one embodiment, the indicated value reading program in the present invention for achieving the above object, when the length is not within the predetermined range, before performing the process of specifying the direction of the pointer from the value. The process of extracting the partial image data and the process of acquiring the value are performed again.

Further, the instruction value reading program in the present invention for achieving the above object includes, in one embodiment, a minimum value included in the range of values instructable by the guideline and a range of values instructable by the guideline. The maximum value, the first orientation, the second orientation indicated by the pointer when the pointer indicates the minimum value, and the pointer when the pointer indicates the maximum value. It is characterized in that the value indicated by the guideline is specified from the third direction.

Further, in one embodiment, the indicated value reading program in the present invention for achieving the above object has a first aspect by subtracting the angle corresponding to the second orientation from the angle corresponding to the first orientation. The second value is calculated by calculating the value and subtracting the angle corresponding to the second direction from the angle corresponding to the third direction, and the second value is calculated by subtracting the minimum value from the maximum value. The value of 3 is calculated, the first value is divided by the second value to calculate the fourth value, and the fourth value is multiplied by the third value to obtain the fifth value. The value specified by the guideline is calculated by calculating the value of and adding the minimum value to the fifth value.

Further, in one embodiment, the instruction value reading program in the present invention for achieving the above object acquires the inspection target image data of the display imaged by the imaging device, and the acquired inspection target image data. The partial image data is extracted from the inspection target image data that has undergone homography conversion.

Further, in one embodiment, the indicated value reading program in the present invention for achieving the above object identifies one or more marks on the display and moves the specified one or more marks to a predetermined position. It is characterized in that the homography conversion is performed on the image data to be inspected so as to be performed.

Further, the instruction value reading program in the present invention for achieving the above object is characterized in that, in one embodiment, an average value of the values indicated by the guideline is calculated and the calculated average value is output. ..

Further, in one embodiment, the instruction value reading program in the present invention for achieving the above object calculates and calculates the standard deviation of the value specified by the guideline specified in the process of specifying the value specified by the guideline. When the standard deviation is equal to or greater than a predetermined threshold value, the average value is output.

Further, in one embodiment, the indicated value reading program in the present invention for achieving the above object calculates the standard deviation of the value specified by the guideline specified in the process of specifying the value specified by the guideline, and the standard deviation of the value specified by the guideline is calculated. It is characterized in that the standard deviation is output in addition to the average value.

Further, in one embodiment, the instruction value reading program in the present invention for achieving the above object calculates and calculates the standard deviation of the value specified by the guideline specified in the process of specifying the value specified by the guideline. When the standard deviation is equal to or greater than a predetermined threshold value, information indicating that the pointer is vibrating is output in addition to the average value.

Further, in one embodiment, the indicated value reading program in the present invention for achieving the above object has a maximum value and a minimum value among the values specified by the guideline specified in the process of specifying the value specified by the guideline. The difference is calculated from the above, and when the calculated difference is equal to or greater than a predetermined threshold value, the average value is output.

Further, in one embodiment, the indicated value reading program in the present invention for achieving the above object has a maximum value and a minimum value among the values specified by the guideline specified in the process of specifying the value specified by the guideline. When the calculated difference is equal to or greater than a predetermined threshold value, information indicating that the pointer is vibrating is output in addition to the average value.

Further, the indicated value reader in the present invention for achieving the above object has a partial image extraction unit for extracting partial image data about the display from inspection target image data including a display having a pointer, and an extraction unit. Based on the partial image data, the direction specifying unit that specifies the first direction specified by the pointer included in the partial image data, the specified first direction, and the pointer indicates on the display. It has a value specifying unit that specifies a value specified by the guideline included in the partial image data based on a range of possible values, and a value output unit that outputs a value specified by the specified guideline. It is characterized by that.

Further, in the method for reading the indicated value in the present invention for achieving the above object, the partial image data about the display is extracted from the inspection target image data including the display having a pointer, and the extracted partial image data is obtained. Based on the above, the first orientation indicated by the pointer included in the partial image data is specified, and the identified first orientation and the range of values that the pointer can indicate on the display are used. It is characterized in that a computer is made to execute a process of specifying a value specified by the guideline included in the partial image data and outputting the value specified by the specified guideline.

According to the indicated value reading program, the indicated value reading device, and the indicated value reading method in the present invention, it is possible to quickly detect an abnormality occurring in the factory.

FIG. 1 is a diagram showing a configuration example of the information processing apparatus 1 according to the first embodiment. FIG. 2 is a diagram illustrating an outline of an instruction value reading process according to the first embodiment. FIG. 3 is a diagram illustrating an outline of the indicated value reading process according to the first embodiment. FIG. 4 is a flowchart illustrating the details of the indicated value reading process according to the first embodiment. FIG. 5 is a flowchart illustrating the details of the indicated value reading process according to the first embodiment. FIG. 6 is a flowchart illustrating the details of the indicated value reading process according to the first embodiment. FIG. 7 is a flowchart illustrating the details of the indicated value reading process according to the first embodiment. FIG. 8 is a diagram illustrating details of the indicated value reading process according to the first embodiment. FIG. 9 is a diagram illustrating details of the indicated value reading process according to the first embodiment. FIG. 10 is a diagram illustrating details of the indicated value reading process according to the first embodiment. FIG. 11 is a diagram illustrating details of the indicated value reading process according to the first embodiment. FIG. 12 is a diagram illustrating details of the indicated value reading process according to the first embodiment. FIG. 13 is a diagram illustrating details of the indicated value reading process according to the first embodiment. FIG. 14 is a diagram illustrating details of the indicated value reading process according to the first embodiment. FIG. 15 is a diagram illustrating details of the indicated value reading process according to the first embodiment. FIG. 16 is a diagram illustrating details of the indicated value reading process according to the first embodiment.

Hereinafter, embodiments of the present invention will be described with reference to the drawings. However, such embodiments do not limit the technical scope of the invention.

First, a configuration example of the information processing device 1 (hereinafter, also referred to as an indicated value reading device 1) according to the first embodiment will be described. FIG. 1 is a diagram showing a configuration example of the information processing apparatus 1 according to the first embodiment.

The information processing device 1 is a computer device, for example, a general-purpose PC (Personal Computer). Then, the information processing apparatus 1 performs, for example, a process of reading a value indicated by a pointer of an analog meter (hereinafter, also referred to as a display) (hereinafter, also referred to as an indicated value reading process) (hereinafter, also referred to as an indicated value reading process).

The information processing apparatus 1 has a hardware configuration of a general-purpose computer apparatus, and has, for example, as shown in FIG. 1, a CPU 101 which is a processor, a memory 102, a communication interface 103, and a storage medium 104. The parts are connected to each other via the bus 105.

The storage medium 104 has, for example, a program storage area (not shown) for storing a program (not shown) for performing the indicated value reading process.

Further, the storage medium 104 has, for example, a storage unit 110 (hereinafter, also referred to as a storage area 110) for storing information used when performing an instruction value reading process. The storage medium 104 may be, for example, an HDD (Hard Disk Drive) or an SSD (Solid State Drive).

The CPU 101 executes a program loaded from the storage medium 104 into the memory 102 to perform an instruction value reading process.

The communication interface 103 communicates with an operation terminal 2 such as a smartphone via a network NW such as an Internet network, for example.

[Outline of the first embodiment]
2 and 3 are diagrams illustrating an outline of the indicated value reading process according to the first embodiment. Specifically, FIG. 2 is a diagram illustrating a process in the learning stage among the indicated value reading processes in the first embodiment. Further, FIG. 3 is a diagram illustrating a process in the inference stage among the indicated value reading processes in the first embodiment.

First, among the instruction value reading processes in the first embodiment, the process in the learning stage will be described.

As shown in FIG. 2, the first data generation unit 111 of the information processing apparatus 1 has, for example, first with respect to the learning image data (hereinafter, also referred to as the first learning image data) in which the analog meter is partially displayed. Learning data (hereinafter, also referred to as first learning data) is generated by adding information about the position and size of the analog meter in the learning image data (hereinafter, these are collectively referred to as position information).

Then, the first model generation unit 112 of the information processing apparatus 1 performs machine learning using a plurality of first learning data generated by the first data generation unit 111, thereby performing a learning model (hereinafter, also referred to as a first learning model). Call) is generated. After that, the first model generation unit 112 stores the generated first learning model in the storage area 110.

That is, the first model generation unit 112 generates the first training model used when extracting the partial image data of the analog meter from the inspection target image data by using the plurality of first training data.

Further, the second data generation unit 113 of the information processing apparatus 1 is generated by the first model generation unit 112 with respect to the learning image data (hereinafter, also referred to as the second learning image data) of the analog meter, for example. 1 Training data (hereinafter, also referred to as second training data) is generated by adding information indicating the direction of the pointer of the analog meter to the partial image data extracted by the training model.

Then, the second model generation unit 114 of the information processing apparatus 1 performs machine learning using the plurality of second learning data generated by the second data generation unit 113, thereby performing a learning model (hereinafter, also referred to as a second learning model). Call) is generated. After that, the second model generation unit 114 stores the generated second learning model in the storage area 110.

That is, the second model generation unit 114 generates a second learning model used when specifying (estimating) the direction indicated by the pointer of the analog meter by using the plurality of second learning data.

Next, among the instruction value reading processes in the first embodiment, the inference stage process will be described.

The partial image extraction unit 121 of the information processing apparatus 1 extracts the partial image data of the analog meter from the verification target image data including the analog meter in a part.

Specifically, the partial image extraction unit 121 specifies the position information indicated by the value output with the input of the verification target image data for the first learning model stored in the storage area 110. Then, the partial image extraction unit 121 extracts the partial image data corresponding to the specified position information from the verification target image data.

Then, the direction determination unit 122 of the information processing apparatus 1 is directed by the pointer of the analog meter reflected in the partial image data based on the partial image data extracted by the partial image extraction unit 121 (hereinafter, also the first direction). Call).

Specifically, the direction determination unit 122 specifies the first direction indicated by the value output with the input of the partial image data for the second learning model stored in the storage area 110.

Subsequently, the value specifying unit 123 of the information processing apparatus 1 is extracted by the partial image extraction unit 121 based on the first direction specified by the direction determination unit 122 and the range of values that can be indicated by the pointer of the analog meter. Specify the value indicated by the pointer of the analog meter included in the partial image data.

After that, the value output unit 124 of the information processing apparatus 1 outputs, for example, the value specified by the value specifying unit 123 to the operation terminal 2.

That is, the information processing apparatus 1 in the present embodiment automatically identifies the direction in which the pointer of the analog meter is facing by using the image data of the analog meter arranged in the factory as an input to the learning model. conduct. Then, the information processing apparatus 1 specifies the value indicated by the pointer of the analog meter based on the specified direction.

After that, the information processing apparatus 1 calculates (estimates) the probability of occurrence of an abnormal value in the equipment in the factory from, for example, the current value and the past value indicated by the pointers of each analog meter. Then, the information processing apparatus 1 determines, for example, that there is a high possibility that an abnormality has occurred in the factory when the probability of occurrence of the calculated abnormal value exceeds a predetermined threshold value.

As a result, the information processing apparatus 1 can automatically and quickly determine whether or not an abnormality is likely to occur in the factory. Then, when it is determined that there is a high possibility that an abnormality has occurred in the factory, for example, the information processing apparatus 1 can notify the operator that the necessary work (operation) is to be performed. Become.

In this respect, the occurrence of an abnormality in the factory may lead to the occurrence of a major accident in the factory. Therefore, the worker needs to take prompt measures against the abnormality generated in the factory from the viewpoint of preventing the occurrence of a major accident in the factory.

Therefore, for example, when the information processing apparatus 1 determines that there is a high possibility that an abnormality will occur in the factory, the information processing apparatus 1 instructs the operator to take appropriate measures. Further, when the information processing apparatus 1 determines, for example, that there is an extremely high possibility that an abnormality has occurred in the factory (when it is determined that an abnormality has occurred is in a situation where it can be determined), the operator Instruct to take emergency action.

Further, the information processing apparatus 1 automatically reads the value indicated by the pointer of each analog meter, so that an operator can make a human error (reading error, value input error, etc.) due to the reading. It becomes possible to prevent the occurrence.

[Details of the first embodiment]
Next, the details of the indicated value reading process in the first embodiment will be described. 4 to 7 are flowcharts illustrating the details of the indicated value reading process according to the first embodiment. 8 to 16 are diagrams illustrating details of the indicated value reading process according to the first embodiment.

[Processing in the learning stage]
First, among the details of the indicated value reading process in the first embodiment, the details of the process in the learning stage will be described. FIG. 4 is a diagram illustrating details of processing in the learning stage.

As shown in FIG. 4, the first data generation unit 111 waits until, for example, the learning timing is reached (NO in S11). The learning timing may be, for example, the timing at which the operator inputs the first learning image data via the operation terminal 2. Further, the learning timing may be, for example, the timing at which the worker inputs information to the effect that each learning model is generated.

Then, when the learning timing is reached (YES in S11), the first data generation unit 111 adds the position information of the analog meter included in each image data to each of the first learning image data, so that the first data is generated. 1 Generate training data (S12).

Specifically, as shown in FIG. 7, the first data generation unit 111 first uses, for example, a bounding box BB11 (a bounding box BB11 designated by an operator) indicating the position and size of an analog meter. The position information of the analog meter is added to the image data DT11 for learning.

More specifically, the operator prepares, for example, image data corresponding to each frame included in the moving image data taken by the image pickup device (not shown) as the first learning image data. Then, the worker adds the position information of the analog meter to each of the first learning image data by using the bounding box. After that, the first data generation unit 111, for example, by associating the first learning image data with the upper left coordinates (X coordinate and Y coordinate) of the bounding box and the vertical and horizontal lengths of the bounding box. 1 Generate training data.

After that, the first model generation unit 112 generates the first learning model by performing machine learning using the first learning data generated in the process of S12 (S13).

Subsequently, the second data generation unit 113 generates the second learning data by adding information indicating the direction of the guideline in each image data to each of the second learning image data (S14).

Specifically, as shown in FIG. 8, the second data generation unit 113 uses, for example, a bounding box BB12 (a bounding box BB12 designated by the operator) indicating the position of the pointer of the analog meter, so that the second data generation unit 113 can be second. The position information of the pointer is added to the learning image data DT12.

More specifically, the second data generation unit 113 uses, for example, the image data output in connection with the input of the first learning image data to the first learning model generated in the process of S13 as the second learning image data. get. Then, the worker adds the position information of the pointer to each of the second learning image data by using the bounding box. After that, the second data generation unit 113 associates the second learning image data with the upper left coordinates (X coordinate and Y coordinate) of the bounding box and the vertical and horizontal lengths of the bounding box. 2 Generate training data.

Here, the second data generation unit 113 needs to generate a second learning model with high estimation accuracy. Therefore, for example, for each scale of the analog meter, an analog meter in a state where each scale is indicated by a pointer is displayed. 2 It is necessary to generate image data for learning in units of hundreds to thousands. Further, for example, when a plurality of analog meters are arranged in the factory, the second data generation unit 113 needs to generate, for example, the second learning image data for each analog meter.

In this regard, when the operator performs the indicated value reading process in the present embodiment, for example, the area of 1/4 of the scale of the analog meter (for example, "0.1" to "0" in the analog meter shown in FIG. 8). The image data for the second learning may be generated by preparing only the image data for the first learning corresponding to the area up to ".3" and inputting them into the first learning model. Then, the worker generates the second learning data from the generated second learning image data, and then flips the generated second learning image data left / right or upside down (left / right inversion and upside down). (For example, a region other than the region between “0.1” and “0.3” in the analog meter shown in FIG. 8), new second training data corresponding to the region of (for example, the region other than the region between “0.1” and “0.3”) is further generated, and the scale of the analog meter is scaled. The second learning data corresponding to all the regions of the above may be generated.

Specifically, for example, when the value pointed to by the pointer of the analog meter reflected in the second learning image data DT12 shown in FIG. 8 is "0.178", the image data obtained by reversing the second learning image data DT12 left and right. The value indicated by the pointer of the analog meter reflected in is "0.422", and the value indicated by the pointer of the analog meter reflected in the image data obtained by flipping the second learning image data DT12 upside down is "0.022". The value indicated by the pointer of the analog meter reflected in the image data obtained by reversing the second learning image data DT12 horizontally and vertically is "0.578".

Therefore, in this case, the operator adds new second learning data to which the position information of the pointer when pointing to "0.422" is added to the image data obtained by reversing the second learning image data DT12 left and right. The new second learning data to which the position information of the pointer when pointing to "0.022" is added to the image data obtained by inverting the image data DT12 for the second learning, and the image data DT12 for the second learning are left and right. With respect to the inverted and inverted image data, new second learning data to which the position information of the pointer when pointing to "0.578" is added is generated.

That is, the second learning model in the present embodiment is not a learning model that recognizes the verification target image data on which the analog meter is displayed, but a learning model that detects the direction (angle) indicated by the pointer of the analog meter. Therefore, when generating the second learning model, the operator can also use, for example, image data in which the character indicating the scale (“0.1” or the like) is inverted. Therefore, for example, the operator generates only the second learning data corresponding to a part of the scale area of the analog meter, and further inverts the second learning image data included in the generated second learning data. It may generate new second learning data by.

As a result, the worker can reduce the load required for generating the second learning data. Further, in this case, the operator can generate a second learning model that can specify (estimate) the direction of the pointer in other analog meters having different types of pointers, the number of scales, and the like.

After that, the second model generation unit 114 generates a second learning model by performing machine learning using the second learning data generated in the process of S14 (S15).

The worker may input new first learning image data to the information processing apparatus 1 at any time. Then, the information processing apparatus 1 may generate new first learning data including new first learning image data at any time (S12).

After that, for example, when the ratio of the added new image data for the first learning exceeds a predetermined ratio of all the image data for the first learning, the information processing apparatus 1 uses the first learning model and the second learning model. May be regenerated (S13 to S15).

This enables the worker to continuously improve the determination accuracy of the first learning model and the second learning model.

[Processing in inference stage]
Next, among the details of the indicated value reading process in the first embodiment, the details of the process in the inference stage will be described. 5 and 6 are diagrams illustrating details of processing in the inference stage.

As shown in FIG. 5, the partial image extraction unit 121 waits until the determination timing is reached (NO in S21). The determination timing may be, for example, the timing at which the operator inputs the verification target image data via the operation terminal 2. Further, the determination timing may be, for example, the timing at which the worker inputs information to the effect that the verification target learning data is determined.

Then, when the determination timing is reached (YES in S21), the partial image extraction unit 121 acquires the inspection target image data of the analog meter imaged by the image pickup device (not shown) (S22).

Specifically, as shown in FIG. 9, the partial image extraction unit 121 receives, for example, the inspection target image data DT21 of the analog meter transmitted from the image pickup apparatus on which the operator takes a picture. The image pickup device may be, for example, a non-fixed camera. Further, the image pickup device may be, for example, one built in the operation terminal 2.

Subsequently, the partial image extraction unit 121 acquires the value output from the first learning model as the inspection target image data acquired in the process of S22 is input (S23).

Then, the partial image extraction unit 121 extracts the partial image data corresponding to the value acquired in the process of S23 from the inspection target image data acquired in the process of S22 (S24).

Specifically, as shown in FIG. 10, the partial image extraction unit 121 refers to, for example, the position information (coordinates) corresponding to the value acquired in the process of S23, and the inspection target image data acquired in the process of S21. Of these, the portion where the analog meter is displayed is acquired as the partial image data DT22.

Next, as shown in FIG. 6, the direction determination unit 122 acquires the value output from the second learning model as the partial image data extracted in the process of S24 is input (S31).

In this case, the direction determination unit 112 uses the value obtained by the process of S31 for the length from the rotation axis of the pointer to the tip of the analog meter (the length of the diagonal line of the bounding box corresponding to the position of the pointer). It may be calculated by the above. Then, when the calculated diagonal length is not within a predetermined allowable range (for example, within ± 5 (%)), the direction determination unit 112 causes an error (partial image data) in the processing of S24. It may be determined that a reading error) has occurred, and the processing after S24 may be performed again.

As a result, the direction determination unit 112 can improve the reading accuracy of the partial image data in the processing of S24.

Then, the direction determination unit 122 identifies the first direction in which the pointer in the analog meter is directed from the value acquired in the process of S31 (S32).

Specifically, as shown in FIG. 11, the direction determination unit 122 specifies the direction DRp as the first direction in which the pointer of the analog meter reflected in the partial image data DT22 extracted in the process of S24 is facing, for example.

Subsequently, the value specifying unit 123 is instructed by the pointer included in the partial image data extracted in the processing of S24 based on the first orientation specified in the processing of S32 and the range of values indicated by the pointer in the analog meter. Specify the value (S33).

Specifically, the value specifying unit 123 includes, for example, the minimum value in the range of values that the pointer can specify (hereinafter, also simply referred to as the minimum value) and the maximum value in the range of values that the pointer can specify (hereinafter, also simply referred to as the minimum value). Hereinafter, the pointer is simply referred to as the maximum value), the first direction specified in the processing of S32, the direction indicated by the pointer when the pointer indicates the minimum value (hereinafter, also referred to as the second direction), and the pointer. The value specified by the guideline included in the partial image data extracted in the process of S24 is specified from the direction specified by the guideline when the maximum value is specified (hereinafter, also referred to as a third direction). Hereinafter, a specific example of the processing of S33 will be described.

[Specific example of processing of S33]
The value specifying unit 123 specifies, for example, the value specified by the guideline included in the partial image data extracted in the process of S24 according to the following formula 1. Hereinafter, as shown in FIG. 12, the second direction is also referred to as a directional DRm, the third direction is also referred to as a directional DRM, and the reference direction (X-axis direction) when calculating each angle is the directional DRX. Also called.

In the above equation 1, θ _p indicates the angle when traveling clockwise from the direction DRX to the direction DRp, θ _m indicates the angle when traveling clockwise from the direction DRX to the direction DR m, and θ _M. Indicates the angle when traveling clockwise from the direction DRX to the direction DRM. Further, in the above equation 1, v _p indicates a value corresponding to the direction DR p in the range of values that can be specified by the pointer, and v _m is the minimum value (direction DR m) among the values that can be specified by the pointer. (Value corresponding to), and v _M indicates the maximum value (value corresponding to the direction DRM) among the values that can be specified by the pointer. When θ _p and θ _M are smaller than θ _m , the value specifying unit 123 sets θ _M as the angle obtained by adding 360 degrees to the angle from the direction DRX to the direction DRM.

Specifically, as shown in FIG. 12, for example, when θ _m is 135 degrees, θ _M is 405 degrees, and θ _p is 215 degrees, the value specifying unit 123 is about v _p . Calculate 0.178.

Returning to FIG. 6, the value output unit 124 outputs the value specified by the pointer specified in the process of S33 (S34).

Specifically, the value output unit 124 outputs, for example, the value specified by the pointer specified in the process of S33 to the operation terminal 2.

The value output unit 124 may, for example, perform a predetermined calculation using the value indicated by the guideline specified in the process of S33, and output the calculation result to the operation terminal 2. That is, the value output unit 124 performs a calculation for determining whether or not an abnormality has occurred in the factory by using, for example, the value indicated by the guideline specified in the process of S33, and operates the calculation result. It may be output to the terminal 2.

As described above, the information processing apparatus 1 in the present embodiment extracts partial image data about the analog meter from the image data to be inspected on which the analog meter having the pointer is displayed. Then, the information processing apparatus 1 specifies the first direction indicated by the guideline included in the partial image data based on the extracted partial image data.

Further, the information processing apparatus 1 specifies a value specified by the pointer included in the partial image data based on the specified first orientation and a range of values that the pointer can indicate in the analog meter. After that, the information processing apparatus 1 outputs a value instructed by the specified pointer.

That is, the information processing apparatus 1 in the present embodiment automatically identifies (reads) the value indicated by each analog meter by inputting the image data of each analog meter in the factory.

As a result, the information processing apparatus 1 can calculate (estimate) the probability that an abnormality has occurred in the factory by using, for example, the value indicated by each analog meter. Then, for example, when the information processing apparatus 1 determines that an abnormality has occurred in the factory based on the calculated probability, the information processing apparatus 1 may notify the worker that the necessary work (operation) is to be performed. It will be possible.

In addition, the information processing device 1 automatically reads the value indicated by the pointer of each analog meter, so that an operator can make a human error (reading error, value input error, etc.) due to this reading. It becomes possible to prevent the occurrence.

Here, the information processing device 1 may acquire moving image data about an analog meter imaged by an image pickup device (not shown) in the process of S22. Then, the information processing apparatus 1 processes S23 to S34 for each of a plurality of inspection target image data (for example, a plurality of inspection target image data corresponding to 10 consecutive frames) included in the moving image data. May be.

Specifically, in this case, the information processing apparatus 1 calculates, for example, the standard deviation of the value calculated in the process of S33 (the value calculated for each of the plurality of inspection target image data). As a result, when it is determined that the standard deviation exceeds the threshold value, the information processing apparatus 1 determines that the pointer of the analog meter reflected in the moving image data acquired in the processing of S22 is vibrating, and calculates it in the processing of S33. Outputs the average value of the values.

This makes it possible for the information processing apparatus 1 to determine whether or not the pointer of the analog meter is vibrating. Further, the information processing apparatus 1 can accurately output the value indicated by the pointer even when the pointer of the analog meter is vibrating.

In this case, in this case, the information processing apparatus 1 has, for example, in addition to the average value of the values calculated in the processing of S33, the standard deviation of the values calculated in the processing of S33, and the analog reflected in the moving image data acquired in the processing of S22. It may output information indicating that the pointer of the meter is vibrating. Further, in the information processing apparatus 1, instead of determining whether or not the standard deviation of the value calculated in the process of S33 exceeds the threshold value, the maximum value and the minimum value of the values calculated in the process of S33 are set. It may be used to determine whether or not the difference exceeds the threshold value.

Further, the information processing apparatus 1 may output the average value of the values calculated in the process of S33 even when it is determined that the standard deviation does not exceed the threshold value. As a result, the information processing apparatus 1 can improve the estimation accuracy of the value pointed to by the pointer of the analog meter.

[Homography transformation]
Next, the homography conversion of the partial image data will be described.

For example, the information processing apparatus 1 may perform homography transformation on the partial image data extracted by the processing of S24 before the processing of S31. Specifically, the information processing apparatus 1 performs homography conversion so that one or more marks on the analog meter reflected in the partial image data extracted in the process of S24 move to a position designated in advance. It's okay. Then, the information processing apparatus 1 may input the homographized partial image data to the second learning model in the processing of S31.

As a result, the information processing device can convert, for example, the detection target image data in which the tilted analog meter is displayed into the detection target image data in which the analog meter in the correct posture (the same posture as when taken from the front) is displayed. It will be possible. Therefore, the information processing apparatus 1 can suppress a decrease in the estimation accuracy of the value indicated by the pointer even when the detection target image data in which the tilted analog meter is displayed is acquired. Hereinafter, a specific example of the homography transformation will be described.

[Specific example of homography transformation]
13 and 14 are diagrams illustrating a specific example of the homography transformation.

For example, as shown in FIG. 13, the operator has four marks MR31 and MR32, which are color stickers not used in the analog meter, on the outer frame FL of the analog meter to be imaged by the image pickup device (not shown). , MR33 and MR34 are pasted.

Specifically, as shown in FIG. 13, for example, the operator makes the straight line connecting the mark MR31 and the mark MR33 (hereinafter, also referred to as a first straight line) perpendicular to the ground and the first thereof. Marks MR31 and MR33 are attached to the analog meter to be inspected so that the straight line of the pointer passes through the base end of the pointer (rotation axis of the pointer). Further, the operator, for example, makes sure that the straight line connecting the mark MR32 and the mark MR34 (hereinafter, also referred to as a second straight line) is horizontal to the ground, and the second straight line serves as the base end of the pointer. Mark MR32 and MR34 are attached to the analog meter to be inspected so as to pass through.

Then, as shown in FIG. 14 (A), for example, in the process of S24, when the partial image data DT31 in which the analog meter tilted to the left is reflected, the information processing apparatus 1 is as shown in FIG. 14 (B). For example, homography conversion is performed so that the first straight line is parallel to the horizontal side of the partial image data and the second straight line is parallel to the vertical side of the partial image data. Image data DT31a is generated. After that, the information processing apparatus 1 inputs the partial image data DT31a into the second learning model in the processing of S31.

Similarly, as shown in FIG. 14 (C), for example, in the process of S24, when the partial image data DT32 in which the analog meter tilted to the right is reflected, the information processing apparatus 1 is shown in FIG. 14 (D). So, for example, the homography transformation is performed so that the first straight line is parallel to the horizontal side of the partial image data and the second straight line is parallel to the vertical side of the partial image data. Partial image data DT32a is generated. After that, the information processing apparatus 1 inputs the partial image data DT32a into the second learning model in the processing of S31.

[Estimation result of the second learning model when homography transformation is performed]
Next, the estimation result of the second learning model when the homography transformation is performed will be described. 15 and 16 are diagrams illustrating the estimation results of the second learning model when the homography transformation is performed. Specifically, FIG. 15 (A) is a scatter diagram showing the estimation result of the second learning model when the homography transformation is not performed, and FIG. 15 (B) is a scatter diagram when the homography transformation is not performed. It is a histogram which shows the estimation result of the 2nd learning model. Further, FIG. 16A is a scatter diagram showing the estimation result of the second learning model when the homography transformation is performed, and FIG. 16B is a second learning model when the homography transformation is performed. It is a histogram showing the estimation result of.

The horizontal axis of the scatter plot shown in FIGS. 15 (A) and 16 (A) indicates the value actually pointed by the analog meter, and the vertical axis indicates the value specified by the processing of S33 and the analog meter. It shows the error (error) from the value actually pointed. Further, the horizontal axis of the histogram shown in FIGS. 15B and 16B shows an error between the value specified by the processing of S33 and the value actually pointed by the analog meter, and the vertical axis shows the error. Indicates the number of partial image data input to the second training model.

Specifically, each figure shown in FIG. 16 had an error of 0 between the value specified in the process of S33 and the value actually pointed by the analog meter when compared with each figure shown in FIG. It shows that the number of partial image data is increasing. Further, each figure shown in FIG. 16 shows that the error corresponding to each of the partial image data is smaller as a whole when compared with each figure shown in FIG.

That is, the examples shown in FIGS. 15 and 16 show that the estimation accuracy of the second learning model is improved by performing the homography transformation on the partial image data extracted by the process of S24.

1: Information processing device 2: Operation terminal 101: CPU
102: Memory 103: Communication interface 104: Storage medium 105: Bus

Claims (17)

Partial image data about the display is extracted from the image data to be inspected including the display having the pointer.
Based on the extracted partial image data, the first orientation indicated by the guideline included in the partial image data is specified.
Based on the specified first orientation and the range of values that the pointer can indicate on the display, the value specified by the pointer included in the partial image data is specified.
Outputs the value specified by the specified pointer,
An instruction value reading program characterized by having a computer perform processing. In claim 1, further
A plurality of learning data are generated by adding the position information of the display included in each learning image data to each of the plurality of learning image data including the display having the pointer.
A learning model is generated by performing machine learning using the generated plurality of training data.
Let the computer do the processing,
In the process of extracting the partial image data,
The value output from the learning model is acquired in connection with the input of the inspection target image data, and the value is acquired.
The partial image data corresponding to the acquired value is extracted from the inspection target image data.
An instruction value reading program characterized by this. In claim 1, further
A plurality of learning data are generated by adding information indicating the direction of the guideline included in each learning image data to each of the plurality of learning image data of the display having the pointer.
A learning model is generated by performing machine learning using the generated plurality of training data.
Let the computer do the processing,
In the process of specifying the first orientation,
The value output from the learning model is acquired with the input of the partial image data, and the value is acquired.
The direction of the pointer is specified from the acquired value.
An instruction value reading program characterized by this. In claim 3,
In the process of specifying the first orientation,
Calculate the length of the pointer from the value,
When the calculated length is within a predetermined range, a process of specifying the direction of the pointer from the value is performed.
An instruction value reading program characterized by this. In claim 4,
In the process of specifying the first orientation,
When the length is not within the predetermined range, the process of extracting the partial image data and the process of acquiring the value are performed again before the process of specifying the direction of the pointer from the value is performed.
An instruction value reading program characterized by this. In claim 1,
In the process of specifying the value specified by the guideline, the minimum value included in the range of values that can be specified by the guideline, the maximum value included in the range of values that can be specified by the guideline, and the first orientation. , The pointer indicates from the second direction indicated by the pointer when the pointer indicates the minimum value, and the third direction indicated by the pointer when the pointer indicates the maximum value. Specify the value to be
An instruction value reading program characterized by this. In claim 6,
In the process of specifying the value specified by the guideline,
The first value is calculated by subtracting the angle corresponding to the second orientation from the angle corresponding to the first orientation.
The second value is calculated by subtracting the angle corresponding to the second orientation from the angle corresponding to the third orientation.
A third value is calculated by subtracting the minimum value from the maximum value.
The fourth value is calculated by dividing the first value by the second value.
The fifth value is calculated by multiplying the fourth value by the third value.
The value indicated by the guideline is calculated by adding the minimum value to the fifth value.
An instruction value reading program characterized by this. In claim 1, further
The inspection target image data of the display imaged by the image pickup device is acquired, and the inspection target image data is acquired.
Homography conversion is performed on the acquired image data to be inspected.
Let the computer do the processing,
In the process of extracting the partial image data, the partial image data is extracted from the inspection target image data that has undergone homography conversion.
An instruction value reading program characterized by this. In claim 8,
In the process of performing the homography conversion,
Identify one or more landmarks on the display and
Homography transformation is performed on the inspection target image data so that the specified one or more marks move to a position specified in advance.
An instruction value reading program characterized by this. In claim 1, further
The process of extracting the partial image data, the process of specifying the first orientation, and the process of specifying the value specified by the guideline are performed on the plurality of the image data to be inspected.
Calculate the average value of the values specified by the guideline specified in the process of specifying the value specified by the guideline.
Let the computer do the processing,
In the process of outputting the value specified by the guideline, the calculated average value is output.
An instruction value reading program characterized by this. In claim 10, further
Calculate the standard deviation of the value specified by the guideline specified in the process of specifying the value specified by the guideline.
Let the computer do the processing,
In the process of outputting the specified value, the guideline outputs the average value when the calculated standard deviation is equal to or greater than a predetermined threshold value.
An instruction value reading program characterized by this. In claim 10, further
Calculate the standard deviation of the value specified by the guideline specified in the process of specifying the value specified by the guideline.
Let the computer do the processing,
In the process of outputting the specified value, the guideline outputs the standard deviation in addition to the average value.
An instruction value reading program characterized by this. In claim 10, further
Calculate the standard deviation of the value specified by the guideline specified in the process of specifying the value specified by the guideline.
Let the computer do the processing,
In the process of outputting the specified value, the above guideline
When the calculated standard deviation is equal to or greater than a predetermined threshold value, information indicating that the pointer is vibrating is output in addition to the average value.
An instruction value reading program characterized by this. In claim 10, further
The difference between the maximum value and the minimum value among the values specified by the guideline specified in the process of specifying the value specified by the guideline is calculated.
Let the computer do the processing,
In the process of outputting the specified value, the above guideline
When the calculated difference is equal to or greater than a predetermined threshold value, the average value is output.
An instruction value reading program characterized by this. In claim 10, further
The difference between the maximum value and the minimum value among the values specified by the guideline specified in the process of specifying the value specified by the guideline is calculated.
Let the computer do the processing,
In the process of outputting the specified value, the above guideline
When the calculated difference is equal to or greater than a predetermined threshold value, information indicating that the pointer is vibrating is output in addition to the average value.
An instruction value reading program characterized by this. A partial image extraction unit that extracts partial image data about the display from inspection target image data that includes a display with a pointer, and a partial image extraction unit.
Based on the extracted partial image data, a direction specifying portion that specifies a first direction indicated by the pointer included in the partial image data, and
A value specifying unit that specifies a value specified by the pointer included in the partial image data based on the specified first orientation and a range of values that the pointer can specify on the display.
It has a value output unit that outputs a value specified by the specified pointer.
A reading device for reading, characterized in that. Partial image data about the display is extracted from the image data to be inspected including the display having the pointer.
Based on the extracted partial image data, the first orientation indicated by the guideline included in the partial image data is specified.
Based on the specified first orientation and the range of values that the pointer can indicate on the display, the value specified by the pointer included in the partial image data is specified.
Outputs the value specified by the specified pointer,
A method of reading an indicated value, which comprises causing a computer to execute a process.

Images