[go: up one dir, main page]

CN113221881A - Multi-level smart phone screen defect detection method - Google Patents

Multi-level smart phone screen defect detection method Download PDF

Info

Publication number
CN113221881A
CN113221881A CN202110489131.0A CN202110489131A CN113221881A CN 113221881 A CN113221881 A CN 113221881A CN 202110489131 A CN202110489131 A CN 202110489131A CN 113221881 A CN113221881 A CN 113221881A
Authority
CN
China
Prior art keywords
image
defect
area
foreground
feature
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
CN202110489131.0A
Other languages
Chinese (zh)
Other versions
CN113221881B (en
Inventor
陈垣毅
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Hangzhou Zhenwei Food Collection Technology Co ltd
Original Assignee
Hangzhou City University
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Hangzhou City University filed Critical Hangzhou City University
Priority to CN202110489131.0A priority Critical patent/CN113221881B/en
Publication of CN113221881A publication Critical patent/CN113221881A/en
Application granted granted Critical
Publication of CN113221881B publication Critical patent/CN113221881B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/10—Segmentation; Edge detection
    • G06T7/194—Segmentation; Edge detection involving foreground-background segmentation
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
    • G06F18/214—Generating training patterns; Bootstrap methods, e.g. bagging or boosting
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/045—Combinations of networks
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T3/00—Geometric image transformations in the plane of the image
    • G06T3/40—Scaling of whole images or parts thereof, e.g. expanding or contracting
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T5/00—Image enhancement or restoration
    • G06T5/20—Image enhancement or restoration using local operators
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T5/00—Image enhancement or restoration
    • G06T5/50—Image enhancement or restoration using two or more images, e.g. averaging or subtraction
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T5/00—Image enhancement or restoration
    • G06T5/70—Denoising; Smoothing
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T5/00—Image enhancement or restoration
    • G06T5/73—Deblurring; Sharpening
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T5/00—Image enhancement or restoration
    • G06T5/90—Dynamic range modification of images or parts thereof
    • G06T5/94—Dynamic range modification of images or parts thereof based on local image properties, e.g. for local contrast enhancement
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/0002—Inspection of images, e.g. flaw detection
    • G06T7/0004—Industrial image inspection
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/20—Image preprocessing
    • G06V10/25—Determination of region of interest [ROI] or a volume of interest [VOI]
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20212—Image combination
    • G06T2207/20221—Image fusion; Image merging

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Data Mining & Analysis (AREA)
  • General Engineering & Computer Science (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Evolutionary Computation (AREA)
  • Artificial Intelligence (AREA)
  • Biophysics (AREA)
  • General Health & Medical Sciences (AREA)
  • Bioinformatics & Cheminformatics (AREA)
  • Health & Medical Sciences (AREA)
  • Biomedical Technology (AREA)
  • Bioinformatics & Computational Biology (AREA)
  • Computational Linguistics (AREA)
  • Evolutionary Biology (AREA)
  • Molecular Biology (AREA)
  • Computing Systems (AREA)
  • Mathematical Physics (AREA)
  • Software Systems (AREA)
  • Multimedia (AREA)
  • Quality & Reliability (AREA)
  • Image Analysis (AREA)
  • Image Processing (AREA)

Abstract

The invention relates to a multi-level smart phone screen defect detection method, which comprises the following steps: manually marking defects in the image, and then separating the foreground defect image from the background image; performing multi-scale defect region feature extraction on the data set by using a depth residual error network and a feature pyramid network to obtain a multi-scale feature image; and extracting a target interest detection area. The invention has the beneficial effects that: through the extraction of the interest detection area, the efficiency of image preprocessing is effectively improved, the problem that the existing general target detection technology is greatly interfered by background factors in a small target detection task of mobile phone screen defect detection is solved, a targeted enhancement mode is provided for the problem of data set shortage in actual floor application, the early-stage manpower marking investment is greatly reduced, and the efficiency of the whole scheme is further improved.

Description

Multi-level smart phone screen defect detection method
Technical Field
The invention belongs to the field of mobile phone screen defect detection, and particularly relates to a multi-level smart phone screen defect detection method.
Background
The screen is used as a core component of the smart phone and is a key point of human-computer interaction, and the quality of the screen seriously affects the experience of a user on the smart phone. Therefore, the production process of the mobile phone screen by each large mobile phone manufacturer is more and more demanding. However, the mobile phone screen is very susceptible to the production environment and the production process during the production process. In order to prevent the mobile phone with the defective screen from flowing into the market to damage the benefits of consumers and influence the credit of mobile phone screen manufacturers, the mobile phone screen manufacturers adopt some necessary means to detect the quality of the mobile phone screen. The traditional detection means is that workers are arranged on a production line for watching, and the workers detect screens on the production line one by one with naked eyes. However, the method has the defects of low detection efficiency, high labor cost, lack of uniform judgment standard and the like. In addition, some mobile phone screen defect detection technologies for calculating the traditional computer vision technology exist, but most of the methods only carry out algorithm design aiming at one or more specific defect types, once a new defect is met, a new algorithm needs to be designed in a targeted mode, and the universality is poor. Specifically, in the task of appearance defect of the screen of the smart phone, the following main difficulties exist:
1) defects have a strong correlation with background. Whether the defects exist in the region of interest or not needs to be comprehensively judged by combining background information of the position of the region of interest, and under different backgrounds, whether the defects exist or not is uncertain. For example, the different color defects are determined under the condition that the foreground defects and the background image have significant differences in hue, lightness, and the like.
2) Background information is extremely disturbing. For an irrelevant background region between the two-dimensional rectangular image and the irregular target detection region, there is a high possibility that there is a region similar to the defect.
3) Defects are of a wide variety and define ambiguities. Due to the differences of different production environments, production standards, production technologies and the like, different types of defect types exist in mobile phone screens in different production scenes, a unified classification standard is lacked at present, and the problem that the defect types are different from person to person in calibration exists.
4) The calibration data sets are of varying quality. Unlike detection data sets such as VOC, COCO and the like widely accepted and used by the academic community, the data sets applied in the field of the industry need to be specifically and specifically customized collected according to different requirements of task scenes. Based on the factors that subjectively calibrated personnel lack of cognition deficiency on an artificial intelligence technology and objectively defect sample distinguishing degree is not high, the industrial defect data set of the screen defect of the smart phone has two main problems: firstly, the sample data quantity under different types has large difference and does not meet the independent same distribution condition; the second is that the number of whole samples is not sufficient.
The former three problems are specific difficult problems in the task of detecting the appearance defects of the mobile phone screen, the reason is that the definition of the defects is different according to scenes, and the 4 th problem is a common difficulty in landing application of various deep learning algorithms at present.
Disclosure of Invention
The invention aims to overcome the defects in the prior art and provides a multi-level smart phone screen defect detection method.
The method for detecting the defects of the multi-level smart phone screen comprises the following steps:
step 1, manually marking defects in an image, and then separating a foreground defect image from a background image; on the basis of the labeled foreground defect image, the color and the size of the foreground defect image are enhanced, and the sample number and the form diversity of the foreground defect are expanded; finally, combining the enhanced foreground defect image and the enhanced background image to generate a data set which is reliable in annotation, balanced in category and suitable for an actual production scene;
step 1.1, manually marking the position and the category information of a rectangular area where each foreground defect image in a small number of images is located, and then separating the foreground defect image from the background image;
step 1.2, performing image denoising treatment on the foreground defect image: processing image noise by median filtering, setting the gray value of the target point as the median of all points in a region around the target point, and enabling the pixel value in the field of the target point to be closer to the true value so as to eliminate independent noise signals;
after the image filtering process is carried out and the noise is reduced, the definition of the image contour and the edge can be reduced, the image after the noise is reduced is sharpened by adopting a Laplacian operator, and the difference between an object in the image and a background gray value is enhanced:
Figure BDA0003050158810000021
Figure BDA0003050158810000022
Figure BDA0003050158810000023
Figure BDA0003050158810000024
in the above formula, f (x, y) is an image after noise reduction;
Figure BDA0003050158810000025
the image is subjected to Laplace transform; c is an enhancement coefficient and is generally 1; g (x, y) is the enhanced image;
randomly adjusting the chromaticity, the sharpness and the brightness of the foreground defect image, and randomly rotating the foreground defect image by a random angle and randomly zooming in a fixed proportion range;
1.3, four defects such as black spots, dirt, scratches, hairs and the like are obvious in characteristics, and for heterochromatic defects, the defects are characterized by color difference with a region adjacent to a background, so that the enhanced defect foreground is difficult to be universally matched with a background image; in the randomly selected background image area, adjusting the pixel value:
Figure BDA0003050158810000026
Figure BDA0003050158810000031
in the above formula, valuenewIs the enhanced pixel value; valueoldThe value range is 0-255 for the original pixel value; brightness represents the brightness ratio and takes a value of 0.9 or 1.1, the brightness value represents 0.9 dark heterochrosis, and the brightness value 1.1 represents bright heterochrosis; the value range of exp is 1.5-3, and the method is suitable for different color defects of different forms; a and b represent half the width and height of the rectangular background image area, respectively; the geometric meaning of the factor represents the ratio of the distance from a certain point to the defect center to the distance from the defect area boundary to the defect center, the value is less than 1 and is 0.95, the defect area contains a part of background information, and the generalization is improved; centerxAnd centeryThe horizontal and vertical coordinates of the center of the rectangular background image region are expressed by (center)x，centery) Establishing a rectangular coordinate system for an origin, wherein x and y respectively represent horizontal and vertical coordinates of points in a rectangular background image area;
step 1.4, performing image multi-band fusion on the foreground defect image and the random position of the target interest detection area, constructing a Laplacian pyramid on the foreground defect image and the background image, and fusing each layer:
Figure BDA0003050158810000032
in the above formula, superscript i denotes the ith layer of the laplacian pyramid;
Figure BDA0003050158810000033
features of an i-th layer of the laplacian pyramid representing the output fused image;
Figure BDA0003050158810000034
laplaca representing foreground defect imagesFeatures of the ith layer of the pyramid;
Figure BDA0003050158810000035
features of the ith layer of the laplacian pyramid representing a background image; riA fusion area of an ith layer is represented, and the value of i is 2 generally; the multi-band fusion uses larger scale fusion at a low frequency position to avoid the cutting of a foreground defect and a background, and uses smaller scale fusion at a high frequency position to avoid the interference of background information on the foreground defect;
step 2, performing multi-scale defect region feature extraction on the data set by using a depth residual error network and a feature pyramid network to obtain a multi-scale feature image; the multi-scale feature images are fused by utilizing a feature pyramid network, so that the loss of small target defects in training is reduced, the defect detection generalization capability is improved, and a high-level feature map with high-level semantics is generated, wherein the high-level features have high-level semantic information and a larger receptive field and are suitable for detecting large objects; generating a shallow feature map with low-level semantics, wherein the shallow feature has low-level detail semantic information and a smaller receptive field and is suitable for detecting small objects; providing a series of possible defect candidate areas in the image by using an area detection network, and then adding the input and the output of the depth residual error network together by using a depth residual error network through jump layer connection to extract the image characteristics of the defect candidate areas; finally, classifying the feature map fused with all the information to complete the target classification and regression task; by means of convolution of a depth residual error network with the highest 152 layers, image features can be better learned, meanwhile, by means of a feature pyramid network, features of the image on different scales are extracted, large-scale position information and small-scale semantic information are combined, and the method is very suitable for a small-area defect detection task on a mobile phone screen;
and 3, extracting a target interest detection area based on the characteristic image in the step 2, aiming at reducing the detection area, improving the detection efficiency and avoiding background information interference: the background information is filtered, so that the accuracy of defect detection can be greatly improved; detecting a target interest detection area of an image to be detected shot at different stations through a large-scale anchored area detection network;
and 4, step 4: generating potential defect areas to be detected in the target interest detection area by using an area detection network, and judging whether defects exist in each potential defect area to be detected by combining the characteristic images in the step 2 to obtain the types of the defects; extracting feature vectors with equal length, judging the probability of each region as a foreground defect image or a background by a softmax function through setting anchoring with different scales by the region detection network, and regressing the positions of candidate regions of the foreground defect image;
and 5, outputting a defect detection result.
Preferably, the step 4 specifically comprises the following steps:
step 4.1, counting the sizes and proportions of defect samples in the data set, and setting a targeted anchoring scale; the anchored length-width proportion comprises five types of 0.2, 0.5, 1, 2 and 5, the pixel size comprises five types of 32 pixels, 64 pixels, 128 pixels, 256 pixels and 512 pixels, the defect area can be screened out better, and the regression process of the candidate frame can be accelerated;
step 4.2, combining the characteristic image, performing target interest detection area pooling operation on the defect candidate area, performing normalization processing on the image characteristic, and mapping the normalized image characteristic to probability distribution of defect categories through a softmax function, wherein the target function is as follows:
Figure BDA0003050158810000041
in the above formula, J (theta) is the objective function, N is the number of samples, y is the label class,
Figure BDA0003050158810000042
for prediction classes, θ is the model parameter, R (θ) is L to prevent overfitting2The regularization term, λ, is an empirically set parameter term.
Preferably, the defects in step 1 include hair, dirt, black spots, scratches and off-colors.
Preferably, in the step 2, the high-level semantics fuses the feature map with strong low-resolution semantic information or the high-level semantics fuses the feature map with weak high-resolution semantic information but rich spatial information.
Preferably, the target interest detection area in step 3 refers to an approximately rectangular mobile phone screen area in the image.
Preferably, the feature pyramid network in step 3 is composed of three parts, the first part is up-sampled by a convolutional network, the second part is down-sampled from the feature map, the third part is transversely connected, and the feature maps of the first two parts with the same size are fused to obtain the multi-scale feature map.
Preferably, in step 4.2, feature vectors with equal length are extracted by pooling the target interest detection region, and the maximum value of the defect classification probability is the classification category of the detection result.
The invention has the beneficial effects that:
aiming at the two points that the defect has stronger relevance with the background and the background information has great interference, the invention provides a method for extracting the interest area to be detected by the target by adding a layer of segmentation extraction process between the defect area detection networks, which can effectively avoid the interference of the background information; aiming at the two points of various defects, fuzzy definition and uneven quality of a calibration data set, the invention provides an image local area enhancement mechanism for the specific task requirement of the appearance defect detection of a mobile phone screen, can abandon the subjective difference of different annotators, has high expansibility and can be suitable for the continuous amplification of the task requirement; the defect detection method can effectively improve the defect detection capability and the defect classification accuracy, improve the defect detection efficiency and the real-time performance and meet the actual production requirement.
According to the invention, through the extraction of the interest detection area, the image preprocessing efficiency is effectively improved, the problem that the existing general target detection technology is greatly interfered by background factors in a small target detection task of mobile phone screen defect detection is solved, a targeted enhancement mode is provided for the problem of data set shortage in actual landing application, the early-stage manpower marking investment is greatly reduced, and the efficiency of the whole scheme is further improved.
Drawings
FIG. 1 is an overall flowchart of defect target detection according to the present invention;
FIG. 2 is a flow chart of defect sample enhancement according to the present invention;
FIG. 3 is a flow chart of an enhancement of the defect prospect of the present invention;
FIG. 4 is a flow chart of defect detection according to the present invention;
fig. 5 is a diagram of a feature pyramid network structure for extracting multi-scale image features according to the present invention.
Detailed Description
The present invention will be further described with reference to the following examples. The following examples are set forth merely to aid in the understanding of the invention. It should be noted that, for a person skilled in the art, several modifications can be made to the invention without departing from the principle of the invention, and these modifications and modifications also fall within the protection scope of the claims of the present invention.
Example 1:
the overall flow of a multi-level smart phone screen defect detection method is shown in fig. 1, and comprises the following steps:
step 1, as shown in FIG. 2, artificially marking defects in an image, and then separating a foreground defect image from a background image; on the basis of the labeled foreground defect image, the color and the size of the foreground defect image are enhanced (as shown in fig. 3), and the sample number and the form diversity of the foreground defect are expanded; finally, combining the enhanced foreground defect image and the enhanced background image to generate a data set which is reliable in annotation, balanced in category and suitable for an actual production scene; defects include hair, dirt, black spots, scratches, and off-colors;
step 1.1, manually marking the position and the category information of a rectangular area where each foreground defect image in a small number of images is located, and then separating the foreground defect image from the background image;
step 1.2, performing image denoising treatment on the foreground defect image: processing image noise by median filtering, setting the gray value of the target point as the median of all points in a region around the target point, and enabling the pixel value in the field of the target point to be closer to the true value so as to eliminate independent noise signals;
after the image filtering process is carried out and the noise is reduced, the definition of the image contour and the edge can be reduced, the image after the noise is reduced is sharpened by adopting a Laplacian operator, and the difference between an object in the image and a background gray value is enhanced:
Figure BDA0003050158810000061
Figure BDA0003050158810000062
Figure BDA0003050158810000063
Figure BDA0003050158810000064
in the above formula, f (x, y) is an image after noise reduction;
Figure BDA0003050158810000065
the image is subjected to Laplace transform; c is an enhancement coefficient and is generally 1; g (x, y) is the enhanced image;
randomly adjusting the chromaticity, the sharpness and the brightness of the foreground defect image, and randomly rotating the foreground defect image by a random angle and randomly zooming in a fixed proportion range;
1.3, four defects such as black spots, dirt, scratches, hairs and the like are obvious in characteristics, and for heterochromatic defects, the defects are characterized by color difference with a region adjacent to a background, so that the enhanced defect foreground is difficult to be universally matched with a background image; in the randomly selected background image area, adjusting the pixel value:
Figure BDA0003050158810000066
Figure BDA0003050158810000067
in the above formula, valuenewIs the enhanced pixel value; valueoldThe value range is 0-255 for the original pixel value; brightness represents the brightness ratio and takes a value of 0.9 or 1.1, the brightness value represents 0.9 dark heterochrosis, and the brightness value 1.1 represents bright heterochrosis; the value range of exp is 1.5-3, and the method is suitable for different color defects of different forms; a and b represent half the width and height of the rectangular background image area, respectively; the geometric meaning of the factor represents the ratio of the distance from a certain point to the defect center to the distance from the defect area boundary to the defect center, the value is less than 1 and is 0.95, the defect area contains a part of background information, and the generalization is improved; centerxAnd centeryThe horizontal and vertical coordinates of the center of the rectangular background image region are expressed by (center)x，centery) Establishing a rectangular coordinate system for an origin, wherein x and y respectively represent horizontal and vertical coordinates of points in a rectangular background image area;
step 1.4, performing image multi-band fusion on the foreground defect image and the random position of the target interest detection area, constructing a Laplacian pyramid on the foreground defect image and the background image, and fusing each layer:
Figure BDA0003050158810000071
in the above formula, superscript i denotes the ith layer of the laplacian pyramid;
Figure BDA0003050158810000072
features of an i-th layer of the laplacian pyramid representing the output fused image;
Figure BDA0003050158810000073
features of the ith layer of the laplacian pyramid representing the foreground defect image;
Figure BDA0003050158810000074
features of the ith layer of the laplacian pyramid representing a background image; riA fusion area of an ith layer is represented, and the value of i is 2 generally; the multi-band fusion uses larger scale fusion at a low frequency position to avoid the cutting of a foreground defect and a background, and uses smaller scale fusion at a high frequency position to avoid the interference of background information on the foreground defect;
step 2, as shown in fig. 4, performing multi-scale defect region feature extraction on the data set by using a depth residual error network and a feature pyramid network to obtain a multi-scale feature image; the multi-scale feature images are fused by utilizing a feature pyramid network, so that the loss of small target defects in training is reduced, the defect detection generalization capability is improved, and a high-level feature map with high-level semantics is generated, wherein the high-level features have high-level semantic information and a larger receptive field and are suitable for detecting large objects; generating a shallow feature map with low-level semantics, wherein the shallow feature has low-level detail semantic information and a smaller receptive field and is suitable for detecting small objects; providing a series of possible defect candidate areas in the image by using an area detection network, and then adding the input and the output of the depth residual error network together by using a depth residual error network through jump layer connection to extract the image characteristics of the defect candidate areas; finally, classifying the feature map fused with all the information to complete the target classification and regression task; by means of convolution of a depth residual error network with the highest 152 layers, image features can be better learned, meanwhile, by means of a feature pyramid network, features of the image on different scales are extracted, large-scale position information and small-scale semantic information are combined, and the method is very suitable for a small-area defect detection task on a mobile phone screen; in the high-level feature map, high-level semantics are fused with a feature map with strong low-resolution semantic information, or high-level semantics are fused with a feature map with weak high-resolution semantic information but rich spatial information;
step 3, as shown in fig. 4, extracting a target interest detection region based on the feature image in step 2, aiming at reducing the detection region, improving the detection efficiency and avoiding background information interference:
the method is different from the conventional target detection task which focuses on the whole image for target detection, the mobile phone screen appearance defect detection task only focuses on target detection of a specific area, images of different stations have different detection interest areas, and the proportion of the area of the detection area in the whole image is usually 50% -80%. The defects are defined relative to the screen appearance, and are not absolute standards, so that background factors outside the detection area have great interference, and the accuracy of defect detection can be greatly improved by filtering the background information.
The traditional edge contour detection algorithm can better extract the image area where the screen object is located, but has the following problems:
1) the time consumption is long, the time occupation ratio of image cutting in the whole model detection process is up to 50%, and the bottleneck problem greatly influencing the detection efficiency is solved.
2) The generalization is poor, the image needs to be binarized in the preposing step of the edge contour detection algorithm, the artificially set binarization threshold cannot adapt to different shooting stations and illumination conditions, the self-adaptive binarization algorithm, such as a watershed algorithm, still cannot achieve an ideal effect under certain illumination conditions, and false detection of interference factors in a background area is easily caused.
The interested detection area of the image to be detected shot at different stations is relatively fixed and is limited by the position deviation and the illumination deviation of different degrees under the condition of hardware. The image characteristics are relatively fixed, and the network is easy to detect through large-scale anchoring area detection. Compared with an extraction method based on edge contour detection, the method can greatly improve the detection time efficiency due to the fact that the characteristic images of the preamble steps are shared.
The background information is filtered, so that the accuracy of defect detection can be greatly improved; detecting a target interest detection area of an image to be detected shot at different stations through a large-scale anchored area detection network;
the target interest detection area refers to a mobile phone screen area which is approximately rectangular in the image; as shown in fig. 5, the feature pyramid network is composed of three parts, the first part is up-sampled by a convolutional network, the second part is down-sampled from the feature map, and the third part is transversely connected, and the feature maps of the first two parts with the same size are fused to obtain a multi-scale feature map;
and 4, step 4: as shown in fig. 4, generating potential defect regions to be detected in the target interest detection region by using a region detection network, and determining whether a defect exists in each potential defect region to be detected by combining the feature images in step 2 to obtain a defect type; extracting feature vectors with equal length, judging the probability of each region as a foreground defect image or a background by a softmax function through setting anchoring with different scales by the region detection network, and regressing the positions of candidate regions of the foreground defect image;
step 4.1, counting the sizes and proportions of defect samples in the data set, and setting a targeted anchoring scale; the anchored length-width proportion comprises five types of 0.2, 0.5, 1, 2 and 5, the pixel size comprises five types of 32 pixels, 64 pixels, 128 pixels, 256 pixels and 512 pixels, the defect area can be screened out better, and the regression process of the candidate frame can be accelerated;
step 4.2, combining the characteristic image, performing target interest detection area pooling operation on the defect candidate area, performing normalization processing on the image characteristic, and mapping the normalized image characteristic to probability distribution of defect categories through a softmax function, wherein the target function is as follows:
Figure BDA0003050158810000081
in the above formula, J (theta) is the objective function, N is the number of samples, y is the label class,
Figure BDA0003050158810000082
for prediction classes, θ is the model parameter, R (θ) is L to prevent overfitting2A regular term, wherein lambda is a parameter term set by experience;
extracting feature vectors with equal length in a target interest detection area pooling mode, wherein the maximum value of defect classification probability is a classification category of a detection result;
and 5, outputting a defect detection result.
Example 2:
on the basis of embodiment 1, multi-level convolutional neural network model training and testing are performed by using the data set generated in step 1, the experimental data includes 5422 pieces of real scene labeling data, the experimental evaluation indexes are defect detection accuracy, recall rate and F1 score, the accuracy (precision) indicates how many samples in the predicted result are correct, and the recall (recall) indicates how many positive samples in the predicted result are correctly detected. F1 is defined as follows:
Figure BDA0003050158810000091
the definition of correct detection is that IoU (overlap degree) between the detection area and the label area is 0.5 or more, and the detection type and the label type are consistent. IoU, the ratio of the areas of the intersection and union of the "predicted region" and the "true region" was calculated, and the results of the experiment are shown in Table 1 below.
TABLE 1 Defect Category test result Table
Type of defect Rate of accuracy Recall rate F1
Black spot 97.2％ 93.1％ 0.95
Hair, hair-care product and method for producing the same 96.1％ 97.0％ 0.96
Scratch mark 95.2％ 95.1％ 0.95
Smudge 94.9％ 93.6％ 0.94
Different colors 93.3％ 94.7％ 0.94

Claims (7)

1.一种多层级的智能手机屏幕缺陷检测方法，其特征在于，包括以下步骤：1. a multi-level smart phone screen defect detection method, is characterized in that, comprises the following steps: 步骤1、人工标注图像中的缺陷，然后对前景缺陷图像和背景图像进行分离；在标注的前景缺陷图像基础上，对前景缺陷图像在颜色和尺寸上进行增强；最后结合增强后的前景缺陷图像和背景图像生成数据集；Step 1. Manually mark the defects in the image, and then separate the foreground defect image and the background image; on the basis of the annotated foreground defect image, enhance the color and size of the foreground defect image; finally combine the enhanced foreground defect image and background images to generate datasets; 步骤1.1、人工标注出少量图像中每个前景缺陷图像所处的矩形区域位置和类别信息，然后对前景缺陷图像和背景图像进行分离；Step 1.1. Manually mark the position and category information of the rectangular area where each foreground defect image in a small number of images is located, and then separate the foreground defect image and the background image; 步骤1.2、对前景缺陷图像进行图像降噪处理：采用中值滤波处理图像噪声，将目标点的灰度值设定为目标点周围一区域所有点的中值，剔除独立噪声信号；Step 1.2, perform image noise reduction processing on the foreground defect image: use median filtering to process image noise, set the gray value of the target point as the median value of all points in an area around the target point, and eliminate independent noise signals; 采用拉普拉斯算子对降噪后的图像进行锐化处理，增强图像中物体和背景灰度值之间的差距：The Laplacian operator is used to sharpen the denoised image to enhance the difference between the gray value of the object and the background in the image:
Figure FDA0003050158800000011
Figure FDA0003050158800000011
Figure FDA0003050158800000012
Figure FDA0003050158800000012
Figure FDA0003050158800000013
Figure FDA0003050158800000013
Figure FDA0003050158800000014
Figure FDA0003050158800000014
 上式中，f(x,y)为降噪后的图像；
Figure FDA0003050158800000015
为拉普拉斯变换后的图像；c为增强系数；g(x,y)为增强后的图像；In the above formula, f(x, y) is the image after noise reduction;
Figure FDA0003050158800000015
is the image after Laplace transformation; c is the enhancement coefficient; g(x, y) is the image after enhancement; 对前景缺陷图像进行色度、锐度、亮度上的随机调整，对前景缺陷图像进行随机角度的旋转、固定比例范围内的随机缩放；Randomly adjust the chroma, sharpness and brightness of the foreground defect image, and perform random angle rotation and random scaling within a fixed scale range for the foreground defect image; 步骤1.3、在随机选取的背景图像区域，对其像素值做调整：Step 1.3. Adjust the pixel value in the randomly selected background image area:
Figure FDA0003050158800000016
Figure FDA0003050158800000016
Figure FDA0003050158800000017
Figure FDA0003050158800000017
 上式中，valuenew是增强后的像素值；valueold为原像素值，取值范围为0～255；brightness表示亮度比值，取值为0.9或1.1，brightness取值表示0.9暗异色，brightness取值1.1表示亮异色；exp取值范围为1.5～3；a和b分别表示矩形背景图像区域宽度和高度的一半；factor表示某点到缺陷中心的距离和缺陷区域边界到缺陷中心的距离之比；centerx和centery表示矩形背景图像区域中心的横纵坐标，并以(centerx，centery)为原点建立直角坐标系，x和y分别表示在矩形背景图像区域内点的横纵坐标；In the above formula, value new is the enhanced pixel value; value old is the original pixel value, ranging from 0 to 255; brightness represents the brightness ratio, which is 0.9 or 1.1, and the value of brightness represents 0.9 dark color, brightness A value of 1.1 means bright different colors; exp ranges from 1.5 to 3; a and b represent half of the width and height of the rectangular background image area, respectively; factor represents the distance from a point to the center of the defect and the distance from the boundary of the defect area to the center of the defect Ratio; center x and center y represent the horizontal and vertical coordinates of the center of the rectangular background image area, and use (center x , center y ) as the origin to establish a rectangular coordinate system, x and y represent the horizontal and vertical points in the rectangular background image area respectively. coordinate; 步骤1.4、将前景缺陷图像与目标兴趣检测区域的随机位置进行图像多频段融合，对前景缺陷图像和背景图像构建拉普拉斯金字塔，对每一层进行融合：Step 1.4. Perform multi-band image fusion of the foreground defect image and the random position of the target interest detection area, construct a Laplacian pyramid for the foreground defect image and the background image, and fuse each layer:
Figure FDA0003050158800000021
Figure FDA0003050158800000021
 上式中，上标i表示拉普拉斯金字塔的第i层；
Figure FDA0003050158800000022
表示输出的融合图像的拉普拉斯金字塔的第i层的特征；
Figure FDA0003050158800000023
表示前景缺陷图像的拉普拉斯金字塔的第i层的特征；
Figure FDA0003050158800000024
表示背景图像的拉普拉斯金字塔的第i层的特征；Ri表示第i层的融合区域；In the above formula, the superscript i represents the i-th layer of the Laplacian pyramid;
Figure FDA0003050158800000022
The feature of the i-th layer of the Laplacian pyramid representing the output fused image;
Figure FDA0003050158800000023
The features of the i-th layer of the Laplacian pyramid representing the foreground defect image;
Figure FDA0003050158800000024
Represents the feature of the i-th layer of the Laplacian pyramid of the background image; R i represents the fusion region of the i-th layer; 步骤2、利用深度残差网络和特征金字塔网络对数据集进行多尺度缺陷区域特征抽取，得到多尺度特征图像；利用特征金字塔网络将多尺度特征图像融合，生成具有高级语义的高层特征图，生成具有低级语义的浅层特征图；由区域检测网络提供图像中的缺陷候选区，再使用深度残差网络通过跳层连接将深度残差网络的输入、输出加在一起提取缺陷候选区的图像特征；最后将特征图进行分类；Step 2. Use the deep residual network and the feature pyramid network to extract the multi-scale defect area features of the dataset to obtain multi-scale feature images; use the feature pyramid network to fuse the multi-scale feature images to generate high-level feature maps with high-level semantics. Shallow feature map with low-level semantics; the defect candidate area in the image is provided by the region detection network, and then the input and output of the deep residual network are added together through the skip layer connection to extract the image features of the defect candidate area using the deep residual network. ; Finally, classify the feature map; 步骤3、基于步骤2中特征图像抽取目标兴趣检测区域：过滤背景信息；通过大尺度锚定的区域检测网络检出不同工位拍摄的待检测图像的目标兴趣检测区域；Step 3. Extract the target interest detection area based on the feature image in step 2: filter the background information; detect the target interest detection area of the to-be-detected image captured by different stations through the large-scale anchored area detection network; 步骤4：在目标兴趣检测区域用区域检测网络生成待检测潜在缺陷区域，并结合步骤2中特征图像判断每个待检测潜在缺陷区域中是否有缺陷存在，得到缺陷的种类；再抽取等长的特征向量，区域检测网络通过设定不同尺度的锚定，以softmax函数判断每个区域为前景缺陷图像或背景的概率，并对前景缺陷图像候选区域的位置进行回归；Step 4: In the target interest detection area, use the area detection network to generate the potential defect area to be detected, and combine the feature images in step 2 to determine whether there is a defect in each potential defect area to be detected, and obtain the type of defect; Feature vector, the region detection network uses the softmax function to determine the probability that each region is a foreground defect image or background by setting anchors of different scales, and regresses the position of the candidate region of the foreground defect image; 步骤5、输出缺陷检测结果。Step 5, output the defect detection result. 2.根据权利要求1所述多层级的智能手机屏幕缺陷检测方法，其特征在于，步骤4具体包括如下步骤：2. The multi-level smartphone screen defect detection method according to claim 1, wherein step 4 specifically comprises the following steps: 步骤4.1、对数据集中缺陷样本的尺寸及比例进行统计，进行锚定尺度的设定；锚定的长宽款比例包括0.2、0.5、1、2和5，像素尺寸包括32像素、64像素、128像素、256像素和512像素；Step 4.1. Count the size and proportion of defect samples in the data set, and set the anchor scale; the anchor length and width ratios include 0.2, 0.5, 1, 2 and 5, and the pixel size includes 32 pixels, 64 pixels, 128 pixels, 256 pixels and 512 pixels; 步骤4.2、结合特征图像，对缺陷候选区进行目标兴趣检测区域池化操作，对图像特征进行归一化处理，通过softmax函数将归一化的图像特征映射到缺陷类别的概率分布：Step 4.2. Combine the feature image, perform the target interest detection area pooling operation on the defect candidate area, normalize the image features, and map the normalized image features to the probability distribution of defect categories through the softmax function:
Figure FDA0003050158800000025
Figure FDA0003050158800000025
 上式中，J(θ)为目标函数，N为样本数，y为标注类别，
Figure FDA0003050158800000026
为预测类别，θ为模型参数，R(θ)为防止过拟合的L2正则项，λ为参数项。In the above formula, J(θ) is the objective function, N is the number of samples, y is the labeling category,
Figure FDA0003050158800000026
is the prediction category, θ is the model parameter, R(θ) is the L 2 regular term to prevent overfitting, and λ is the parameter term. 3.根据权利要求1所述多层级的智能手机屏幕缺陷检测方法，其特征在于：步骤1中缺陷包括毛发、脏污、黑点、划痕和异色。3 . The multi-level smartphone screen defect detection method according to claim 1 , wherein the defects in step 1 include hair, dirt, black spots, scratches and different colors. 4 . 4.根据权利要求1所述多层级的智能手机屏幕缺陷检测方法，其特征在于：步骤2高层特征图中高级语义融合低分辨率语义信息较强的特征图，或高级语义融合高分辨率语义信息较弱但空间信息丰富的特征图。4. The multi-level smart phone screen defect detection method according to claim 1, is characterized in that: the feature map that high-level semantic fusion low-resolution semantic information is stronger in step 2 high-level feature map, or high-level semantic fusion high-resolution semantics Feature maps with weak information but rich spatial information. 5.根据权利要求1所述多层级的智能手机屏幕缺陷检测方法，其特征在于：步骤3中目标兴趣检测区域指图像中的手机屏幕区域。5 . The multi-level smart phone screen defect detection method according to claim 1 , wherein the target interest detection area in step 3 refers to the mobile phone screen area in the image. 6 . 6.根据权利要求1所述多层级的智能手机屏幕缺陷检测方法，其特征在于：步骤3中特征金字塔网络由三部分组成，第一部分通过卷积网络上采样，第二部分从特征图出发下采样，第三部分横向连接，将前两部分同大小的特征图进行融合，得到多尺度的特征图。6. The multi-level smart phone screen defect detection method according to claim 1 is characterized in that: in step 3, the feature pyramid network consists of three parts, the first part is upsampled by the convolutional network, and the second part is downsampled from the feature map. Sampling, the third part is horizontally connected, and the feature maps of the same size in the first two parts are fused to obtain a multi-scale feature map. 7.根据权利要求2所述多层级的智能手机屏幕缺陷检测方法，其特征在于：步骤4.2中通过目标兴趣检测区域池化的方式来抽取等长的特征向量，缺陷分类概率的最大值为检测结果的分类类别。7. The multi-level smart phone screen defect detection method according to claim 2, is characterized in that: in step 4.2, the feature vector of equal length is extracted by means of target interest detection area pooling, and the maximum value of defect classification probability is detection The categorical category of the result.
CN202110489131.0A 2021-04-30 2021-04-30 A multi-level smartphone screen defect detection method Active CN113221881B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202110489131.0A CN113221881B (en) 2021-04-30 2021-04-30 A multi-level smartphone screen defect detection method

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202110489131.0A CN113221881B (en) 2021-04-30 2021-04-30 A multi-level smartphone screen defect detection method

Publications (2)

Publication Number Publication Date
CN113221881A true CN113221881A (en) 2021-08-06
CN113221881B CN113221881B (en) 2022-03-22

Family

ID=77090932

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202110489131.0A Active CN113221881B (en) 2021-04-30 2021-04-30 A multi-level smartphone screen defect detection method

Country Status (1)

Country Link
CN (1) CN113221881B (en)

Cited By (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN114022657A (en) * 2022-01-06 2022-02-08 高视科技(苏州)有限公司 Screen defect classification method, electronic equipment and storage medium
CN114445483A (en) * 2022-01-28 2022-05-06 泗阳三江橡塑有限公司 Injection molding part quality analysis method based on image pyramid
CN116542975A (en) * 2023-07-05 2023-08-04 成都数之联科技股份有限公司 Defect classification method, device, equipment and medium for glass panel
CN119180811A (en) * 2024-10-11 2024-12-24 荣耀终端有限公司 Method, apparatus, device and readable storage medium for model training and defect detection

Citations (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20030053673A1 (en) * 2001-09-18 2003-03-20 Piet Dewaele Radiographic scoring method
CN107016664A (en) * 2017-01-18 2017-08-04 华侨大学 A kind of bad pin flaw detection method of large circle machine
CN107678192A (en) * 2017-07-16 2018-02-09 中科院成都信息技术股份有限公司 A kind of Mura defects detection method and system based on machine vision
CN110514665A (en) * 2019-09-03 2019-11-29 博科视(苏州)技术有限公司 A kind of detection method of electronic product plastic shell scratch defects
US20200167907A1 (en) * 2017-08-29 2020-05-28 Ping An Technology (Shenzhen) Co., Ltd. Automatic screen state detection robot, method and computer-readable storage medium
CN111553929A (en) * 2020-05-12 2020-08-18 重庆邮电大学 Mobile phone screen defect segmentation method, device and equipment based on fusion network
CN112184744A (en) * 2020-11-29 2021-01-05 惠州高视科技有限公司 Display screen edge defect detection method and device

Patent Citations (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20030053673A1 (en) * 2001-09-18 2003-03-20 Piet Dewaele Radiographic scoring method
CN107016664A (en) * 2017-01-18 2017-08-04 华侨大学 A kind of bad pin flaw detection method of large circle machine
CN107678192A (en) * 2017-07-16 2018-02-09 中科院成都信息技术股份有限公司 A kind of Mura defects detection method and system based on machine vision
US20200167907A1 (en) * 2017-08-29 2020-05-28 Ping An Technology (Shenzhen) Co., Ltd. Automatic screen state detection robot, method and computer-readable storage medium
CN110514665A (en) * 2019-09-03 2019-11-29 博科视(苏州)技术有限公司 A kind of detection method of electronic product plastic shell scratch defects
CN111553929A (en) * 2020-05-12 2020-08-18 重庆邮电大学 Mobile phone screen defect segmentation method, device and equipment based on fusion network
CN112184744A (en) * 2020-11-29 2021-01-05 惠州高视科技有限公司 Display screen edge defect detection method and device

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
A.F. LIMAS SERAFIM: "Object detection in images of natural scenes represented by AR models using Laplacian pyramids: application to leather defects localization", 《PROCEEDINGS OF THE 1992 INTERNATIONAL CONFERENCE ON INDUSTRIAL ELECTRONICS, CONTROL, INSTRUMENTATION, AND AUTOMATION》 *

Cited By (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN114022657A (en) * 2022-01-06 2022-02-08 高视科技(苏州)有限公司 Screen defect classification method, electronic equipment and storage medium
CN114022657B (en) * 2022-01-06 2022-05-24 高视科技(苏州)有限公司 Screen defect classification method, electronic equipment and storage medium
CN114445483A (en) * 2022-01-28 2022-05-06 泗阳三江橡塑有限公司 Injection molding part quality analysis method based on image pyramid
CN116542975A (en) * 2023-07-05 2023-08-04 成都数之联科技股份有限公司 Defect classification method, device, equipment and medium for glass panel
CN116542975B (en) * 2023-07-05 2023-09-12 成都数之联科技股份有限公司 Defect classification method, device, equipment and medium for glass panel
CN119180811A (en) * 2024-10-11 2024-12-24 荣耀终端有限公司 Method, apparatus, device and readable storage medium for model training and defect detection
CN119180811B (en) * 2024-10-11 2025-03-25 荣耀终端股份有限公司 Model training and defect detection method, device, equipment and readable storage medium

Also Published As

Publication number Publication date
CN113221881B (en) 2022-03-22

Similar Documents

Publication Publication Date Title
CN113221881B (en) A multi-level smartphone screen defect detection method
CN111310558B (en) An intelligent extraction method for pavement defects based on deep learning and image processing
CN107833220B (en) Fabric defect detection method based on deep convolutional neural network and visual saliency
CN101620060B (en) Automatic detection method of particle size distribution
CN107578418B (en) Indoor scene contour detection method fusing color and depth information
CN108562589B (en) Method for detecting surface defects of magnetic circuit material
CN107610114B (en) Detection method of cloud, snow and fog in optical satellite remote sensing images based on support vector machine
CN107437243B (en) Tire impurity detection method and device based on X-ray image
CN113240623B (en) Pavement disease detection method and device
CN108648168A (en) IC wafer surface defects detection methods
CN111915704A (en) Apple hierarchical identification method based on deep learning
CN106846344A (en) A kind of image segmentation optimal identification method based on the complete degree in edge
CN105809121A (en) Multi-characteristic synergic traffic sign detection and identification method
CN107093175B (en) A kind of power battery welding quality detection method
CN104182985B (en) Remote sensing image change detection method
CN111161222B (en) A Visual Saliency Based Defect Detection Method for Printing Cylinders
CN105354865A (en) Automatic cloud detection method and system for multi-spectral remote sensing satellite image
CN102521600A (en) Method and system for identifying white-leg shrimp disease on basis of machine vision
CN118967672A (en) Industrial defect detection method, system, device and storage medium
CN110648330B (en) Defect detection method for camera glass
CN110211112A (en) A kind of casting defect inspection method based on filtering selective search
CN115082776A (en) Electric energy meter automatic detection system and method based on image recognition
CN111626150A (en) Commodity identification method
CN117372339A (en) PCB defect detection and identification method based on YOLO-SEE
CN114387226A (en) Rice grading method, device, equipment and storage medium

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant
TR01 Transfer of patent right
TR01 Transfer of patent right

Effective date of registration: 20231127

Address after: Room 521, Unit 2, Building 4, Dexin Yunchuan Business Center, Xihu District, Hangzhou City, Zhejiang Province, China (self declared)

Patentee after: Hangzhou Zhenwei Food Collection Technology Co.,Ltd.

Address before: 310015 No. 51 Huzhou street, Hangzhou, Zhejiang, Gongshu District

Patentee before: HANGZHOU City University