CN114862866B

CN114862866B - Calibration plate detection method and device, computer equipment and storage medium

Info

Publication number: CN114862866B
Application number: CN202210807726.0A
Authority: CN
Inventors: 朱烙盛; 吴锐正; 蒋念娟; 沈小勇; 吕江波
Original assignee: Shenzhen Smartmore Technology Co Ltd
Current assignee: Shenzhen Smartmore Technology Co Ltd
Priority date: 2022-07-11
Filing date: 2022-07-11
Publication date: 2022-09-20
Anticipated expiration: 2042-07-11
Also published as: CN114862866A

Abstract

The application relates to a calibration plate detection method, a calibration plate detection device, computer equipment and a storage medium. The method comprises the following steps: combining the segmented sub-images to obtain segmented combined images, and taking the segmented combined images with area values meeting preset conditions as maximum area closed contours; performing polygon fitting on the maximum area closed contour to obtain a fitted polygon, and determining at least one vertex two-dimensional image coordinate and vertex two-dimensional physical coordinates according to the fitted polygon; performing homography matrix solving on the vertex two-dimensional image coordinates and the vertex two-dimensional physical coordinates to obtain two-dimensional image coordinates of the vertex of the interior angle; performing homography matrix solving on the two-dimensional image coordinates of the inner corner vertex and the two-dimensional physical coordinates of the inner corner vertex to obtain the two-dimensional image coordinates of all checkerboard corner points in the calibration board; and obtaining pixel coordinates of all corner points according to the two-dimensional image coordinates of all the checkerboard corner points. The method can improve the detection efficiency of the calibration plate.

Description

Calibration plate detection method and device, computer equipment and storage medium

Technical Field

The present application relates to the field of computer technologies, and in particular, to a calibration plate detection method and apparatus, a computer device, and a storage medium.

Background

With the development of computer technology, a calibration plate detection technology appears, and a camera-based vision system plays an increasingly important role in the field of industrial manufacturing at present. In the tasks of target detection, three-dimensional reconstruction and the like, the calculation of the pose relationship between a detected target and a camera is an important link, and the pose calculation needs to be carried out by the camera firstly.

The existing calibration board has various types, such as checkerboards, dots, hexagonal dots, ArUco (square area coding of binary two-dimensional codes) and ChArUco (combination of the checkerboards and the ArUco), and has high requirements on robustness and precision of angular point detection, so that the calibration board has low anti-interference performance when used in a complex environment, and the calibration board is easy to distort, thereby causing low detection efficiency of the calibration board.

Disclosure of Invention

In view of the above, it is necessary to provide a calibration board detection method, apparatus, computer device and storage medium capable of improving calibration board detection efficiency.

In a first aspect, the present application provides a method for detecting a calibration plate. The method comprises the following steps: inputting the image of the calibration plate to be detected into a semantic segmentation model to obtain each segmented subimage corresponding to the calibration plate to be detected; combining the segmented sub-images based on the segmented sub-images to obtain segmented combined images, and taking the segmented combined images with area values meeting preset conditions as maximum area closed contours; performing polygon fitting on the maximum area closed contour to obtain a fitted polygon, and determining a vertex two-dimensional image coordinate and a vertex two-dimensional physical coordinate corresponding to at least one fitted polygon according to the fitted polygon; performing homography matrix solving based on the vertex two-dimensional image coordinates and the vertex two-dimensional physical coordinates to obtain the vertex two-dimensional image coordinates and the two-dimensional image coordinates of the internal angle vertex corresponding to the vertex two-dimensional physical coordinates; performing the homography matrix solving based on the two-dimensional image coordinates of the inner corner vertex and the two-dimensional physical coordinates of the inner corner vertex to obtain the two-dimensional image coordinates of all checkerboard corner points in the calibration board; and obtaining corner pixel coordinates corresponding to the two-dimensional image coordinates of all the checkerboard corner points according to the two-dimensional image coordinates of all the checkerboard corner points.

In one embodiment, the obtaining the vertex two-dimensional image coordinate and the two-dimensional image coordinate of the inner corner vertex corresponding to the vertex two-dimensional physical coordinate by performing homography matrix solving based on the vertex two-dimensional image coordinate and the vertex two-dimensional physical coordinate includes: performing single mapping transformation matrix calculation based on the vertex two-dimensional image coordinates and the vertex two-dimensional physical coordinates to obtain a first homography matrix; and after the first homography matrix is subjected to inversion matrix, normalization calculation is carried out by combining the vertex two-dimensional image coordinate and the two-dimensional physical coordinate of the vertex two-dimensional physical coordinate corresponding to the inner angle vertex to obtain the vertex two-dimensional image coordinate and the two-dimensional image coordinate of the inner angle vertex corresponding to the vertex two-dimensional physical coordinate.

In one embodiment, after the step of performing normalization calculation on the first homography matrix by combining the vertex two-dimensional image coordinate and the two-dimensional physical coordinate of the vertex of the internal angle corresponding to the vertex two-dimensional physical coordinate to obtain the vertex two-dimensional image coordinate and the two-dimensional image coordinate of the vertex of the internal angle corresponding to the vertex two-dimensional physical coordinate, the method further includes: and solving pixel coordinates based on the two-dimensional image coordinates of the vertex of the internal angle to obtain corner point pixel coordinates corresponding to the two-dimensional image coordinates of the vertex of the internal angle.

In one embodiment, the obtaining the homography matrix based on the two-dimensional image coordinates of the vertex of the internal angle and the two-dimensional physical coordinates of the vertex of the internal angle to obtain the two-dimensional image coordinates of all the checkerboard corner points in the calibration board includes: performing single mapping transformation matrix calculation based on the two-dimensional image coordinates of the vertex of the inner angle and the two-dimensional physical coordinates of the vertex of the inner angle to obtain a second homography matrix; and after the second homography matrix is subjected to matrix inversion, normalization calculation is carried out by combining the two-dimensional physical coordinates of all the checkerboard angular points in the calibration plate, and the two-dimensional image coordinates of all the checkerboard angular points in the calibration plate are obtained.

In one embodiment, the determining vertex two-dimensional image coordinates and vertex two-dimensional physical coordinates corresponding to at least one fitted polygon according to the fitted polygons includes: simplifying line segments based on the fitted polygon to obtain the fitted polygon after the line segments corresponding to the fitted polygon are simplified; judging the fitted polygon after simplifying the line segment by using a convex hull detection algorithm; and if the fitted polygon is a convex polygon after the line segments are simplified, determining at least one vertex two-dimensional image coordinate and vertex two-dimensional physical coordinate corresponding to the fitted polygon based on a preset maximum area threshold and the number of vertices corresponding to the convex polygon.

In one embodiment, the method further comprises: acquiring a calibration plate image for training, wherein the calibration plate image for training is used for training the semantic segmentation model; performing data enhancement on the training calibration plate image to obtain an enhanced training calibration plate image; and adjusting the image of the calibration plate for enhanced training to a preset image sampling size, and inputting the image to an untrained semantic segmentation model to obtain the trained semantic segmentation model.

In one embodiment, the performing data enhancement on the training calibration plate image to obtain an enhanced training calibration plate image includes: randomly rotating the training calibration plate image, turning over the training calibration plate image in the horizontal direction and turning over the training calibration plate image in the vertical direction to obtain a primary enhanced image, wherein the angle corresponding to the random rotation is between minus 30 degrees and 30 degrees; and adjusting the brightness of the preliminary enhanced image to obtain the enhanced calibration plate image for training.

In a second aspect, the application further provides a detection device for the calibration plate. The device comprises: the segmented subimage obtaining module is used for inputting the image of the calibration plate to be detected into the semantic segmentation model to obtain each segmented subimage corresponding to the calibration plate to be detected; a maximum area closed contour obtaining module, configured to combine the segmented sub-images to obtain each segmented combined image, and take the segmented combined image with an area value meeting a preset condition as a maximum area closed contour; the vertex two-dimensional coordinate determination module is used for performing polygon fitting on the maximum area closed contour to obtain a fitted polygon, and determining a vertex two-dimensional image coordinate and a vertex two-dimensional physical coordinate corresponding to at least one fitted polygon according to the fitted polygon; the two-dimensional image coordinate obtaining module of the vertex of the inner angle is used for carrying out homography matrix solving on the basis of the vertex two-dimensional image coordinate and the vertex two-dimensional physical coordinate to obtain the vertex two-dimensional image coordinate and the two-dimensional image coordinate of the vertex of the inner angle corresponding to the vertex two-dimensional physical coordinate; a two-dimensional image coordinate obtaining module of the checkerboard angular points, configured to perform the homography matrix solving based on the two-dimensional image coordinates of the internal corner vertices and the two-dimensional physical coordinates of the internal corner vertices, so as to obtain two-dimensional image coordinates of all the checkerboard angular points in the calibration board; and the checkerboard corner pixel coordinate module is used for obtaining corner pixel coordinates corresponding to the two-dimensional image coordinates of all the checkerboard corners according to the two-dimensional image coordinates of all the checkerboard corners.

In one embodiment, the two-dimensional image coordinate obtaining module of the vertex of the interior angle is configured to perform single mapping transformation matrix calculation based on the vertex two-dimensional image coordinate and the vertex two-dimensional physical coordinate to obtain a first homography matrix; and after the first homography matrix is subjected to inversion matrix, normalization calculation is carried out by combining the vertex two-dimensional image coordinate and the two-dimensional physical coordinate of the vertex two-dimensional physical coordinate corresponding to the inner angle vertex to obtain the vertex two-dimensional image coordinate and the two-dimensional image coordinate of the inner angle vertex corresponding to the vertex two-dimensional physical coordinate.

In one embodiment, the two-dimensional image coordinate obtaining module of the inner corner vertex is configured to perform pixel coordinate calculation based on the two-dimensional image coordinate of the inner corner vertex to obtain a corner pixel coordinate corresponding to the two-dimensional image coordinate of the inner corner vertex.

In one embodiment, the two-dimensional image coordinate obtaining module of the checkerboard corner point is configured to perform single mapping transformation matrix calculation based on the two-dimensional image coordinate of the internal corner vertex and the two-dimensional physical coordinate of the internal corner vertex to obtain a second homography matrix; and after the second homography matrix is subjected to matrix inversion, normalization calculation is carried out by combining the two-dimensional physical coordinates of all the checkerboard angular points in the calibration plate, and the two-dimensional image coordinates of all the checkerboard angular points in the calibration plate are obtained.

In one embodiment, the vertex two-dimensional coordinate determination module is configured to perform line segment simplification based on the fitted polygon to obtain a fitted polygon after line segment simplification corresponding to the fitted polygon; judging the fitted polygon after simplifying the line segment by using a convex hull detection algorithm; and if the fitted polygon is a convex polygon after the line segments are simplified, determining at least one vertex two-dimensional image coordinate and vertex two-dimensional physical coordinate corresponding to the fitted polygon based on a preset maximum area threshold and the number of vertices corresponding to the convex polygon.

In one embodiment, the semantic segmentation model training module is configured to acquire a calibration plate image for training, where the calibration plate image for training is used to train the semantic segmentation model; performing data enhancement on the training calibration plate image to obtain an enhanced training calibration plate image; and adjusting the image of the calibration plate for enhanced training to a preset image sampling size, and inputting the image to an untrained semantic segmentation model to obtain the trained semantic segmentation model.

In one embodiment, the semantic segmentation model training module is configured to perform random rotation, horizontal direction inversion and vertical direction inversion on the calibration plate image for training to obtain a preliminary enhanced image, where an angle corresponding to the random rotation is between negative 30 degrees and 30 degrees; and adjusting the brightness of the preliminary enhanced image to obtain the enhanced calibration board image for training.

In a third aspect, the present application also provides a computer device. The computer device comprises a memory storing a computer program and a processor implementing the following steps when executing the computer program: inputting the image of the calibration plate to be detected into a semantic segmentation model to obtain each segmented subimage corresponding to the calibration plate to be detected; combining the segmented sub-images based on the segmented sub-images to obtain segmented combined images, and taking the segmented combined images with area values meeting preset conditions as maximum area closed contours; performing polygon fitting on the maximum area closed contour to obtain a fitted polygon, and determining a vertex two-dimensional image coordinate and a vertex two-dimensional physical coordinate corresponding to at least one fitted polygon according to the fitted polygon; performing homography matrix solving based on the vertex two-dimensional image coordinates and the vertex two-dimensional physical coordinates to obtain the vertex two-dimensional image coordinates and the two-dimensional image coordinates of the internal angle vertex corresponding to the vertex two-dimensional physical coordinates; performing the homography matrix solving based on the two-dimensional image coordinates of the inner corner vertex and the two-dimensional physical coordinates of the inner corner vertex to obtain the two-dimensional image coordinates of all checkerboard corner points in the calibration board; and obtaining corner pixel coordinates corresponding to the two-dimensional image coordinates of all the checkerboard corner points according to the two-dimensional image coordinates of all the checkerboard corner points.

In a fourth aspect, the present application further provides a computer-readable storage medium. The computer-readable storage medium having stored thereon a computer program which, when executed by a processor, performs the steps of: inputting the image of the calibration plate to be detected into a semantic segmentation model to obtain each segmented subimage corresponding to the calibration plate to be detected; combining the segmented sub-images based on the segmented sub-images to obtain segmented combined images, and taking the segmented combined images with area values meeting preset conditions as maximum area closed contours; performing polygon fitting on the maximum area closed contour to obtain a fitted polygon, and determining a vertex two-dimensional image coordinate and a vertex two-dimensional physical coordinate corresponding to at least one fitted polygon according to the fitted polygon; performing homography matrix solving based on the vertex two-dimensional image coordinates and the vertex two-dimensional physical coordinates to obtain the vertex two-dimensional image coordinates and the two-dimensional image coordinates of the internal angle vertex corresponding to the vertex two-dimensional physical coordinates; performing the homography matrix solving based on the two-dimensional image coordinates of the inner corner vertex and the two-dimensional physical coordinates of the inner corner vertex to obtain the two-dimensional image coordinates of all checkerboard corner points in the calibration board; and obtaining corner pixel coordinates corresponding to the two-dimensional image coordinates of all the checkerboard corner points according to the two-dimensional image coordinates of all the checkerboard corner points.

The detection method, the detection device, the computer equipment and the storage medium of the calibration plate are characterized in that the image of the calibration plate to be detected is input into the semantic segmentation model to obtain each segmented subimage corresponding to the calibration plate to be detected; combining the segmented sub-images based on the segmented sub-images to obtain segmented combined images, and taking the segmented combined images with area values meeting preset conditions as maximum area closed contours; performing polygon fitting on the maximum area closed contour to obtain a fitted polygon, and determining a vertex two-dimensional image coordinate and a vertex two-dimensional physical coordinate corresponding to at least one fitted polygon according to the fitted polygon; performing homography matrix solving based on the vertex two-dimensional image coordinates and the vertex two-dimensional physical coordinates to obtain the vertex two-dimensional image coordinates and the two-dimensional image coordinates of the inner angle vertex corresponding to the vertex two-dimensional physical coordinates; performing homography matrix solving based on the two-dimensional image coordinates of the vertex of the inner angle and the two-dimensional physical coordinates of the vertex of the inner angle to obtain the two-dimensional image coordinates of all checkerboard angular points in the calibration board; and obtaining corner pixel coordinates corresponding to the two-dimensional image coordinates of all the checkerboard corner points according to the two-dimensional image coordinates of all the checkerboard corner points.

The calibration plate detection method based on the deep learning semantic segmentation model can be suitable for various cameras to detect calibration plates in a calibration link, firstly, the semantic segmentation model is used for segmenting a checkerboard area, and then, the repeated reprojection algorithm is used for obtaining the angular point coordinates of the calibration plates. The advantage of deep learning semantic segmentation can be combined, the anti-interference performance to complex environment and the detection capability to calibration plates with different placing angles can be enhanced, and the detection efficiency of the calibration plates is improved.

Drawings

FIG. 1 is a diagram illustrating an exemplary embodiment of a calibration plate detection method;

FIG. 2 is a schematic flow chart illustrating a method for inspecting a calibration plate according to an embodiment;

FIG. 3 is a schematic flow chart illustrating a method for obtaining coordinates of two-dimensional images of vertices of interior angles according to an embodiment;

FIG. 4 is a schematic flow chart illustrating a method for obtaining coordinates of corner pixels of an interior corner vertex in one embodiment;

FIG. 5 is a schematic flow chart of a method for obtaining coordinates of two-dimensional images of all corner points of a checkerboard in one embodiment;

FIG. 6 is a schematic flow chart diagram illustrating a method for determining two-dimensional image coordinates and two-dimensional physical coordinates of vertices, according to one embodiment;

FIG. 7 is a schematic flow chart diagram illustrating a method for obtaining a trained semantic segmentation model in one embodiment;

FIG. 8 is a schematic flow chart diagram illustrating a method for obtaining enhanced training calibration plate images, in accordance with one embodiment;

FIG. 9 is a schematic flow chart diagram illustrating a method for model inference computation in one embodiment;

FIG. 10 is a schematic diagram illustrating the definition of the vertices of the interior angles of the checkerboard calibration board in one embodiment;

FIG. 11 is a block diagram showing a structure of a detecting unit for a calibration plate according to an embodiment;

FIG. 12 is a diagram showing an internal structure of a computer device in one embodiment;

FIG. 13 is a diagram of a computer storage medium in a configuration of one embodiment.

Detailed Description

In order to make the objects, technical solutions and advantages of the present application more apparent, the present application is described in further detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the present application and are not intended to limit the present application.

The method for detecting the calibration plate provided by the embodiment of the application can be applied to the application environment shown in fig. 1. The terminal 102 acquires data, the server 104 receives the data of the terminal 102 in response to an instruction of the terminal 102 and performs calculation on the acquired data, and the server 104 transmits the calculation result of the data back to the terminal 102 and is displayed by the terminal 102. Wherein the terminal 102 communicates with the server 104 via a network. The data storage system may store data that the server 104 needs to process. The data storage system may be integrated on the server 104, or may be located on the cloud or other network server. The server 104 acquires an edge top plate image to be detected from the terminal 102, and then inputs the image of the calibration plate to be detected into the semantic segmentation model to obtain each segmented subimage corresponding to the calibration plate to be detected; combining the segmented sub-images based on the segmented sub-images to obtain segmented combined images, and taking the segmented combined images with area values meeting preset conditions as maximum area closed contours; performing polygon fitting on the maximum area closed contour to obtain a fitted polygon, and determining a vertex two-dimensional image coordinate and a vertex two-dimensional physical coordinate corresponding to at least one fitted polygon according to the fitted polygon; performing homography matrix solving based on the vertex two-dimensional image coordinates and the vertex two-dimensional physical coordinates to obtain the vertex two-dimensional image coordinates and the two-dimensional image coordinates of the inner angle vertex corresponding to the vertex two-dimensional physical coordinates; performing homography matrix solving based on the two-dimensional image coordinates of the vertex of the inner angle and the two-dimensional physical coordinates of the vertex of the inner angle to obtain the two-dimensional image coordinates of all checkerboard angular points in the calibration board; and obtaining corner pixel coordinates corresponding to the two-dimensional image coordinates of all the checkerboard corner points according to the two-dimensional image coordinates of all the checkerboard corner points. The terminal 102 may be, but not limited to, various personal computers, notebook computers, smart phones, tablet computers, internet of things devices and portable wearable devices, and the internet of things devices may be smart speakers, smart televisions, smart air conditioners, smart car-mounted devices, and the like. The portable wearable device can be a smart watch, a smart bracelet, a head-mounted device, and the like. The server 104 may be implemented as a stand-alone server or as a server cluster comprised of multiple servers.

In one embodiment, as shown in fig. 2, a calibration board detection method is provided, which is described by taking the application of the method to the server in fig. 1 as an example, and includes the following steps:

step 202, inputting the image of the calibration board to be detected into the semantic segmentation model to obtain each segmented sub-image corresponding to the calibration board to be detected.

The calibration plate image to be detected can be an image corresponding to a geometric model of camera imaging, which is used in machine vision, image measurement, photogrammetry, three-dimensional reconstruction and other applications, and is used for correcting lens distortion, determining a conversion relation between a physical size and a pixel, and determining a correlation between a three-dimensional geometric position of a certain point on the surface of a space object and a corresponding point in the image. The camera shoots the array flat plate with the fixed-spacing pattern, and a geometric model of the camera can be obtained through calculation of a calibration algorithm, so that high-precision measurement and reconstruction results are obtained. Commonly used calibration plate images to be detected generally have a solid circle array pattern and a checkerboard pattern.

The trained semantic segmentation model can be obtained by performing multiple times of cyclic training on untrained semantic segmentation models through a large number of training samples and adjusting the parameters of the models until the accuracy rate meets the service requirement, and can be used for performing semantic segmentation on the image of the calibration plate to be detected, wherein the semantic segmentation model meets the service requirement. The semantic segmentation model may be, but is not limited to, a convolutional neural network model, a cyclic neural network model, a long-short term memory network model, or the like.

The segmented sub-images may be a plurality of segmented images obtained by segmenting the calibration board to be detected by using the trained semantic segmentation model, and the segmented images are used for calculation corresponding to detection of the subsequent calibration board.

Specifically, an image of a calibration board to be detected, which needs to be used for detection, is input into a trained semantic segmentation model, the image of the calibration board to be detected obtains multi-level image features through a hierarchical network encoder, then the multi-level image features are input into a decoder of a network in a jumping connection mode, and a checkerboard segmentation result, namely each segmented sub-image, is obtained through the decoder.

For example, an image a of a calibration board to be detected, which needs to be used for detection, is input into a trained semantic segmentation model, the semantic segmentation model is obtained by using an Encoder-Decoder model as a basis, and a plurality of segmented sub-images corresponding to the image of the calibration board to be detected are obtained through semantic segmentation.

And 204, combining the segmented sub-images based on the segmented sub-images to obtain the segmented combined images, and taking the segmented combined images with area values meeting preset conditions as maximum area closed contours.

The divided combined image may be a combined image formed by combining the divided sub-images by various combining methods, and the specific number of the divided combined images is determined according to the number of the divided sub-images and the number of the combining methods. The combination method may be to splice the segmented sub-images according to a conditional random field, an edge-meeting code, a fully differentiable conditional random field, and the like.

Wherein the maximum area closed contour may be the one of the segmented combined images having the largest area.

Specifically, a binarization mask as a segmentation result is input, that is, each segmented sub-image has a pixel value of 1 detected as a checkerboard and a pixel value of 0 detected as a non-checkerboard. And finding out the corresponding outline of each segmented combined image by using an OpenCV closed outline algorithm, and sequencing according to the size of the area to obtain the maximum area closed outline.

For example, 100 segmented sub-images are combined to obtain N segmented combined images, the OpenCV closed contour algorithm is used to obtain the contours of the N segmented combined images, the contours of the N segmented combined images are sorted according to the size of the area, and the contour corresponding to the segmented combined image with the largest area is selected as the maximum area closed contour.

And step 206, performing polygon fitting on the maximum area closed contour to obtain a fitted polygon, and determining vertex two-dimensional image coordinates and vertex two-dimensional physical coordinates corresponding to at least one fitted polygon according to the fitted polygon.

The polygon fitting may be to fit the selected contour in the segmented combined image by using a computer to obtain a closed continuous curve, where the curve is a polygon.

The vertex two-dimensional physical coordinates may be two-dimensional coordinates obtained by labeling each vertex in the fitted polygon, and the coordinates can display the physical properties of the remarked vertex.

The vertex two-dimensional image coordinates may be two-dimensional image coordinates generated for each of a plurality of vertices in the polygon after fitting the contour in the divided combined image by using a polygon fitting method.

Specifically, carrying out polygon fitting on the maximum area closed contour by using a Ramer-Douglas-Peucker calculation method; then, using the function iscontourConvex of OpenCV to judge whether the Polygon is a Convex Polygon, wherein the Convex Polygon (Convex Polygon) can be a straight line formed by extending any one side of the Polygon to two directions infinitely, other sides are on the same side of the straight line, the inner angle is not a reflex angle, and the line segment between any two vertexes is positioned in the Polygon or on the side; the final polygon vertex coordinates are finally obtained by setting a maximum area threshold (set to 200 pixels) and the number of vertices of the polygon (e.g., scaling the plate to a rectangle, set to 4). The polygon vertex coordinates are divided into vertex two-dimensional physical coordinates and vertex two-dimensional image coordinates, the two-dimensional image coordinates and the two-dimensional physical coordinates are homography mapping relations, and the two relations are affine transformation from one plane to the other plane, so that the homography matrix homography can be used for solving the homography.

For example, polygonal fitting is performed on the maximum area closed contour M by using a Ramer-Douglas-Peucker calculation method; then, judging that the polygon M is a convex polygon by using the function iscontourConvex of OpenCV; and finally, 6 vertex coordinates of the final polygon are obtained by setting a maximum area threshold and the number of the vertexes of the polygon M to be 6, wherein the vertex coordinates are respectively 6 vertex two-dimensional image coordinates and 6 vertex two-dimensional physical coordinates.

And 208, performing homography matrix solving based on the vertex two-dimensional image coordinates and the vertex two-dimensional physical coordinates to obtain the vertex two-dimensional image coordinates and the two-dimensional image coordinates of the internal angle vertex corresponding to the vertex two-dimensional physical coordinates.

Wherein, the homography matrix may describe a mapping relationship between two planes. In the perceived scene, the feature points all fall on the same plane, and then the motion estimation can be performed through the homography matrix.

The two-dimensional image coordinates of the vertex of the internal angle can be obtained by performing mathematical calculations such as homography matrix calculation on the vertex two-dimensional image coordinates, the vertex two-dimensional physical coordinates and the two-dimensional physical coordinates of the vertex of the internal angle. The two-dimensional physical coordinates of the inner angle vertexes are obtained by manually marking each inner angle vertex of the image of the calibration board to be detected.

Specifically, according to a plurality of vertex two-dimensional image coordinates corresponding to the fitted polygon and a plurality of vertex two-dimensional physical coordinates labeled according to the fitted polygon, a homography matrix is obtained by using a findhomography algorithm of OpenCV, and a first homography matrix is obtained. And according to the two-dimensional physical coordinates of the plurality of internal angle vertexes labeled by the fitted polygon, multiplying the inverse matrix of the first homography matrix by the two-dimensional physical coordinates of the plurality of internal angle vertexes, and normalizing the obtained coordinates in the middle process to obtain vertex two-dimensional image coordinates and two-dimensional image coordinates of the plurality of internal angle vertexes corresponding to the vertex two-dimensional physical coordinates.

After the steps are realized, corner sub-pixelation is carried out on the two-dimensional image coordinates of the internal corner vertices corresponding to the fitted polygon by using a cornerSubPixel algorithm of OpenCV, and then the internal corner coordinates of a sub-pixel level are obtained.

For example, according to the 4 vertex two-dimensional image coordinates corresponding to the fitted rectangle and the 4 vertex two-dimensional physical coordinates labeled according to the fitted rectangle, the homography matrix is obtained by using the findhomography algorithm of OpenCV, and the first homography matrix H1 is obtained. And according to the two-dimensional physical coordinates of the 4 interior angle vertexes marked by the fitted rectangle, multiplying the inverse matrix of the first homography matrix H1 by the two-dimensional physical coordinates of the 4 interior angle vertexes to obtain a coordinate B1 of the middle process, and then normalizing to obtain a vertex two-dimensional image coordinate and a two-dimensional image coordinate of the 4 interior angle vertexes corresponding to the vertex two-dimensional physical coordinate.

And step 210, performing homography matrix solving based on the two-dimensional image coordinates of the vertex of the internal angle and the two-dimensional physical coordinates of the vertex of the internal angle to obtain the two-dimensional image coordinates of all the checkerboard angular points in the calibration board.

The two-dimensional image coordinates of all the checkerboard corner points can be obtained by performing homography matrix solving and other mathematical calculations on the two-dimensional physical coordinates of the vertex of the internal angle. The two-dimensional physical coordinates of the inner angle vertexes are obtained by manually marking each inner angle vertex of the image of the calibration board to be detected.

Specifically, according to two-dimensional physical coordinates of a plurality of internal angle vertices and two-dimensional image coordinates of the plurality of internal angle vertices (or coordinates of a plurality of internal angle points at a sub-pixel level) which are labeled by a fitted polygon, homography matrix calculation is performed by using a findhomography algorithm of OpenCV to obtain a second homography matrix, then an inverse matrix corresponding to the second homography matrix is multiplied by the two-dimensional physical coordinates of all internal angle points which are labeled by the fitted polygon, and the obtained middle process coordinates are normalized to obtain the two-dimensional image coordinates of all checkerboard corner points in the calibration board.

For example, according to the two-dimensional physical coordinates of 4 interior vertices and the two-dimensional image coordinates of 4 interior vertices (or the coordinates of 4 interior vertices at a subpixel level) labeled by the fitted rectangle, a homography matrix is obtained by using a findhomography algorithm of OpenCV, so as to obtain a second homography matrix H2, then an inverse matrix corresponding to the second homography matrix H2 is multiplied by the two-dimensional physical coordinates of all interior corners labeled by the fitted rectangle, and the obtained intermediate process coordinates B2 are normalized, so as to obtain the two-dimensional image coordinates of all checkerboard corners in the calibration plate.

And step 212, obtaining corner pixel coordinates corresponding to the two-dimensional image coordinates of all the checkerboard corner points according to the two-dimensional image coordinates of all the checkerboard corner points.

The corner pixel coordinates can be corner sub-pixel coordinates, namely a sub-pixel level corner detection algorithm in camera calibration, and a simple, convenient and effective sub-pixel level precision extraction algorithm is provided for the corner detection problem of a camera calibration template image.

Specifically, the sub-pixel coordinates of the two-dimensional image coordinates of all the checkerboard corner points are solved by using a sub-pixel corner detection algorithm corersubpixel in OpenCV, so as to obtain corner pixel coordinates corresponding to the two-dimensional image coordinates of all the checkerboard corner points.

For example, the two-dimensional image coordinates of all checkerboard corners in the rectangular calibration plate are 100, and a sub-pixel corner detection algorithm corensubpixel in OpenCV is used for obtaining sub-pixel coordinates, so as to obtain corner pixel coordinates corresponding to the two-dimensional image coordinates of 100 checkerboard corners in the rectangular calibration plate.

In the detection method of the calibration plate, the image of the calibration plate to be detected is input into the semantic segmentation model to obtain each segmented subimage corresponding to the calibration plate to be detected; combining the segmented sub-images based on the segmented sub-images to obtain segmented combined images, and taking the segmented combined images with area values meeting preset conditions as maximum area closed contours; performing polygon fitting on the maximum area closed contour to obtain a fitted polygon, and determining a vertex two-dimensional image coordinate and a vertex two-dimensional physical coordinate corresponding to at least one fitted polygon according to the fitted polygon; performing homography matrix solving based on the vertex two-dimensional image coordinates and the vertex two-dimensional physical coordinates to obtain the vertex two-dimensional image coordinates and the two-dimensional image coordinates of the inner angle vertex corresponding to the vertex two-dimensional physical coordinates; performing homography matrix solving based on the two-dimensional image coordinates of the vertex of the inner angle and the two-dimensional physical coordinates of the vertex of the inner angle to obtain the two-dimensional image coordinates of all checkerboard angular points in the calibration board; and obtaining corner pixel coordinates corresponding to the two-dimensional image coordinates of all the checkerboard corner points according to the two-dimensional image coordinates of all the checkerboard corner points.

The calibration plate detection method based on the deep learning semantic segmentation model can be suitable for various cameras to detect calibration plates in a calibration link, firstly, the semantic segmentation model is used for segmenting a checkerboard area, and then, the repeated reprojection algorithm is used for obtaining the angular point coordinates of the calibration plates. The advantage of deep learning semantic segmentation can be combined, the anti-interference performance to complex environment and the detection capability to different calibration plates with different placing angles can be enhanced, and the detection efficiency of the calibration plates is improved.

In an embodiment, as shown in fig. 3, performing homography matrix solving based on the vertex two-dimensional image coordinates and the vertex two-dimensional physical coordinates to obtain the vertex two-dimensional image coordinates and the two-dimensional image coordinates of the internal angle vertex corresponding to the vertex two-dimensional physical coordinates, includes:

step 302, performing single mapping transformation matrix calculation based on the vertex two-dimensional image coordinates and the vertex two-dimensional physical coordinates to obtain a first homography matrix.

The single mapping transformation matrix may be a transformation matrix with only one mapping relationship, where the transformation matrix may be a vertex two-dimensional image coordinate and a linear transformation of the vertex two-dimensional physical coordinate expressed by a matrix.

The first homography matrix can be a matrix obtained by linear transformation of vertex two-dimensional image coordinates and vertex two-dimensional physical coordinates, the matrix describes a mapping relation between two planes, and motion estimation can be performed through the homography matrix when feature points in a perceived scene all fall on the same plane.

Specifically, according to a plurality of vertex two-dimensional image coordinates corresponding to the fitted polygon and a plurality of vertex two-dimensional physical coordinates labeled according to the fitted polygon, a homography matrix is obtained by using a findhomography algorithm of OpenCV, and a first homography matrix is obtained.

For example, according to the 4 vertex two-dimensional image coordinates corresponding to the fitted rectangle and the 4 vertex two-dimensional physical coordinates labeled according to the fitted rectangle, the homography matrix is obtained by using the findhomography algorithm of OpenCV, and the first homography matrix H1 is obtained.

And 304, after the first homography matrix is subjected to inversion matrix, normalization calculation is carried out by combining the vertex two-dimensional image coordinates and the two-dimensional physical coordinates of the vertex two-dimensional physical coordinates corresponding to the inner angle vertex to obtain the vertex two-dimensional image coordinates and the two-dimensional image coordinates of the inner angle vertex corresponding to the vertex two-dimensional physical coordinates.

Specifically, according to the two-dimensional physical coordinates of the multiple interior angle vertices labeled by the fitted polygon, the inverse matrix of the first homography matrix is multiplied by the two-dimensional physical coordinates of the multiple interior angle vertices corresponding to the vertex two-dimensional physical coordinates, and the obtained coordinates in the middle process are normalized to obtain vertex two-dimensional image coordinates and two-dimensional image coordinates of the multiple interior angle vertices corresponding to the vertex two-dimensional physical coordinates.

For example, according to the two-dimensional physical coordinates of the 4 interior angle vertices labeled by the fitted rectangle, the inverse matrix of the first homography matrix H1 is multiplied by the two-dimensional physical coordinates of the 4 interior angle vertices corresponding to the vertex two-dimensional physical coordinates, and the obtained intermediate process coordinate B1 is normalized to obtain the vertex two-dimensional image coordinates and the two-dimensional image coordinates of the 4 interior angle vertices corresponding to the vertex two-dimensional physical coordinates.

In this embodiment, the vertex two-dimensional image coordinates and the vertex two-dimensional physical coordinates are subjected to single mapping transformation matrix calculation, and the obtained result is further subjected to normalization processing to obtain the two-dimensional image coordinates of the vertex of the inner angle, so that a plane homography mapping relationship from the calibration plate image coordinates to the calibration plate physical coordinates can be determined, and the second time of mapping transformation matrix calculation is facilitated.

In an embodiment, as shown in fig. 4, after the step of performing normalization calculation on the first homography matrix by using the vertex two-dimensional image coordinate and the two-dimensional physical coordinate of the vertex corresponding to the vertex two-dimensional physical coordinate to obtain the vertex two-dimensional image coordinate and the two-dimensional image coordinate of the vertex corresponding to the vertex two-dimensional physical coordinate, the method further includes:

and step 402, performing pixel coordinate calculation based on the two-dimensional image coordinates of the vertex of the internal angle to obtain corner point pixel coordinates corresponding to the two-dimensional image coordinates of the vertex of the internal angle.

Specifically, corner sub-pixelation is performed on the two-dimensional image coordinates of the multiple inner corner vertices corresponding to the fitted polygon by using a cornerSubPixel algorithm of the OpenCV, and then the multiple inner corner coordinates of the sub-pixel level corresponding to the two-dimensional image coordinates of the inner corner vertices are obtained.

For example, corner sub-pixelation is performed on the two-dimensional image coordinates of the 4 inner corner vertices corresponding to the fitted rectangle by using a cornerSubPixel algorithm of OpenCV, and then the two-dimensional image coordinates of the inner corner vertices corresponding to the 4 inner corner coordinates of the sub-pixel level are obtained.

In this embodiment, the pixel coordinates of the two-dimensional image coordinates of the inner corner vertex are obtained by solving the pixel coordinates, so as to obtain the corner pixel coordinates corresponding to the two-dimensional image coordinates of the inner corner vertex, and the inner corner coordinates of the subpixel level can be obtained preliminarily and used for calculating the second single mapping transformation matrix.

In an embodiment, as shown in fig. 5, performing homography matrix solving based on the two-dimensional image coordinates of the vertex of the internal angle and the two-dimensional physical coordinates of the vertex of the internal angle to obtain the two-dimensional image coordinates of all the checkerboard corner points in the calibration board includes:

step 502, performing single mapping transformation matrix calculation based on the two-dimensional image coordinates of the vertex of the internal angle and the two-dimensional physical coordinates of the vertex of the internal angle to obtain a second homography matrix.

The second homography matrix can be a matrix obtained by linear transformation of two-dimensional image coordinates of an internal angle vertex and two-dimensional physical coordinates of the internal angle vertex, the matrix describes a mapping relation between two planes, and motion estimation can be performed through the homography matrix if feature points in a perceived scene all fall on the same plane.

Specifically, according to the two-dimensional physical coordinates of the plurality of internal angle vertices and the two-dimensional image coordinates of the plurality of internal angle vertices (or the coordinates of the plurality of internal angle points at the sub-pixel level) labeled by the fitted polygon, a homography matrix is obtained by using a findhomography algorithm of OpenCV, so that a second homography matrix is obtained.

For example, according to the two-dimensional physical coordinates of 4 interior angle vertices labeled by the fitted rectangle and the two-dimensional image coordinates of 4 interior angle vertices (or the coordinates of 4 interior angle points at the subpixel level), the findhomograph algorithm of OpenCV is used to perform homography matrix calculation, so as to obtain the second homography matrix H2.

And step 504, after the second homography matrix is subjected to inverse matrix, normalization calculation is carried out by combining the two-dimensional physical coordinates of all the checkerboard angular points in the calibration plate, and the two-dimensional image coordinates of all the checkerboard angular points in the calibration plate are obtained.

Specifically, the inverse matrix corresponding to the second homography matrix is multiplied by the two-dimensional physical coordinates of all inner corner points marked by the fitted polygon, and the obtained intermediate process coordinates are normalized to obtain the two-dimensional image coordinates of all checkerboard corner points in the calibration plate.

For example, the inverse matrix corresponding to the second homography matrix H2 is multiplied by the two-dimensional physical coordinates of all the inner corner points labeled by the fitted rectangle, and the obtained intermediate process coordinates B2 are normalized to obtain the two-dimensional image coordinates of all the checkerboard corner points in the calibration board.

In the embodiment, through twice single mapping transformation matrix calculation, the planar homography mapping relationship between the two-dimensional image coordinates of all the checkerboard angular points in the calibration plate and the two-dimensional physical coordinates of all the checkerboard angular points in the calibration plate can be more accurate, and accurate extraction of corner sub pixel angular points in the next step can be facilitated.

In one embodiment, as shown in fig. 6, determining vertex two-dimensional image coordinates and vertex two-dimensional physical coordinates corresponding to at least one fitted polygon from the fitted polygons includes:

and step 602, simplifying line segments based on the fitted polygon to obtain the fitted polygon after simplifying the line segments corresponding to the fitted polygon.

The line segment simplification may be an operation of simplifying curve elements in the continuous curve of the fitted polygon.

Wherein, the fitted polygon after line segment simplification may be a fitted polygon after line segment simplification.

Specifically, segmenting the continuous curve of the fitted polygon according to the connection points in the curve to obtain a plurality of line segments. And (3) virtually connecting a straight line to the head point and the tail point of each curve by using a Ramer-Douglas-Peucker calculation method, solving the distance between all points and the straight line, finding out the maximum distance value, and comparing the maximum distance value with the limit difference. If the maximum distance value is smaller than the tolerance, the middle points on the curve are all rounded off; if the maximum distance value is larger than or equal to the tolerance, a coordinate point corresponding to the maximum distance value is reserved, the point is taken as a boundary, the curve is divided into two parts, and the method is repeatedly used for the two parts. Finally, the fitted polygon after the line segments corresponding to the fitted polygon are simplified is obtained.

For example, the continuous curve of the fitted polygon is segmented according to 99 connection points in the curve, and 100 line segments are obtained. And (3) virtually connecting a straight line to the head point and the tail point of each curve by using a Ramer-Douglas-Peucker calculation method, solving the distance between all points and the straight line, finding out the maximum distance value M, and comparing the maximum distance value M with the tolerance D. If the maximum distance value M is smaller than the tolerance D, all intermediate points on the curve are omitted; if the maximum distance value M is larger than or equal to the tolerance D, a coordinate point corresponding to the maximum distance value is reserved, the point is taken as a boundary, the curve is divided into two parts, and the method is repeatedly used for the two parts. Finally, the fitted polygon is obtained after the line segment corresponding to the fitted polygon is simplified.

And step 604, judging the fitted polygon after the line segment is simplified by using a convex hull detection algorithm.

The convex hull detection algorithm may be used to determine the convex condition of the contour of the fitted polygon after the line segment is simplified, and the convex hull detection algorithm may use a function iscontourconevex of OpenCV.

Specifically, a convex hull detection algorithm is used for detecting the outline of the fitted polygon after the line segment is simplified, whether a convex hull exists is judged, if the detection result is yes, the fitted polygon after the line segment is simplified is a convex polygon, and if the detection result is not, the fitted polygon after the line segment is simplified is not a convex polygon.

For example, the contour of the fitted polygon after the line segment is simplified is detected by using the function iscontourconevex of OpenCV, and the judgment result shows that 5 convex hulls exist in the polygon, thus indicating that the fitted polygon after the line segment is simplified is a convex polygon.

Step 606, if the fitted polygon is a convex polygon after the line segments are simplified, determining a vertex two-dimensional image coordinate and a vertex two-dimensional physical coordinate corresponding to at least one fitted polygon based on a preset maximum area threshold and the number of vertices corresponding to the convex polygon.

The maximum area threshold may be an area corresponding to the number of pixels included in one value, for example: the value is the area corresponding to 300 pixels.

Specifically, if the fitted polygon after the line segment simplification is determined to be the convex polygon through the convex hull detection algorithm, the vertex two-dimensional image coordinate and the vertex two-dimensional physical coordinate corresponding to at least one fitted polygon in the convex polygon are further determined based on the preset maximum area threshold and the number of vertices of the convex polygon detected by the convex hull detection algorithm.

For example, if it is determined that the fitted polygon after the line segment reduction is the convex polygon with five convex hulls through the convex hull detection algorithm, based on the preset maximum area threshold (500 pixel points) and the number of vertices of the convex polygon detected by the convex hull detection algorithm is 3, it is further determined that one value of 1 to 3 of the convex polygons is selected as the vertex two-dimensional image coordinate and the vertex two-dimensional physical coordinate corresponding to the fitted polygon.

In this embodiment, the fitted polygon is detected by using a convex hull detection algorithm, and the maximum area threshold and the number of vertices in the convex polygon are further determined, so that the vertex two-dimensional image coordinates and the vertex two-dimensional physical coordinates corresponding to the fitted polygon can be more accurately determined.

In one embodiment, as shown in fig. 7, the method further comprises:

step 702, obtaining a calibration plate image for training.

The training calibration plate image can be a training set image used for training a semantic segmentation model, and the training calibration plate image has similarity with a calibration plate image to be detected, so that the training calibration plate image can be used in machine vision, image measurement, photogrammetry, three-dimensional reconstruction and other applications to correct lens distortion; determining a conversion relation between the physical size and the pixel; and determining the mutual relationship between the three-dimensional geometric position of a certain point on the surface of the space object and the corresponding point in the image, and establishing the image corresponding to the geometric model imaged by the camera. Meanwhile, the training calibration board image generally has a solid circle array pattern and a chessboard pattern.

Specifically, a calibration board image for training is obtained, a segmentation network in an untrained semantic segmentation model adopts a high-resolution network structure HRNet, and a network framework adopts a lightweight-level HRnetv2_ w18 to accelerate network reasoning speed.

Step 704, performing data enhancement on the training calibration plate image to obtain an enhanced training calibration plate image.

Specifically, data enhancement is carried out on the calibration board image for training, wherein the data enhancement includes but is not limited to picture random rotation of-30 to 30 degrees, the picture is randomly turned in the horizontal direction, the picture is randomly turned in the vertical direction, the picture is randomly adjusted in brightness, and the enhanced calibration board image for training is obtained after the data enhancement.

For example, the calibration board image for training is randomly rotated by 15 degrees, the image of 60 degrees is randomly turned in the horizontal direction, the image of 130 degrees is randomly turned in the vertical direction, and the image is randomly adjusted in brightness, so as to obtain the enhanced calibration board image for training.

Step 706, adjusting the image of the calibration plate for enhanced training to a preset image sampling size, and inputting the image to the untrained semantic segmentation model to obtain the trained semantic segmentation model.

The image sampling size can be input into an untrained semantic segmentation model, and a sample object trained by the untrained semantic segmentation model is input by adopting a preset image size, so that the detection and segmentation functions are achieved at the same time.

The untrained semantic segmentation model can be an artificial intelligence model which is used for performing semantic segmentation but is not trained yet, and the model is generally realized by adopting an Encoder-Decoder model or an extended convolution method.

Specifically, the image sampling size is fixed in the training process, so that the segmentation network obtained through training is suitable for the condition of a small amount of labeled data, and meanwhile, the network has the functions of detection and segmentation. Then inputting the result into BOTTLENECK _ BLOCK and BASIC _ BLOCK, inputting the obtained result into FCN _ HEAD, and finally obtaining the segmentation result through calculation of FCN _ HEAD. The BOTTLENECK _ BLOCK and BASIC _ BLOCK are two BASIC modules, and the FCN _ HEAD refers to the feature decoding module in HRnet to obtain the final segmentation result.

For example, in the training process, the length of the fixed image sample is X, and the width of the fixed image sample is Y, then the fixed image sample is input into the bottomelneckk _ BLOCK and the BASIC _ BLOCK, the obtained result is input into the FCN _ HEAD, and the FCN _ HEAD is calculated to finally obtain the segmentation result corresponding to the training sample with the length of X and the width of Y.

In the embodiment, the calibration board detection method based on the deep learning semantic segmentation model is provided, so that the problem of stability of the current calibration board detection is effectively solved, the calibration board can be effectively detected and sub-pixel-level corner point detection is carried out under various complex scenes or under the condition that the calibration board is randomly placed, and a stable effect is provided for subsequent camera calibration.

In one embodiment, as shown in fig. 8, the data enhancement of the training calibration plate image to obtain an enhanced training calibration plate image includes:

and 802, randomly rotating the calibration board image for training, turning over in the horizontal direction and turning over in the vertical direction to obtain a primary enhanced image.

Specifically, the data enhancement is carried out on the calibration board image for training in a moving transformation mode, wherein the data enhancement includes but is not limited to random picture rotation of-30 to 30 degrees, random picture horizontal direction turning and random picture vertical direction turning.

For example, the calibration board image for training is subjected to data enhancement by randomly rotating pictures by 15 degrees, randomly turning pictures by 68 degrees in the horizontal direction and randomly turning pictures by 130 degrees in the vertical direction.

And step 804, adjusting the brightness of the preliminary enhanced image to obtain an enhanced calibration board image for training.

Specifically, data enhancement is performed on the calibration plate image for training in a brightness adjustment mode, wherein the brightness adjustment range is preset, and during adjustment, arbitrary and continuous brightness change can be performed within the preset range.

For example, the data enhancement is performed on the training calibration plate image in a manner of adjusting the brightness to be 500 lumens to 1000 lumens.

In this embodiment, the generalization of the network can be greatly enhanced by data enhancement in the network training process.

In one embodiment, as shown in fig. 9, the calibration board image is passed through a hierarchical network encoder to obtain multi-level image features, and then the multi-level image features are input into a network decoder by way of skip connection, and a checkerboard segmentation result is obtained by the decoder.

In one embodiment, the inner corner vertices of the checkerboard calibration panel are defined as shown in fig. 10, with corresponding two-dimensional image coordinates of the inner corner vertices and two-dimensional physical coordinates of the inner corner vertices for all the inner corner vertices.

It should be understood that, although the steps in the flowcharts related to the embodiments are shown in sequence as indicated by the arrows, the steps are not necessarily executed in sequence as indicated by the arrows. The steps are not performed in the exact order shown and described, and may be performed in other orders, unless explicitly stated otherwise. Moreover, at least a part of the steps in the flowcharts related to the above embodiments may include multiple steps or multiple stages, which are not necessarily performed at the same time, but may be performed at different times, and the order of performing the steps or stages is not necessarily sequential, but may be performed alternately or alternately with other steps or at least a part of the steps or stages in other steps.

Based on the same inventive concept, the embodiment of the application also provides a calibration board detection device for realizing the calibration board detection method. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the method, so that specific limitations in the embodiment of the detection device for one or more calibration plates provided below can be referred to the limitations on the detection method for the calibration plate in the foregoing, and details are not repeated herein.

In one embodiment, as shown in fig. 11, there is provided a calibration plate detecting apparatus including: the system comprises a segmented subimage obtaining module, a maximum area closed contour obtaining module, a vertex two-dimensional coordinate determining module, an inner corner vertex two-dimensional image coordinate obtaining module, a checkerboard corner two-dimensional image coordinate obtaining module and a checkerboard corner pixel coordinate module, wherein:

a segmented subimage obtaining module 1102, configured to input the image of the calibration board to be detected to the semantic segmentation model, and obtain each segmented subimage corresponding to the calibration board to be detected;

a maximum area closed contour obtaining module 1104, configured to combine the divided sub-images to obtain each divided combined image, and take the divided combined image with an area value meeting a preset condition as a maximum area closed contour;

a vertex two-dimensional coordinate determining module 1106, configured to perform polygon fitting on the maximum-area closed contour to obtain a fitted polygon, and determine a vertex two-dimensional image coordinate and a vertex two-dimensional physical coordinate corresponding to at least one fitted polygon according to the fitted polygon;

a two-dimensional image coordinate obtaining module 1108 for obtaining a homography matrix based on the vertex two-dimensional image coordinate and the vertex two-dimensional physical coordinate to obtain a vertex two-dimensional image coordinate and a two-dimensional image coordinate of the internal vertex corresponding to the vertex two-dimensional physical coordinate;

a two-dimensional image coordinate obtaining module 1110 for obtaining two-dimensional image coordinates of the checkerboard corner points, which is used for performing homography matrix solving based on the two-dimensional image coordinates of the inner corner vertices and the two-dimensional physical coordinates of the inner corner vertices to obtain the two-dimensional image coordinates of all the checkerboard corner points in the calibration board;

a checkerboard corner pixel coordinate module 1112, configured to obtain, according to the two-dimensional image coordinates of all checkerboard corners, corner pixel coordinates corresponding to the two-dimensional image coordinates of all checkerboard corners.

In one embodiment, the two-dimensional image coordinate obtaining module of the vertex of the inner angle is used for performing single mapping transformation matrix calculation based on the two-dimensional image coordinate of the vertex and the two-dimensional physical coordinate of the vertex to obtain a first homography matrix; and after the first homography matrix is subjected to inversion matrix, normalization calculation is carried out by combining the vertex two-dimensional image coordinates and the two-dimensional physical coordinates of the vertex two-dimensional physical coordinates corresponding to the vertex of the inner angle, so as to obtain the vertex two-dimensional image coordinates and the two-dimensional image coordinates of the vertex two-dimensional physical coordinates corresponding to the vertex of the inner angle.

In one embodiment, the two-dimensional image coordinate obtaining module is configured to perform single mapping transformation matrix calculation based on the two-dimensional image coordinates of the vertex of the internal angle and the two-dimensional physical coordinates of the vertex of the internal angle to obtain a second homography matrix; and after the second homography matrix is subjected to matrix inversion, normalization calculation is carried out by combining the two-dimensional physical coordinates of all the checkerboard angular points in the calibration plate, and the two-dimensional image coordinates of all the checkerboard angular points in the calibration plate are obtained.

In one embodiment, the vertex two-dimensional coordinate determination module is configured to perform line segment simplification based on the fitted polygon, and obtain the fitted polygon after line segment simplification corresponding to the fitted polygon; judging the fitted polygon after simplifying the line segment by using a convex hull detection algorithm; and if the fitted polygon is a convex polygon after the line segments are simplified, determining a vertex two-dimensional image coordinate and a vertex two-dimensional physical coordinate corresponding to at least one fitted polygon based on a preset maximum area threshold value and the number of vertices corresponding to the convex polygon.

In one embodiment, the semantic segmentation model training module is used for acquiring a calibration plate image for training, and the calibration plate image for training is used for training the semantic segmentation model; performing data enhancement on the training calibration plate image to obtain an enhanced training calibration plate image; and adjusting the image of the calibration plate for enhanced training to a preset image sampling size, and inputting the image to the untrained semantic segmentation model to obtain the trained semantic segmentation model.

In one embodiment, the semantic segmentation model training module is used for randomly rotating, horizontally turning and vertically turning the calibration plate image for training to obtain a primary enhanced image, wherein the angle corresponding to the random rotation is between negative 30 degrees and 30 degrees; and adjusting the brightness of the primary enhanced image to obtain an enhanced calibration board image for training. By performing semantic enhancement on training sample data of the semantic segmentation model, the training sample data has higher pertinence to the target training direction of the semantic segmentation model, the generalization performance of the semantic segmentation model is improved, and the segmentation efficiency of the semantic segmentation model is higher.

The modules in the detection device of the calibration plate can be wholly or partially realized by software, hardware and a combination thereof. The modules can be embedded in a hardware form or independent from a processor in the computer device, and can also be stored in a memory in the computer device in a software form, so that the processor can call and execute operations corresponding to the modules.

In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as shown in fig. 12. The computer device includes a processor, a memory, and a network interface connected by a system bus. Wherein the processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program, and a database. The internal memory provides an environment for the operation of an operating system and computer programs in the non-volatile storage medium. The database of the computer device is used for storing server data. The network interface of the computer device is used for communicating with an external terminal through a network connection. The computer program is executed by a processor to implement a calibration plate detection method.

Those skilled in the art will appreciate that the architecture shown in fig. 12 is merely a block diagram of some of the structures associated with the disclosed aspects and is not intended to limit the computing devices to which the disclosed aspects apply, as particular computing devices may include more or less components than those shown, or may combine certain components, or have a different arrangement of components.

In one embodiment, a computer device is further provided, which includes a memory and a processor, the memory stores a computer program, and the processor implements the steps of the above method embodiments when executing the computer program.

In one embodiment, a computer-readable storage medium is provided, as shown in fig. 13, storing a computer program, which when executed by a processor, performs the steps in the above-described method embodiments.

In one embodiment, a computer program product or computer program is provided that includes computer instructions stored in a computer-readable storage medium. The computer instructions are read by a processor of a computer device from a computer-readable storage medium, and the computer instructions are executed by the processor to cause the computer device to perform the steps in the above-mentioned method embodiments.

It should be noted that, the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, presented data, etc.) referred to in the present application are information and data authorized by the user or sufficiently authorized by each party.

It will be understood by those skilled in the art that all or part of the processes of the methods of the embodiments described above can be implemented by hardware instructions of a computer program, which can be stored in a non-volatile computer-readable storage medium, and when executed, can include the processes of the embodiments of the methods described above. Any reference to memory, database, or other medium used in the embodiments provided herein may include at least one of non-volatile and volatile memory. The nonvolatile Memory may include a Read-Only Memory (ROM), a magnetic tape, a floppy disk, a flash Memory, an optical Memory, a high-density embedded nonvolatile Memory, a resistive Random Access Memory (ReRAM), a Magnetic Random Access Memory (MRAM), a Ferroelectric Random Access Memory (FRAM), a Phase Change Memory (PCM), a graphene Memory, and the like. Volatile Memory can include Random Access Memory (RAM), external cache Memory, and the like. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM), among others. The databases referred to in various embodiments provided herein may include at least one of relational and non-relational databases. The non-relational database may include, but is not limited to, a block chain based distributed database, and the like. The processors referred to in the embodiments provided herein may be general purpose processors, central processing units, graphics processors, digital signal processors, programmable logic devices, quantum computing based data processing logic devices, etc., without limitation.

The technical features of the above embodiments can be arbitrarily combined, and for the sake of brevity, all possible combinations of the technical features in the above embodiments are not described, but should be considered as the scope of the present specification as long as there is no contradiction between the combinations of the technical features.

The above-mentioned embodiments only express several embodiments of the present application, and the description thereof is more specific and detailed, but not construed as limiting the scope of the present application. It should be noted that, for a person skilled in the art, several variations and modifications can be made without departing from the concept of the present application, which falls within the scope of protection of the present application. Therefore, the protection scope of the present application shall be subject to the appended claims.

Claims

1. A method of testing a calibration plate, the method comprising:

inputting the image of the calibration plate to be detected into a semantic segmentation model to obtain each segmented subimage corresponding to the calibration plate to be detected;

combining the segmented sub-images based on the segmented sub-images to obtain segmented combined images, and taking the segmented combined images with area values meeting preset conditions as maximum area closed contours;

performing polygon fitting on the maximum area closed contour to obtain a fitted polygon, and determining a vertex two-dimensional image coordinate and a vertex two-dimensional physical coordinate corresponding to at least one fitted polygon according to the fitted polygon;

performing homography matrix solving based on the vertex two-dimensional image coordinates and the vertex two-dimensional physical coordinates to obtain the vertex two-dimensional image coordinates and the two-dimensional image coordinates of the internal angle vertex corresponding to the vertex two-dimensional physical coordinates;

performing the homography matrix solving based on the two-dimensional image coordinates of the inner corner vertex and the two-dimensional physical coordinates of the inner corner vertex to obtain the two-dimensional image coordinates of all checkerboard corner points in the calibration board;

and obtaining corner pixel coordinates corresponding to the two-dimensional image coordinates of all the checkerboard corner points according to the two-dimensional image coordinates of all the checkerboard corner points.

2. The method according to claim 1, wherein performing homography matrix solving based on the vertex two-dimensional image coordinates and the vertex two-dimensional physical coordinates to obtain the vertex two-dimensional image coordinates and the vertex two-dimensional physical coordinates of an inner corner vertex corresponding to the vertex two-dimensional image coordinates comprises:

performing single mapping transformation matrix calculation based on the vertex two-dimensional image coordinates and the vertex two-dimensional physical coordinates to obtain a first homography matrix;

and after the first homography matrix is subjected to inversion matrix, normalization calculation is carried out by combining the vertex two-dimensional image coordinate and the two-dimensional physical coordinate of the vertex two-dimensional physical coordinate corresponding to the inner angle vertex to obtain the vertex two-dimensional image coordinate and the two-dimensional image coordinate of the inner angle vertex corresponding to the vertex two-dimensional physical coordinate.

3. The method according to claim 2, wherein after the step of performing the normalization calculation by combining the vertex two-dimensional image coordinates and the two-dimensional physical coordinates of the internal angle vertex corresponding to the vertex two-dimensional physical coordinates after the step of performing the inversion matrix on the first homography matrix to obtain the vertex two-dimensional image coordinates and the two-dimensional image coordinates of the internal angle vertex corresponding to the vertex two-dimensional physical coordinates, the method further comprises:

and solving pixel coordinates based on the two-dimensional image coordinates of the internal corner vertex to obtain corner point pixel coordinates corresponding to the two-dimensional image coordinates of the internal corner vertex.

4. The method according to claim 1, wherein the performing the homography matrix solving based on the two-dimensional image coordinates of the inner corner vertices and the two-dimensional physical coordinates of the inner corner vertices to obtain the two-dimensional image coordinates of all tessellated corner points in the calibration board comprises:

performing single mapping transformation matrix calculation based on the two-dimensional image coordinates of the vertex of the inner angle and the two-dimensional physical coordinates of the vertex of the inner angle to obtain a second homography matrix;

and after the second homography matrix is subjected to matrix inversion, normalization calculation is carried out by combining the two-dimensional physical coordinates of all the checkerboard angular points in the calibration plate, and the two-dimensional image coordinates of all the checkerboard angular points in the calibration plate are obtained.

5. The method of claim 1, wherein determining vertex two-dimensional image coordinates and vertex two-dimensional physical coordinates corresponding to at least one of the fitted polygons from the fitted polygons comprises:

simplifying line segments based on the fitted polygon to obtain the fitted polygon after the line segments corresponding to the fitted polygon are simplified;

judging the fitted polygon after simplifying the line segment by using a convex hull detection algorithm;

and if the fitted polygon is a convex polygon after the line segments are simplified, determining at least one vertex two-dimensional image coordinate and vertex two-dimensional physical coordinate corresponding to the fitted polygon based on a preset maximum area threshold and the number of vertices corresponding to the convex polygon.

6. The method according to any one of claims 1 to 5, further comprising:

acquiring a calibration plate image for training, wherein the calibration plate image for training is used for training the semantic segmentation model;

performing data enhancement on the training calibration plate image to obtain an enhanced training calibration plate image;

and adjusting the image of the calibration plate for enhanced training to a preset image sampling size, and inputting the image to an untrained semantic segmentation model to obtain the semantic segmentation model.

7. The method of claim 6, wherein the enhancing the data of the training calibration plate image to obtain an enhanced training calibration plate image comprises:

randomly rotating the training calibration plate image, turning over the training calibration plate image in the horizontal direction and turning over the training calibration plate image in the vertical direction to obtain a primary enhanced image, wherein the angle corresponding to the random rotation is between minus 30 degrees and 30 degrees;

and adjusting the brightness of the preliminary enhanced image to obtain the enhanced calibration board image for training.

8. A calibration plate detection device, said device comprising:

the segmented subimage obtaining module is used for inputting the image of the calibration plate to be detected into the semantic segmentation model to obtain each segmented subimage corresponding to the calibration plate to be detected;

a maximum area closed contour obtaining module, configured to combine the segmented sub-images to obtain each segmented combined image, and take the segmented combined image with an area value meeting a preset condition as a maximum area closed contour;

the vertex two-dimensional coordinate determination module is used for performing polygon fitting on the maximum area closed contour to obtain a fitted polygon, and determining a vertex two-dimensional image coordinate and a vertex two-dimensional physical coordinate corresponding to at least one fitted polygon according to the fitted polygon;

the two-dimensional image coordinate obtaining module of the vertex of the inner angle is used for carrying out homography matrix solving on the basis of the vertex two-dimensional image coordinate and the vertex two-dimensional physical coordinate to obtain the vertex two-dimensional image coordinate and the two-dimensional image coordinate of the vertex of the inner angle corresponding to the vertex two-dimensional physical coordinate;

a two-dimensional image coordinate obtaining module of the checkerboard angular points, configured to perform the homography matrix solving based on the two-dimensional image coordinates of the internal angle vertices and the two-dimensional physical coordinates of the internal angle vertices, so as to obtain two-dimensional image coordinates of all the checkerboard angular points in the calibration board;

and the checkerboard corner pixel coordinate module is used for obtaining corner pixel coordinates corresponding to the two-dimensional image coordinates of all the checkerboard corners according to the two-dimensional image coordinates of all the checkerboard corners.

9. A computer device comprising a memory and a processor, the memory storing a computer program, characterized in that the processor, when executing the computer program, implements the steps of the method of any of claims 1 to 7.

10. A computer-readable storage medium, on which a computer program is stored, which, when being executed by a processor, carries out the steps of the method of any one of claims 1 to 7.