KR102576449B1 - Device and method of image registration for 3d models - Google Patents

Abstract

According to the present invention, the image registration method for a 3D model is performed in an image registration device for a 3D model.
Here, the image registration method for the 3D model includes implementing a 3D model of the facility in a virtual environment; generating a virtual image for the 3D model through virtual photography in the virtual environment; Calculating an error between the virtual image and a captured image obtained by photographing an actual facility by a photographing device; and correcting the calculated error to match the virtual image and the captured image.
The present invention matches an image captured by a camera with a 3D model of the object to be photographed in a virtual environment, making it possible to accurately inspect and diagnose the object to be photographed through the 3D model.

Description

Image registration device and method for 3D model {DEVICE AND METHOD OF IMAGE REGISTRATION FOR 3D MODELS}

The present invention relates to an image registration device and method for a 3D model. More specifically, the present invention relates to an image captured by a camera and a 3D model of a subject implemented in a virtual environment, thereby enabling the subject to be captured through the 3D model. It relates to an image registration device and method for a 3D model that can accurately inspect and diagnose.

As one of the methods for facility management, including safety diagnosis and damage inspection, there is a method of obtaining and utilizing information related to the status and location of the facility using imaging equipment. Applying the information obtained through the above imaging equipment to a 3D model of the facility subject to management implemented in a virtual environment, etc. has the advantage of allowing detailed inspection of the facility subject to management easily even from a remote location.

Meanwhile, in situations or places where human access is not easy, detailed inspections of facilities can be performed by using unmanned aerial vehicles. This method of inspecting and managing facilities using unmanned aerial vehicles requires the ability to accurately determine the location of parts of the facility that appear in images captured by unmanned aerial vehicles, and requires technology to accurately identify minute damage to the facility. It becomes necessary.

Specifically, in order to acquire images from a imaging device mounted on an unmanned aerial vehicle and extract quantitative information about a facility from these images, the location where the image was captured and the posture of the camera at the time of shooting must be determined in three-dimensional space. Accordingly, a GNSS (Global Navigation Satellite System) signal receiving device and an IMU (Inertial Measurement Unit) are devices for measuring the position and attitude of an unmanned aerial vehicle (or an imaging device mounted thereon). may be used, and depending on convenience, GNSS/IMU that combines the functions of the two devices into one may be used.

However, in situations where GNSS signals are not received enough to determine the exact location of the unmanned aerial vehicle, difficulties may arise in determining the shooting location and attitude of the acquired image. For example, when an unmanned aerial vehicle is moved to the lower part of a bridge facility for a detailed inspection of the lower part of the bridge facility, reception of GNSS signals may not be smooth, and in this case, the acquired image and location information and attitude information for the image, etc. Since these do not match exactly, problems may arise where it is difficult to accurately match the acquired image to the facility implemented with the 3D model mentioned above.

In addition, in addition to problems related to GNSS signal reception, there is a problem that difficulties may arise in matching the acquired images even when the camera alignment state of the imaging equipment is not accurately determined or the shape of the implemented 3D model is not accurate. there is.

Korean Patent No. 10-2038127

The present invention was designed to solve the above-mentioned problems, and involves comparing a captured image obtained through a photographing device and a virtual image generated for a 3D model, calculating the error between the two images, and correcting the calculated error. Through this, accurate matching can be achieved.

According to one aspect of the present invention, the image registration method for a 3D model is performed in an image registration device for a 3D model. Here, the image registration method for the 3D model includes implementing a 3D model of the facility in a virtual environment; generating a virtual image for the 3D model through virtual photography in the virtual environment; Calculating an error between the generated virtual image and a captured image obtained by photographing an actual facility by a photographing device; and correcting the calculated error to match the virtual image and the captured image.

In one embodiment, generating a virtual image for the 3D model includes collecting shooting location and shooting posture information for each captured image acquired by the photographing device; and deriving a shooting position and posture in the virtual environment corresponding to the collected shooting location and posture information, and generating a virtual image for the three-dimensional model based on the derived shooting position and posture. May include ;.

In one embodiment, calculating the error may include extracting an outline of a facility included in the virtual image and the captured image.

In one embodiment, matching the virtual image and the captured image may include matching the virtual image and the captured image based on a projection transformation method using the extracted outline.

In one embodiment, calculating the error includes matching images whose shooting positions and shooting postures are determined to be the same among the virtual images and the captured images; and overlapping the matched images so that they are projected onto each other.

In one embodiment, matching the virtual image and the captured image may include matching the virtual image and the captured image by applying the projection transformation method to each of the overlapping images.

In one embodiment, matching the virtual image and the captured image based on a projection transformation method may include calculating transformation coordinates in the virtual environment for coordinates of the obtained captured image.

According to another aspect of the present invention, an image registration device for a 3D model performs an image registration method for a 3D model. Here, the image registration device for the 3D model includes a 3D model generator that implements a 3D model of the facility in a virtual environment; a virtual image generator that generates a virtual image of the three-dimensional model through virtual photography in the virtual environment; an error calculation unit that calculates an error between the generated virtual image and the captured image obtained by photographing an actual facility by a photographing device; and an image matching unit that corrects the calculated error and matches the virtual image and the captured image.

An image registration device and method for a 3D model according to an embodiment of the present invention accurately detects discrepancies between the virtual image of the implemented 3D model and the captured image to be matched, and corrects this to create a 3D image that is almost identical to the actual facility. It has the effect of implementing the model.

The image matching device and method for a 3D model according to an embodiment of the present invention can accurately match captured images to a 3D model for a facility, which has the effect of allowing easy detailed inspection of facilities subject to management even from a remote location. there is.

The image registration device and method for a 3D model according to an embodiment of the present invention automatically calculates and corrects the error between the virtual image and the captured image for the 3D model, so the operator's post-processing work for registration between images is separate. The advantage is that it is not necessary.

FIG. 1 is a reference diagram for explaining a photographing device 10 that can be used in an image registration device for a 3D model according to an embodiment of the present invention.
Figure 2 is a block diagram for explaining an image registration device for a 3D model according to an embodiment of the present invention.
3 to 5 are reference diagrams for explaining an image matching method for a 3D model according to an embodiment of the present invention.
Figure 6 is a flowchart illustrating an image registration method for a 3D model according to an embodiment of the present invention.

The present invention will be described in detail with reference to the attached drawings as follows. Here, repeated descriptions, known functions that may unnecessarily obscure the gist of the present invention, and detailed descriptions of configurations are omitted. Embodiments of the present invention are provided to more completely explain the present invention to those skilled in the art. Accordingly, the shapes and sizes of elements in the drawings may be exaggerated for clearer explanation.

Throughout the specification, when a part is said to “include” a certain component, this means that it may further include other components rather than excluding other components, unless specifically stated to the contrary.

Additionally, the term "... unit" used in the specification refers to a unit that processes one or more functions or operations, and may be implemented as hardware, software, or a combination of hardware and software.

FIG. 1 is a reference diagram for explaining a photographing device 10 that can be used in an image registration device for a 3D model according to an embodiment of the present invention.

Referring to FIG. 1, the imaging device 10 captures an image of a facility subject to detailed inspection, diagnosis, or management in the image registration method for a three-dimensional model according to an embodiment of the present invention and obtains image information. The device may include an image sensor that converts optical signals input through a provided lens into electronic signals, or may refer to a device capable of performing an equivalent function. In FIG. 1, the photographing device 10 is shown as an unmanned air vehicle as a preferred example of the present invention, and the photographing device 10 in the form of an unmanned air vehicle uses GNSS/IMU and photographing to obtain location and attitude information of the photographing device 10. It may be configured to include a camera mounted on the device 10 and a gimbal for fixing the camera.

Additionally, the imaging device 10 may store the acquired image information in an internal storage unit or transmit it to an image registration device for a 3D model or another server in real time through a wireless network. Additionally, the operation of the imaging device 10 may be controlled by various methods including wireless communication control.

Figure 2 is a block diagram showing the configuration of an image registration device 100 for a 3D model according to an embodiment of the present invention.

Referring to Figure 2, the facility precision inspection system 100 according to an embodiment of the present invention includes a 3D model generator 110, a virtual image generator 120, an error calculation unit 130, and an image matching unit ( 140).

The 3D model generator 110, virtual image generator 120, error calculation unit 130, and image matching unit 140 each execute commands such as computer programs or data stored in a computer-readable medium. Various steps described herein may be performed by including one or more processors, memories, and other suitable components to perform the following.

For example, computer programs or instructions may be stored on a medium readable by a computer internal or external to the device.

The 3D model generator 110 can implement a 3D model of a facility in a virtual environment.

Here, the virtual environment may mean a virtual space where a user can implement a 3D model of the facility. For example, the virtual environment can be arbitrarily created through the execution of specific software that can be installed on the user terminal, and may mean a three-dimensional space that is visually expressed through the display device of the user terminal. Accordingly, the 3D model generator 110 may implement a 3D model of the facility in the virtual environment based on information about the 3D model of the facility previously formed by the user.

The virtual image generator 120 may generate a virtual image for the three-dimensional model through virtual photography in the virtual environment.

In one embodiment, the virtual image generator 120 may collect shooting location and shooting posture information for each captured image acquired by the photographing device 10.

For example, the virtual image generator 120 may directly receive and collect location and posture information when photographing the photographing device 10 from the GNSS/IMU of the photographing device 10, and on the other hand, the virtual image generator 120 may collect Location and posture information at the time of shooting can be received and collected from the storage device included in (10).

In one embodiment, the virtual image generator 120 derives a shooting position and a shooting posture in the virtual environment corresponding to the collected shooting location and shooting posture information, and based on the derived shooting position and shooting posture A virtual image for the 3D model can be generated.

For example, the virtual image generator 120 collects shooting position and shooting posture information for each captured image acquired by the photographing device 10, thereby determining the shooting position in the virtual environment for each acquired captured image. And the shooting posture can be derived, and by performing virtual shooting in a virtual environment based on the derived shooting position and posture, a virtual image similar to the captured image captured by the photographing device 10 can be generated.

The error calculation unit 130 may calculate an error between the generated virtual image and the captured image obtained by photographing the actual facility by the photographing device 10. Here, even if the virtual image generator 120 generates a virtual image similar to the captured image captured by the photographing device 10 as described above, a situation occurs in which it is difficult to determine the location and posture through GNSS/IMU, etc. There is a possibility that errors may occur for a variety of reasons, including the case where errors occur, and in order to correct and resolve these errors, it is necessary to first accurately calculate the error between the virtual image and the acquired captured image.

In one embodiment, the error calculation unit 130 may match images that are determined to have the same shooting position and shooting posture among the virtual image and the captured image, and overlap the matched images so that they are projected onto each other.

In this regard, Figures 3 and 4 are reference diagrams for explaining an image registration method for a 3D model according to an embodiment of the present invention.

Here, Figure 3 (a) shows one of the captured images of the facility captured by the photographing device 10 according to an embodiment of the present invention, and Figure 3 (b) shows the photographing device 10. This shows a virtual image created by applying the shooting position and posture to the image (a) captured by . Additionally, Figure 4 shows an example in which the above images (a) and (b) are overlapped so that they are projected onto each other.

As shown in FIG. 4, the error calculation unit 130 can calculate the error for each of the matched images by overlapping and comparing each of the matched images.

In one embodiment, the error calculation unit 130 may extract outlines of facilities included in the virtual image and the captured image. In this regard, Figure 5 shows an example of an outline extracted through an image matching device for a 3D model of the present invention. Here, the error calculation unit 130 can be used to calculate the error between the overlapping images by extracting the outline of the facility included in each image. For example, the error calculation unit 130 calculates the error by comparing the outline of the captured image and the outline of the virtual image in the overlapping images, derives feature points for the outline of each image, and calculates the error for each image. The error can be calculated through comparison of coordinates between feature points.

The image matching unit 140 may correct the calculated error and match the virtual image and the captured image.

In one embodiment, the image registration unit 140 may calculate transformation coordinates in the virtual environment for the coordinates of the acquired captured image.

The image matching method for a 3D model of the present invention calculates transformation coordinates in the virtual environment for the coordinates of the acquired captured image, thereby allowing the acquired captured image to be compared with the virtual image on the same basis. Here, Equation 1 below is used to convert the coordinates of the acquired captured image into coordinates in a virtual environment, and various variables can be used in this conversion process.

[Equation 1]

(From here, is the transformation coordinate for the captured image, s is the scale, r is the y-axis rotation angle, p is the x-axis rotation angle, h is the z-axis rotation angle, is the x-axis movement distance, is the y-axis movement distance, and are the coordinates of the captured image before conversion, respectively)

In one embodiment, the image matching unit 140 may match the virtual image and the captured image based on a projection transformation method using the extracted outline.

In one embodiment, the image registration unit 140 may apply the projection transformation method to each of the overlapping images to match the virtual image and the captured image.

The image registration method for a 3D model of the present invention can perform correction to minimize errors occurring between images based on a projection transformation method using the extracted outline. Specifically, for example, the image registration unit 140 derives random feature points for the extracted outline and applies a weight matrix calculated from the slope of the outline including the feature point to the feature points, thereby determining the location between feature points. Errors can be minimized.

Figure 6 is a flowchart illustrating an image registration method for a 3D model according to an embodiment of the present invention.

Referring to FIG. 6, the user can acquire a captured image of the facility subject to management through a recording device, and create a three-dimensional model of the facility subject to management through specific software, etc., and place it in a virtual environment. It can be implemented (S601).

Next, the virtual image generator may generate a virtual image by photographing a facility implemented in the virtual environment to correspond to the location and posture information of the acquired captured image (S602).

Next, the error calculation unit may calculate the error between the generated virtual image and the previously acquired captured image (S603).

Next, the virtual image and the captured image can be matched by correcting the calculated error through the image matching unit. Through this, the user can inspect and manage the actual facility using the 3D model without a separate post-processing process. (S604).

The image registration method for the above-described 3D model was explained with reference to the flow chart presented in the drawing. For simplicity of illustration, the method is shown and described as a series of blocks; however, the invention is not limited to the order of the blocks, and some blocks may occur simultaneously or in a different order than shown and described herein with other blocks. Various other branches, flow paths, and sequences of blocks may be implemented that achieve the same or similar results. Additionally, not all blocks shown may be required for implementation of the methods described herein.

Although the present invention has been described above with reference to preferred embodiments, those skilled in the art may make various modifications and changes to the present invention without departing from the spirit and scope of the present invention as set forth in the claims below. You will understand that you can do it.

10: Filming device
100: Image registration device for 3D model
110: 3D model creation unit
120: Virtual image creation unit
130: Error calculation unit
140: Image matching unit

Claims (8)

In an image registration method for a 3D model performed in an image registration device for a 3D model,
Implementing a three-dimensional model of a facility in a virtual environment;
generating a virtual image for the 3D model through virtual photography in the virtual environment;
A type of unmanned aerial vehicle that includes a GNSS (Global Navigation Satellite System)/IMU (Inertial Measurement Unit) to acquire location and attitude information of the imaging device, a camera, and a gimbal to fix the camera. A step in which a photographing device photographs an actual facility;
Calculating an error between the generated virtual image and a captured image obtained by photographing an actual facility by a photographing device; and
Comprising: correcting the calculated error to match the virtual image and the captured image,
Generating a virtual image for the three-dimensional model;
Directly receiving location and posture information of the photographing device from the GNSS/IMU of the photographing device and collecting photographing position and posture information for each captured image acquired by the photographing device; and
By deriving a shooting position and shooting posture in the virtual environment corresponding to the collected shooting location and shooting posture information, and performing virtual shooting in the virtual environment based on the derived shooting position and shooting posture, the three-dimensional model Including; generating a virtual image for,
The step of calculating the error is,
Matching images whose shooting positions and shooting postures are determined to be the same among the virtual images and the captured images; and
Characterized in that it includes a step of overlapping the matched images so that they are projected onto each other,
Image registration method for 3D models.
delete According to paragraph 1,
The step of calculating the error is,
Characterized in that it includes; extracting the outline of the facility included in the virtual image and the captured image,
Image registration method for 3D models.
According to paragraph 3,
Matching the virtual image and the captured image;
A step of matching the virtual image and the captured image based on a projection transformation method using the extracted outline,
Image registration method for 3D models.
delete According to paragraph 4,
Matching the virtual image and the captured image;
Characterized in that it comprises a step of matching the virtual image and the captured image by applying the projection transformation method to each of the overlapped images.
Image registration method for 3D models.
According to paragraph 4,
Matching the virtual image and the captured image based on a projection transformation method;
Characterized in that it includes; calculating transformation coordinates in the virtual environment for the coordinates of the acquired captured image,
Image registration method for 3D models.
In an image registration device for a 3D model that performs an image registration method for a 3D model,
A 3D model creation unit that implements a 3D model of the facility in a virtual environment;
a virtual image generator that generates a virtual image of the three-dimensional model through virtual photography in the virtual environment;
GNSS (Global Navigation Satellite System)/IMU (Inertial Measurement Unit) for acquiring location and attitude information of the generated virtual image and imaging device, a camera, and a gimbal for fixing the camera. An error calculation unit that calculates an error between captured images obtained by photographing an actual facility by a photographing device in the form of an unmanned aerial vehicle, including; and
An image matching unit that corrects the calculated error and matches the virtual image and the captured image,
The virtual image generator,
Directly receives location and posture information when shooting of the photographing device from the GNSS/IMU of the photographing device, collects shooting location and posture information for each captured image acquired by the photographing device, and collects the collected shooting location and shooting posture Generating a virtual image for the three-dimensional model by deriving a shooting position and shooting posture in the virtual environment corresponding to posture information and performing virtual shooting in the virtual environment based on the derived shooting position and shooting posture; ,
The error calculation unit,
Characterized in that among the virtual images and captured images, images whose shooting positions and shooting postures are determined to be identical are matched, and the matched images are overlapped so that they are projected onto each other.
Image registration device for 3D models.