KR20200145410A - Apparatus and method for obtaining location information for camera of vehicle - Google Patents

Abstract

According to one embodiment of the present invention, a method for obtaining camera location information of a vehicle may comprise the steps of: matching feature points of a plurality of surrounding images acquired at a plurality of viewpoints by a camera provided in the vehicle; acquiring displacement information of the vehicle based on driving information of the vehicle detected by a sensor provided in the vehicle; and acquiring location information of the camera by fusing the acquired displacement information of the vehicle and matching information for the feature point. Accordingly, a more accurate camera location information can be obtained, and based on this, the accuracy of the estimated location of the vehicle can be improved.

Description

Device and method for acquiring vehicle camera location information {APPARATUS AND METHOD FOR OBTAINING LOCATION INFORMATION FOR CAMERA OF VEHICLE}

The present invention relates to an apparatus and method for obtaining camera location information of a vehicle for obtaining location information of a camera mounted on a driving vehicle from an image obtained by a camera of the vehicle.

In general, a vehicle refers to a transportation device that travels on a road or track using fossil fuel, electricity, or the like as a power source.

Vehicles have been developed to provide various functions to drivers according to the development of technology. In particular, according to the trend of electrification of vehicles, vehicles having an active safety system (ASS) that operate to prevent accidents just before or at the moment of accidents have appeared.

Furthermore, in recent years, an advanced driver assistance system (ADAS: Advanced Driver Assist System) that actively provides information on the driving environment such as vehicle status, driver status, and surrounding environment in order to reduce the burden on the driver and improve convenience. Research on this equipped vehicle is actively underway.

Since the advanced driver assistance system operates according to the driving environment determined from the current position, the position of the own vehicle may be estimated in advance. At this time, the surrounding image acquired by the camera mounted on the vehicle may be used, and in order to effectively utilize the surrounding image, it is necessary to secure accurate location information of the camera.

Korean Registered Patent Publication, No. 10-1584693 (announcement on January 14, 2016)

The problem to be solved by the present invention is to provide an apparatus and method for acquiring camera position information of a vehicle by fusing matching information for a plurality of surrounding images and displacement information of a vehicle acquired by a sensor of a vehicle to obtain camera position information. Is to do.

However, the problem to be solved by the present invention is not limited to those mentioned above, and another problem to be solved that is not mentioned may be clearly understood by those of ordinary skill in the art to which the present invention belongs from the following description. will be.

According to an embodiment of the present invention, a method of obtaining camera location information of a vehicle includes: matching feature points of a plurality of surrounding images acquired at a plurality of viewpoints by a camera provided in the vehicle; Obtaining displacement information of the vehicle based on driving information of the vehicle detected by a sensor provided in the vehicle; And acquiring location information of the camera by fusing the obtained displacement information of the vehicle and matching information for the feature point.

According to an embodiment of the present invention, an apparatus for obtaining camera location information of a vehicle includes: a matching unit for matching feature points of a plurality of surrounding images acquired at a plurality of viewpoints by a camera provided in the vehicle; A displacement information acquisition unit that acquires displacement information of the vehicle based on driving information of the vehicle detected by a sensor provided in the vehicle; And a fusion unit that acquires location information of the camera by fusing the obtained displacement information of the vehicle and matching information on the feature point.

In the apparatus and method for obtaining camera location information of a vehicle according to an embodiment of the present invention, it is possible to more accurately obtain the location information of the camera, and based on this, it is possible to increase the accuracy of the estimated location of the vehicle. Specifically, by fusing matching information for a plurality of surrounding images and displacement information of the vehicle obtained by the sensor of the vehicle, the scale of the camera's position information based on the surrounding images is restored, and the bias of the sensor is also corrected, The three-dimensional position of the vehicle can be accurately estimated.

1 and 2 are functional block diagrams of a system for obtaining camera location information of a vehicle according to various embodiments of the present invention.
3 is a flowchart of a method of obtaining camera location information of a vehicle according to an embodiment of the present invention.
4 is a diagram illustrating a light flow obtained by a matching unit according to an embodiment of the present invention.
5 is a diagram for explaining an offset between a vehicle center and a camera according to an embodiment of the present invention.
6 is a diagram illustrating camera trajectories having different scales satisfying a cost function according to an embodiment of the present invention.

Advantages and features of the present invention, and a method of achieving them will become apparent with reference to the embodiments described below in detail together with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below, but may be implemented in a variety of different forms, and only these embodiments make the disclosure of the present invention complete, and are common knowledge in the technical field to which the present invention pertains. It is provided to completely inform the scope of the invention to those who have, and the invention is only defined by the scope of the claims.

In describing the embodiments of the present invention, if it is determined that a detailed description of a known function or configuration may unnecessarily obscure the subject matter of the present invention, a detailed description thereof will be omitted. In addition, terms to be described later are terms defined in consideration of functions in an embodiment of the present invention, which may vary according to the intention or custom of users or operators. Therefore, the definition should be made based on the contents throughout this specification.

1 and 2 are functional block diagrams of a system for obtaining camera location information of a vehicle according to various embodiments of the present invention.

The system for obtaining camera location information of a vehicle of the present invention refers to any system capable of obtaining location information of a camera using surrounding images obtained by the camera of the vehicle.

Autonomous vehicles need to recognize real-time location using a pre-built HD map, plan/change routes and control vehicles according to the surrounding environment. One of the ways to build a precision map is to use a MMS (Mobile Mapping System) vehicle equipped with a high-performance sensor (RTK GPS, INS, LiDAR, and camera) to drive the actual road driving environment while using 3D point cloud data. It is possible to construct a high-precision map by expressing meaningful data such as lanes, stop lines, signs, and milestones among the acquired and acquired point cloud data in a vector format. Since this method uses an MMS vehicle equipped with a high-performance sensor, when some sections are changed, it may be difficult to recognize the change detection section. In addition, in order to update the changed section, it is necessary to drive the section with an MMS vehicle to obtain 3D point cloud data again, and to perform the task of creating a vector map again, which can require a lot of cost and time. .

To solve this problem, methods of generating a map having an accuracy similar to that of a precision map by collecting and fusing inaccurate maps generated by a plurality of vehicles in a server have been proposed. In order to implement such a method, it is necessary for the vehicle to accurately estimate its own location without the aid of a precision map. As one method, there is a method of fusion of vehicle wheel speed information and yaw rate information to estimate the relative attitude angle and position of the vehicle (hereinafter referred to as odometry). However, since this method assumes motion on a plane, it is difficult to use for 3D map generation, and it may be difficult to estimate the scaling factor of the wheel speed sensor. In addition, since the yaw rate sensor includes a bias, if this is not considered, the attitude angle and position accuracy of the vehicle may be significantly lowered.

As another method, Monocular Visual Odometry technology using a monocular camera is economical compared to using a high-performance sensor, can estimate a three-dimensional attitude angle and position, and can be easily applied to any vehicle. However, this method requires extraction and tracking of image feature points, and it is difficult to use in real time due to a large amount of computation, and unlike a method using separate sensors such as multiple cameras, LiDAR, and GPS, It is difficult to know the actual scale for the size of the space, so it may have scale ambiguity.

Therefore, in order to improve the above-described problems, the system 1 for obtaining camera location information of the vehicle V according to an embodiment of the present invention is obtained by matching information for a plurality of surrounding images and a sensor of the vehicle V. The position information of the camera may be obtained by fusing the displacement information of the vehicle V. The camera location information obtained in this way may be used for location information of the own vehicle, that is, odometry estimation of the vehicle V.

Before describing the configuration of the system 1 for obtaining camera position information of the vehicle V according to an embodiment of the present invention, variables and coordinate systems in the present invention are defined.

, 3차원 방향각(pitch, yaw, roll)를 각각

로 표시하고, 요레이트 바이어스 값을

로 표기한다. 키프레임은 연속되어 입력되는 복수의 주변 영상들 중 오도메트리 계산에 중요하다고 판단되는 영상으로

번째 키프레임에서의 변수

는

로 표시한다. 또

번째 키프레임과

번째 키프레임 사이의

번째 센서 데이터

는

로 표시한다. (

은

을 의미한다.)The present invention assumes a camera, specifically a vehicle (V) (or a robot) equipped with a monocular camera and a sensor as a rigid body, and the state variable of the camera odometry is a three-dimensional posture (6 degrees of freedom) of the camera. It consists of a total of 7 degrees of freedom of the yaw rate bias (1 degree of freedom) of the attitude sensor and the three-dimensional position of the camera.

, 3D direction angle (pitch, yaw, roll) respectively

And the yaw rate bias value

Marked as. A key frame is an image judged to be important for odometry calculation among a plurality of consecutively inputted surrounding images.

Variable at the first keyframe

Is

Denoted by In addition

First keyframe and

Between keyframes

Second sensor data

Is

Denoted by (

silver

Means.)

In addition, all coordinate systems of the present invention basically follow an image coordinate system in which the right direction of the camera is the X axis, the downward direction of the camera is the Y axis, and the front of the camera is the Z axis.

는

, 키프레임 차량(V) 좌표계에서 표현된 변수

는

와 같이 표현한다. 또한 k번째 카메라 좌표계 기준으로 표현된 k+1번째 카메라 좌표계의 변수

는

와 같이 표현한다. 전역 좌표계 기준으로 표현된 k번째 카메라 좌표계의

는

와 같이 표현한다.In addition, the present invention largely defines and uses a (video) odometry coordinate system, a key frame camera coordinate system, and a key frame vehicle (V) coordinate system. The odometry coordinate system expresses the current position and posture of the camera using the starting position of the camera as the origin and the coordinate system. The key frame camera coordinate system expresses the position and posture of the current camera by using the position and posture of the camera in which the key frame is captured as the origin and coordinate system. Similarly, the key frame vehicle (V) coordinate system uses the position and posture of the vehicle (V), not the camera, as the origin and coordinate system. The center of the vehicle V is the center point of the rear axle. Variables not indicated separately are based on the odometry coordinate system, and are expressed in the keyframe camera coordinate system.

Is

, Variable expressed in the key frame vehicle (V) coordinate system

Is

Expressed like this. Also, a variable of the k+1st camera coordinate system expressed as a reference to the kth camera coordinate system

Is

Expressed like this. Of the kth camera coordinate system expressed in terms of the global coordinate system

Is

Expressed like this.

Based on the above-described premise, the following describes the configuration of the system 1 for obtaining camera location information of the vehicle V according to an embodiment of the present invention.

Referring to FIG. 1, a system 1 for obtaining camera position information of a vehicle V according to an embodiment of the present invention includes a vehicle V equipped with a camera and a sensor; And an apparatus 100 for obtaining camera location information of the vehicle V that obtains camera location information of the vehicle V.

The system 1 for obtaining camera location information of the vehicle V may be provided to enable wired or wireless communication between the apparatus 100 for obtaining camera location information of the vehicle V and the vehicle V. Specifically, the system 1 for obtaining camera location information of the vehicle V according to an embodiment adopts a CAN communication method to provide a wired communication environment between the apparatus 100 for obtaining camera location information of the vehicle V and the vehicle V. Can be implemented. In contrast, the system 1 for obtaining camera location information of the vehicle V according to another embodiment provides an environment in which information exchange between the information providing vehicle V and the location estimation device is possible by adopting various known communication methods. I can. The system 1 for obtaining camera location information of the vehicle V according to an embodiment uses a known communication method such as CDMA, GSM, W-CDMA, TD-SCDMA, WiBro, LTE, EPC, etc. to communicate via a base station. You can provide an environment. In contrast, the system 1 for obtaining camera location information of the vehicle V according to another embodiment includes a wireless LAN, Wi-Fi, Bluetooth, Zigbee, and WFD (Wi-Fi). Fi Direct), UWB (Ultra wideband), infrared communication (IrDA; Infrared Data Association), BLE (Bluetooth Low Energy), NFC (Near Field Communication). It is also possible to provide an environment in which communication between position estimation devices is possible. However, the communication method adopted by the system 1 for obtaining camera location information of the vehicle V is not limited to the above-described embodiment.

The vehicle V may be provided with a sensing means for sensing the surrounding environment. The sensing means according to an embodiment is a radar that detects a driving environment by irradiating a pulse around the vehicle V and receiving an echo pulse reflected from an object located in a corresponding direction, and a laser around the vehicle V. And/or a LiDAR that receives a laser reflected from an object located in a corresponding direction, and/or an ultrasonic wave is irradiated around the vehicle (V), and an echo ultrasound reflected from an object located in the corresponding direction is received. It may include an ultrasonic sensor and the like.

Further, the vehicle V may include a camera as a sensing means. The camera is provided to face the front, side, and/or rear of the vehicle V, and the surrounding images in the corresponding direction may be photographed. The captured surrounding image may be used to obtain camera location information through an image processing process.

In addition, the vehicle V may include a GPS module as a sensing means. The GPS module may receive a satellite signal including navigation data from at least one Global Position System (GPS) satellite. The vehicle V may obtain the coordinates of the current location of the vehicle V providing GPS-based information based on a satellite signal.

In addition, the vehicle V may include a sensor that detects driving information. The sensor according to an embodiment may include a wheel speed detection sensor that senses a wheel speed of the vehicle V, and a pose sensor that senses attitude information including a yaw rate of the vehicle V.

In addition, the vehicle V may further include a storage means for storing a program including a command for performing a method of obtaining camera location information of the vehicle V according to an embodiment of the present invention. The storage means according to an embodiment is a flash memory type, a hard disk type, a multimedia card micro type, a card type memory (for example, SD or XD memory, etc.). ), RAM (Random Access Memory, RAM), SRAM (Static Random Access Memory), ROM (Read-Only Memory, ROM), EEPROM (Electrically Erasable Programmable Read-Only Memory), PROM (Programmable Read-Only Memory), Magnetic Memory, magnetic disk, and/or optical disk.

In addition, the vehicle V may further include a communication module capable of communicating with the apparatus 100 for obtaining camera location information of the vehicle V according to a communication method adopted by the system 1 for obtaining camera location information of the vehicle V. . Through the communication module, the vehicle V may provide the surrounding image and driving information to the apparatus 100 for obtaining camera location information of the vehicle V.

The apparatus 100 for obtaining camera location information of the vehicle V acquires camera location information of the vehicle V based on the surrounding images and driving information received from the vehicle V, and finally, the vehicle V You can estimate the location of To this end, the apparatus 100 for obtaining camera location information of the vehicle V according to an exemplary embodiment may be configured with a server or a set of a plurality of servers.

The apparatus 100 for obtaining camera location information of the vehicle V may receive surrounding images and driving information from the vehicle V. To this end, the apparatus 100 for obtaining camera location information of the vehicle V may be provided to enable communication according to a communication method adopted by the system 1 for obtaining camera location information of the vehicle V.

The apparatus 100 for obtaining camera location information of the vehicle V may include a matching unit 110, a displacement information obtaining unit 120, and a fusion unit 130.

The matching unit 110 may match feature points of a plurality of surrounding images acquired at a plurality of viewpoints by a camera provided in the vehicle V. Specifically, the matching unit 110 may extract a feature point from each of a plurality of surrounding images. Then, the matching unit 110 may obtain matching information including an optical flow of the matched feature points by matching feature points in adjacent surrounding images. Finally, the matching unit 110 may exclude a light flow for invalid matching based on an epipolar line.

The displacement information acquisition unit 120 may obtain displacement information of the vehicle V based on driving information of the vehicle V detected by a sensor provided in the vehicle V. Specifically, the displacement information acquisition unit 120 acquires the translational speed of the vehicle V based on the average of the speeds of the rear wheels detected by the wheel speed sensor, and based on the yaw rate value obtained by the attitude sensor, the vehicle ( V) rotation speed can be obtained.

The fusion unit 130 may acquire location information of the camera by fusing the obtained displacement information of the vehicle V and matching information on the feature points. Specifically, the fusion unit 130 may obtain the location information of the camera through a process of minimizing a predetermined cost function, and this cost function is used for odometry calculation, and matching information for feature points and vehicle displacement It may be designed to take information into account.

At least one configuration of the apparatus 100 for obtaining camera location information of the vehicle V according to the exemplary embodiment of FIG. 1 may be implemented as a computing device including a microprocessor. For example, at least one of each of the components of the apparatus 100 for obtaining camera location information of the vehicle V according to an embodiment is an application processor (AP), a communication processor (CP), a graphic processing unit (GPU), and/ Alternatively, it may be implemented by at least one of various processors such as a CPU (Central Processing Unit). In addition, it may be possible to implement at least two of each component of the apparatus 100 for obtaining camera location information of the vehicle V as a system on chip (SOC).

1 illustrates a case in which the device 100 for obtaining camera location information of the vehicle V is implemented as a device separate from the vehicle V, but unlike this, the device 100 for obtaining camera location information of the vehicle V is a vehicle It may be provided integrally with (V). Referring to FIG. 2, the apparatus 100 for obtaining camera location information of the vehicle V may be provided as a component of the vehicle V and may be included in the vehicle V.

So far, each configuration of the system 1 for obtaining camera position information of the vehicle V has been described. Hereinafter, a method of obtaining camera location information of the vehicle V performed by the apparatus 100 for obtaining camera location information of the vehicle V will be described.

3 is a flowchart of a method of obtaining camera location information of a vehicle according to an embodiment of the present invention.

First, the apparatus 100 for obtaining camera location information of the vehicle V may match feature points of a plurality of surrounding images acquired by the camera of the vehicle V (S100). Specifically, the matching unit 110 of the apparatus 100 for obtaining camera location information of the vehicle V may extract a feature point from each of a plurality of surrounding images. Then, the matching unit 110 may obtain matching information including an optical flow of the matched feature points by matching feature points in adjacent surrounding images. Finally, the matching unit 110 may exclude a light flow for invalid matching based on an epipolar line.

4 is a diagram illustrating a light flow obtained by a matching unit according to an embodiment of the present invention.

The matching unit 110 may extract feature points between two adjacent images from among a plurality of surrounding images and match the same feature points. For example, the matching unit 110 extracts GFTT (Good-Feature-to-Track) feature points from any one surrounding image, calculates the Lukas-Kanade light flow to the next surrounding image, and calculates two consecutive pairs of images. The feature points between them can be matched.

In addition, the matching unit 110 may determine the validity of the light flow for matching according to the epipolar constraint according to the epipolar line, and exclude the invalid light flow from the matching information. have. Here, since it is a known technique to determine the effectiveness of the light flow according to the epipolar constraint condition, a detailed description will be omitted. FIG. 4 illustrates the matching of two peripheral image feature points acquired by GFTT and Lukas-Kanade light flow, and a result of removing invalid matching using an epipolar constraint.

Referring back to FIG. 3, the apparatus 100 for obtaining camera location information of the vehicle V may obtain displacement information of the vehicle V based on driving information of the vehicle V detected by a sensor (S110). ). In this case, the displacement information may include a translational displacement and a rotational displacement of the vehicle V.

Specifically, the displacement information acquisition unit 120 of the apparatus 100 for obtaining camera position information of the vehicle V acquires the translational speed of the vehicle V based on the average speed of the rear wheels detected by the wheel speed sensor, The rotational speed of the vehicle V may be obtained based on the yaw rate value obtained by the attitude sensor.

The displacement information acquisition unit 120 may obtain the translational speed V of the vehicle V using the speeds V _RL and V _RR of the two rear wheels detected by the wheel speed detection sensor among sensors provided in the vehicle V. The translational speed V of the vehicle V can be calculated as the average V=(V _RL +V _RR )/2 of the speeds of the two rear wheels according to the kinematic model of the differential drive based on the center point of the rear wheel axis. .

In addition, the displacement information acquisition unit 120 may obtain the rotational speed of the vehicle V by using the yaw rate value ω detected by the attitude sensor among sensors provided in the vehicle V. At this time, since the value sensed by the attitude sensor includes the bias b of the attitude sensor itself, the rotational speed of the vehicle V can be calculated as ω-b excluding this.

In addition, the displacement information acquisition unit 120 An initial bias value of the attitude sensor may be estimated by averaging the attitude sensor values when the vehicle V is in a stopped state. However, when the vehicle V is placed on an inclined surface, an incorrect value may be estimated, and the bias may change even while the vehicle V is moving, so this may be corrected later.

번째 주변 영상의 차량(V) 좌표계를 기준으로 이후

번째까지 얻은

개의 센서 값을 이용하여 2차원 변위 정보인

를 획득할 수 있다.After obtaining the translational speed V and the translational displacement ω, the displacement information acquisition unit 120 may discretely integrate them to obtain 2D translational information of the vehicle V. Specifically, the displacement information acquisition unit 120 according to Equation 1

From the vehicle (V) coordinate system of the surrounding image

Obtained until th

2D displacement information,

Can be obtained.

번째 주변 영상에서의 값을 이용하고,

는 내부센서의 데이터가 들어오는 시간 간격을 의미할 수 있다.Here, the sensor bias is

Using the value from the surrounding image,

May mean the time interval in which the data of the internal sensor comes in.

을 보정하기 위한 요레이트 적분값

을 수학식 2에 따라 구할 수 있다.In addition, the displacement information acquisition unit 120 will

Yaw rate integral value for correcting

Can be obtained according to Equation 2.

After acquiring the displacement information of the vehicle V, the apparatus 100 for obtaining camera location information of the vehicle V may acquire the location information of the camera by fusing the displacement information of the vehicle V and matching information on the feature points. Yes (S120). Specifically, the fusion unit 130 of the apparatus 100 for obtaining camera location information of the vehicle V may obtain the location information of the camera through a process of minimizing a predetermined cost function, and this cost function is used to calculate odometry. It is used for, and may be designed to consider matching information for feature points and displacement information of a vehicle.

번째 주변 영상의 차량(V) 좌표계를 기준으로 한 정보이다. 따라서, 융합부(130)는 카메라의 위치 정보를 획득하는 과정에서, 이러한 변위 정보를 카메라 좌표계 기준의 3차원 변위 정보로 변환할 수 있다. Here, the displacement information obtained by the displacement information acquisition unit 120 is

This is information based on the vehicle (V) coordinate system of the second surrounding image. Accordingly, the fusion unit 130 may convert the displacement information into 3D displacement information based on the camera coordinate system in the process of obtaining the location information of the camera.

Meanwhile, in the case of S100 and S110 shown in FIG. 3, S110 may be performed after S100 is performed, or may be performed in the reverse order, and S100 and S110 may be performed in parallel according to an embodiment.

5 is a diagram for explaining an offset between a vehicle center and a camera according to an embodiment of the present invention.

라 할 수 있다. 오프셋

을 고려하여, 융합부(130)는 2차원 변위 정보로부터 주변 영상의 카메라 좌표계를 기준으로 하는 3차원 위치

와 자세각

을 제 2 위치 정보로서 구할 수 있으며, 수학식 3을 따른다. 이 때, 카메라가 전방으로 정렬되어 있고, 강체인 차량(V)에 고정되어 있기 때문에, 카메라와 차량(V)의 방향각이 동일함을 전제한다.When the camera C is aligned in the front direction of the vehicle V as shown in FIG. 5, the offset between the center P _{v of the} vehicle V and the camera position P _c is determined.

Can be said. offset

In consideration of, the fusion unit 130 is a three-dimensional position based on the camera coordinate system of the surrounding image from the two-dimensional displacement information.

And posture angle

Can be obtained as the second location information, and Equation 3 is followed. At this time, it is assumed that the camera and the vehicle V have the same direction angle because the camera is aligned to the front and is fixed to the rigid vehicle V.

In addition, in order to obtain the location information of the camera, the fusion unit 130 converts the 3D coordinates of the feature points acquired based on the matching result for each of the plurality of surrounding images, and between the 2D coordinates and the feature points in each of the surrounding images. It is possible to obtain location information that minimizes the distance. For example, the fusion unit 130 may acquire location information based on bundle adjustment. Specifically, the fusion unit 130 uses an algorithm such as Levenberg-Marquardt or Gauss-Newton to minimize the cost function of Equation 4, the camera position and attitude angle for the surrounding image, and the solution to the three-dimensional position of the feature point. Can be obtained.

은 bundle adjustment 최적화를 수행하는 주변 영상의 개수를 의미하고,

은 최근

에서 획득된 특징점 중 카메라의 자세 정보 획득에 사용되는 3차원의 표식점의 개수를 의미하고.

는 전역 좌표계 상의

번째 3차원 표식점의 위치(단위: 미터)를 의미하고,

는

번째 3차원 표식점이

번째 주변 영상에서 관찰되었을 때의 주변 영상에서의 위치(단위: 픽셀)를 의미하고,

는

번째 3차원 표식점이

번째 주변 영상에서 관찰되었는지(Visibility) 여부를 나타내는 항이고, 함수

는 최적화 과정에서 이상치(Outlier)의 영향을 줄이기 위한 손실함수(Loss Function)로서 일반적으로 Huber 커널을 사용하며, 함수

는 동치 좌표계(Homogeneous Coordinate)으로 표현된 두 점 사이의 직선 거리를 계산하는 함수이고,

는 초점 거리와 같은 카메라의 내부 파라미터 정보를 담고 있는 카메라 행렬(Camera Matrix)를 의미할 수 있다. here,

Is the number of surrounding images that perform bundle adjustment optimization,

Is recently

It means the number of three-dimensional marker points used to acquire camera attitude information among the feature points acquired in.

Is in the global coordinate system

Refers to the location of the third 3D marker point (unit: meter),

Is

3D marker point

Refers to the position (unit: pixel) in the surrounding image when observed in the 1st surrounding image,

Is

3D marker point

Is a term indicating whether or not it is observed in the surrounding image (Visibility), and a function

Is a loss function to reduce the influence of outliers in the optimization process. Generally, Huber kernel is used.

Is a function that calculates the linear distance between two points expressed in the homogeneous coordinate system,

May mean a camera matrix containing information on internal parameters of a camera such as a focal length.

와

는 각각 전역 좌표계에서

번째 카메라에 의해 촬영된 주변 영상 좌표계로의 좌표 변환에 대한 회전과 변위를 의미할 수 있다. 두 값은

번째 주변 영상이 촬영된 카메라의 자세각

및 위치

와 수학식 5의 관계를 가질 수 있다.Also, in Equation 4

Wow

Each in the global coordinate system

It may mean rotation and displacement for the coordinate transformation into the surrounding image coordinate system photographed by the second camera. The two values are

Angle of the camera where the first surrounding image was taken

And location

And Equation 5.

According to the cost function of Equation 4, the fusion unit 130 uses the location, the posture angle, and the position of the 3D mark point where the surrounding image given to the M-1th surrounding image in the previous process is captured, Not only the mark point, but also the position and attitude angle of the newly added M-th surrounding image can be corrected through the above-described cost function.

In this case, in Equation 4 described above, there may be a plurality of factors that lower the accuracy of obtaining location information of the camera.

First, the cost function of Equation 4 has a number of local minimums, and the initial value deviates greatly from the global minimum, that is, the ground truth, as the solution is obtained by the gradient method. In this case, there is a possibility of convergence to an incorrect value when performing optimization.

In addition, the location information of the camera obtained through the cost function of Equation 4 may have scale ambiguity. If it is assumed that there is one solution that minimizes the cost function of Equation 4, there may be another solution that satisfies Equation 4 when the locations of the 3D mark point and the surrounding image are enlarged.

In addition, the cost function of Equation 4 does not take into account a sensor bias for fusion of the sensor data provided in the vehicle V and the matching result.

To solve this problem, the fusion unit 130 may obtain the location information of the camera using Equation 6 in which a constraint condition is added to Equation 4 described above.

Equation 6 may be a form in which a first constraint starting with λ ₁ , a second constraint starting with λ ₂ , and a third constraint starting with λ ₃ are added to the cost function of Equation 4.

Prior to obtaining the location information of the camera, the fusion unit 130 may first determine an initial value to be applied to the cost function of Equation 6. Specifically, when a new surrounding image is acquired after a plurality of surrounding images are acquired at a plurality of viewpoints, the fusion unit 130 determines an initial value required for obtaining camera position information at the time when the new surrounding images are acquired as It can be determined in the same way, but the spirit of the present invention is not limited thereto.

- Acquisition of driving information of the vehicle detected by the sensor between the time when the last surrounding image is acquired among the plurality of surrounding images described above and the time when a new surrounding image is additionally acquired thereafter.

- Based on this driving information, additional vehicle displacement information is obtained.

- The initial value is determined (estimated) by fusing the position information of the camera, which takes into account the last acquired surrounding image, and the displacement information of the newly acquired vehicle.

라는 수식에서,

는 주변 영상

이후 m 번째까지 얻은 m개의 측정 값을 표기한 것이다. 아울러, 주변 영상과 다음 주변 영상 사이의 측정값들을 이용한 적분표기는

과

을 이용하여 과 같이, k번째 주변 영상부터 k+1번째 주변 영상으로 표기 가능하다.Meanwhile,

In the formula,

The surrounding footage

After that, m measurements obtained up to the m th are displayed. In addition, the integral notation using the measured values between the surrounding image and the next surrounding image

and

It is possible to mark from the k-th surrounding image to the k+1-th surrounding image as shown by using.

Equation 7 below is an example of an equation used by the fusion unit 130 to determine an initial value.

과

은 차량(V)의 센서를 이용해 획득한 변위 정보를 의미하고,

과

은 M번째 주변 영상의 위치 정보 획득을 위한 초기값을 의미할 수 있다.here,

and

Means displacement information acquired using the sensor of the vehicle (V),

and

May mean an initial value for obtaining location information of the M-th surrounding image.

은 Ground Truth 근방에 위치할 수 있다.When following Equation 7, since the initial value of the cost function is determined based on the sensor value, the coordinate transformation matrix of the surrounding image related to the location information of the camera (

Can be located near Ground Truth.

를 수학식 8에 따라 구할 수 있다.After setting the initial value, the fusion unit 130 obtains a time point of each of the plurality of surrounding images acquired based on displacement information and matching information between the acquisition time points of each of the plurality of surrounding images acquired based on the driving information of the vehicle V It is possible to set a first constraint on the difference between the displacement information. To this end, the fusion unit 130 is the relative position of the camera between adjacent images from the cost function according to Equation 4

Can be obtained according to Equation 8.

If, on the assumption that the camera moves on a plane, the moving distance obtained by the camera and the moving distance obtained by the sensor provided in the vehicle V should be the same as in Equation (9).

는 차량(V)에 마련된 센서로 구한 주변 영상 사이의 변위이고,

는 수학식 8을 통해 구한 주변 영상 획득 시점 사이의 카메라의 변위이다.here,

Is the displacement between the surrounding images obtained by the sensor provided in the vehicle (V),

Is the displacement of the camera between the acquisition time points of the surrounding image obtained through Equation 8.

항을 설정할 수 있다. 제 1 제약 조건은 수학식 4의 비용함수로 구한 좌표변환 행렬로부터 추정한 주변 영상 사이 획득 시점 사이의 카메라 이동 거리가 차량(V)에 마련된 센서로 구한 주변 영상 획득 시점 사이의 카메라 이동 거리와 같아지도록 주변 영상의 위치를 조정하는 역할을 한다. 이 때, 수학식 4의 비용함수도 같이 사용하므로, 결과적으로 카메라 궤적의 형상은 그대로 유지하면서 축척만 조정하는 결과를 얻을 수 있다.Based on this, the fusion unit 130 is

You can set the terms. The first constraint is that the camera movement distance between the acquisition points of the surrounding images estimated from the coordinate transformation matrix obtained by the cost function of Equation 4 is the same as the camera movement distance between the acquisition points of the surrounding images determined by the sensor provided in the vehicle (V). It plays a role of adjusting the position of the surrounding image so that it is clear. In this case, since the cost function of Equation 4 is also used, as a result, it is possible to obtain a result of adjusting only the scale while maintaining the shape of the camera trajectory.

6 is a diagram illustrating camera trajectories having different scales satisfying a cost function according to an embodiment of the present invention.

As described above, according to the cost function of Equation 3, the fusion unit 130 may have a myriad of solutions having the same shape of the trajectory of the camera C but different scales. In FIG. 6, camera trajectories P1, P2, P3, P4, and P5 of the same shape with different scales are illustrated. At this time, by adding a first constraint to the cost function of Equation 3, the fusion unit 130 allows the scale between adjacent surrounding images to converge to the same scale as the scale obtained by the sensor provided in the vehicle V. can do. As a result, the fusion unit 130 may acquire location information of the camera whose scale has been restored.

를 추정할 필요가 있다. In addition, the fusion unit 130 may set a second constraint condition on a difference between the attitude angle of the vehicle V and the attitude angle of the vehicle V obtained based on matching information among driving information of the vehicle V. Among the driving information of the vehicle V, the yaw rate information detected by the attitude sensor can strongly estimate the attitude angle of the camera, and thus, when fused with image information, accuracy and stability of the camera's location information can be improved. However, the cost function of Equation 4 does not utilize the yaw rate. In addition, since the yaw rate bias is not considered in Equation 4, the fusion unit 130 is biased through a separate process.

Need to estimate.

To this end, the fusion unit 130 may set a second limiting condition capable of correcting the camera positions and attitude angles and the yaw rate bias for a plurality of peripheral images in the cost function of Equation 4 together. Since the vehicle V generally performs a plane motion in a short time (Shortly and Locally), the fusion unit 130 estimates the relative posture angle change of the camera between the acquisition points of the adjacent surrounding images according to Equation 10 through the yaw rate It can be approximated by the attitude angle.

는 j-1번째 주변 영상과 j번째 주변 영상 사이에서 요레이트 센서 값을 적분한 횟수를 의미할 수 있다. here,

May mean the number of times the yaw rate sensor value is integrated between the j-1 th surrounding image and the j th surrounding image.

를 수학식 11에 따라 구할 수 있다.Then, the fusion unit 130 is a rotation matrix for the difference between the posture angle estimated by the camera and the posture angle estimated through the yaw rate between adjacent images acquisition time points.

Can be obtained according to Equation 11.

와

이 동일하다면,

의 값은 단위 행렬(Identity matrix)이 된다.if,

Wow

If this is the same,

The value of is an identity matrix.

을 설정할 수 있다. 여기서, 함수

는 3차원 회전 행렬의 회전축-각도 표현법(Rodrigues' Rotation Formula)에서 각도를 반환할 수 있다. 단위행렬에 대한

의 값은 0이 되므로, 융합부(130)는 수학식 4의 비용함수에서

의 값이 단위행렬에 가까워지도록 해를 구할 수 있다. Therefore, the fusion unit 130 considers the above-described relationship and applies the second constraint to the cost function of Equation 4

Can be set. Where, the function

Can return the angle from the Rodrigues' Rotation Formula of the 3D rotation matrix. For the unit matrix

Since the value of is 0, the fusion unit 130 is in the cost function of Equation 4

The solution can be solved so that the value of is close to the unit matrix.

를 보정할 수 있다. 그 결과, 시차(Parallax)가 낮은 상황에서도 안정적인 각도 추정 값을 얻을 수 있다.By adding the above-described first and second constraints to Equation 4, the fusion unit 130 performs a posture angle among the position information of the camera so that it becomes similar to the posture angle obtained using the yaw rate.

Can be corrected. As a result, a stable angle estimation value can be obtained even in a situation where the parallax is low.

를 보정함으로써, 요레이트를 기반으로 획득된 자세각의 정확도를 높일 수 있다.In addition, the fusion unit 130 biases the sensor so that the attitude angle obtained using the yaw rate becomes similar to the attitude angle among the position information of the camera.

By correcting for, it is possible to increase the accuracy of the attitude angle obtained based on the yaw rate.

을 설정할 수 있다.Meanwhile, since the sensor bias gradually changes with time, a bias between adjacent neighboring image acquisition times may have substantially similar values. However, when only the first constraint and the second constraint are set in the cost function of Equation 4, the fusion unit 130 may acquire different sensor biases over time. Therefore, the fusion unit 130 uses the third constraint so that the sensor bias between the adjacent image acquisition time points is similar.

Can be set.

The third constraint restricts the difference in sensor bias between the acquisition time points of adjacent surrounding images to be close to zero, thereby increasing the reliability of fusion between the surrounding images and the attitude sensor.

When the initial values and constraints are set as described above, the fusion unit 130 may obtain the location information of the camera using the cost function of Equation 6, and estimate the location of the vehicle V from this. have.

The above-described apparatus and method for obtaining camera location information of a vehicle can more accurately obtain location information of a camera and, based on this, can increase an estimated location accuracy of a vehicle. Specifically, by fusing matching information for a plurality of surrounding images and displacement information of the vehicle obtained by the vehicle sensor, the scale of the camera's location information based on the surrounding images is restored, and the bias of the sensor is also corrected, The three-dimensional position of the vehicle can be accurately estimated.

Meanwhile, each step included in the method for obtaining camera location information of a vehicle according to the above-described exemplary embodiment may be implemented in a computer program programmed to perform these steps and a computer-readable recording medium recording the computer program.

The above description is merely illustrative of the technical idea of the present invention, and those of ordinary skill in the art to which the present invention pertains will be able to make various modifications and variations without departing from the essential quality of the present invention. Accordingly, the embodiments disclosed in the present invention are not intended to limit the technical idea of the present invention, but to explain the technical idea, and the scope of the technical idea of the present invention is not limited by these embodiments. The scope of protection of the present invention should be interpreted by the following claims, and all technical ideas within the scope equivalent thereto should be construed as being included in the scope of the present invention.

According to an embodiment, the above-described apparatus and method for obtaining camera location information of a vehicle may be used in various fields such as indoors or industrial sites, and thus has industrial potential.

1: Vehicle camera location information acquisition system
100: device for obtaining location information of a vehicle camera
110: matching unit
120: displacement information acquisition unit
130: fusion unit
V: Vehicle

Claims (10)

Matching feature points of a plurality of surrounding images acquired at a plurality of viewpoints by a camera provided in the vehicle;
Obtaining displacement information of the vehicle based on driving information of the vehicle detected by a sensor provided in the vehicle; And
Comprising the step of fusing the obtained displacement information of the vehicle and matching information for the feature point to obtain the location information of the camera
How to obtain vehicle camera location information. The method of claim 1,
Acquiring the location information of the camera,
Used for odometry calculation, performed through a process of minimizing a predetermined cost function designed to consider the displacement information and the matching information
How to obtain vehicle camera location information. The method of claim 1,
The above method,
Additionally acquiring a surrounding image after a time point at which the surrounding image is finally acquired among the plurality of surrounding images;
Additionally acquiring displacement information of the vehicle based on driving information of the vehicle detected by the sensor between the time when the last image is acquired and the time when the image is additionally acquired;
Fusing the position information of the camera obtained from the step of obtaining the position information of the camera and the additionally obtained displacement information of the vehicle; And
Further comprising the step of acquiring location information of the camera at a time point at which the additional surrounding image is acquired using a predetermined initial value estimated by the fusing step.
How to obtain vehicle camera location information. The method of claim 1,
Acquiring the location information of the camera,
Performed based on a difference in displacement information between acquisition points of each of the plurality of surrounding images acquired based on the driving information of the vehicle and displacement information between acquisition points of each of the plurality of surrounding images acquired based on the matching information
How to obtain vehicle camera location information. The method of claim 1,
Acquiring the location information of the camera,
Performed based on the difference between the vehicle's attitude angle and the vehicle's attitude angle obtained based on the matching information among the driving information of the vehicle
How to obtain vehicle camera location information. The method of claim 1,
Compensating the bias of the sensor using a difference between the vehicle attitude angle and the vehicle attitude angle obtained based on the matching information among the driving information of the vehicle
How to obtain vehicle camera location information. The method of claim 6,
Compensating the bias of the sensor,
Performed based on the difference between the biases of the sensor acquired at each of the acquisition time points of the plurality of surrounding images
How to obtain vehicle camera location information. A matching unit for matching feature points of a plurality of surrounding images acquired at a plurality of viewpoints by a camera provided in the vehicle;
A displacement information acquisition unit for acquiring displacement information of the vehicle based on driving information of the vehicle detected by a sensor provided in the vehicle; And
Comprising a fusion unit for acquiring the location information of the camera by fusing the obtained displacement information of the vehicle and matching information on the feature point
A device for acquiring vehicle camera location information. A program stored in a computer-readable recording medium for performing each step according to the method according to any one of claims 1 to 7. A computer-readable recording medium on which a program including instructions for performing each step according to the method according to any one of claims 1 to 7 is recorded.

Images