KR20220151572A - Method and System for change detection and automatic updating of road marking in HD map through IPM image and HD map fitting - Google Patents

Abstract

The present invention discloses an automatic map updating method in the autonomous driving field. More specifically, the present invention relates to a method and a system for automatically updating a high definition (HD) road map by automatically determining changes in a road surface object through an IPM image and precision road map fitting, thereby obtaining an updating result of higher accuracy only with very low calculation costs, a low-cost camera, and GPS/IMU sensors. According to an embodiment of the present invention, a method of matching an IPM conversion image with an HD road map is to rapidly determine whether a road object is updated or not, and to calculate spatial coordinates of the object. Since a top-view image conversion using IPM is not a complete orthographic image, when matching with the HD road map, a yaw error of the IMU sensor and GPS latitude and error are corrected to obtain accurate 4DoF (with respect to X, Y, Z, and Yaw).

Description

Method and system for change detection and automatic updating of road marking in HD map through IPM image and HD map fitting}

The present invention relates to a method for automatically updating a map in the field of autonomous driving, in particular, by removing a perspective effect from a front camera image of a vehicle with IPM and remapping it into a 2D domain to fit a map area with precision, resulting in very low computational cost and low cost It relates to a method and system for automatically updating a precision map that automatically determines and updates changes in a road surface object through an IPM image and a precision map fitting that can obtain a high-accuracy update result only with a camera and a GPS/IMU sensor.

In almost all modern applications involving autonomous driving, driver assistance systems, and navigation, road markings are important because they allow them to locate themselves on the road (M. Schreiber, C. Knoppel, and U. Franke, “Laneloc: Lane marking based localization using highly accurate maps,” IEEE Intelligent Vehicles Symposium, 2013.).

Many approaches to autonomous driving (e.g. J. Ziegler, P. Bender, T. Dang, and C. Stiller, "Trajectory planning for bertha local, continuous method," IEEE Intelligent Vehicles Symposium, 2014.) It uses high-precision maps that include traffic lanes and stop lines to plan maneuvers and recognize different traffic situations. Objects to be updated vary from highways to urban environments, depending on many factors such as traffic density, visibility, type of obstacles (vehicles, trucks, bicycles, pedestrians, etc.), number of road lanes, and road markings.

Road and lane marking detection is an important topic in camera-based driver assistance systems, dating back to the early 1990s (E. Dickmanns, B. Mysliwetz, and T. Christians, "An integrated spatio-temporal approach to automatic visual guidance of autonomous vehicles," IEEE Transactions on Systems, Man and Cybernetics, 1990.).

The number of approaches to this subject is very large. Most of the early work was specifically on highways (e.g. P. Charbonnier, F. Diebolt, Y. Guillard, and F. Peyret, "Road markings recognition using image processing," IEEE Conference on Intelligent Transportatoin Systems, 1997., K. Kluge and S. Lakshamanan, "A deformable-template approach to lane detection," IEEE Intelligent Vehicles Symposium, 2005. and, J. C. McCall and M. M. Trivedi, "An integrated robust approach to lane marking detection and lane tracking," IEEE Intelligent Vehicles Symposium, 2004 .) focused on lane marking detection.

A more recent approach is towards tracking road geometry in urban areas. For example, S. Vacec, C. Schimmel, and R. Dillman, "Road-marking analysis for antonomous vehicle guidance," European Conference on Mobile Robots, 2007. It became. These approaches are designed mostly for tracking purposes, not object detection or scene understanding, which is necessary to obtain high-precision maps. One early approach to object-based detection of road markings is from J. Rebut, A. Bensrhair, and G. Toulminet, "Image segmentation and pattern recognition for road marking analysis," IEEE International Symposium on Industrial Electronics, 2004. Similarly, we dealt with frequency transformation for feature extraction and arrow classification using the K-Nearest Neighbor method for classification. Another proposal, W. Nan, L. Wei, Z. Chunmin, Y. Huai, and J. Jiren, "The detection and recognition of arrow markings recognition based on monocular vision," IEEE International Symposium on Industrial Electronics, 2004. We use the Haar Wavelet for arrow classification. P. Foucher, Y. Sebsadji, J.-P. In Tarel, P. Charbonnier, and P. Nicolle, "Detection and recognition of urban road markings using images," IEEE International conference on Intelligent Transportation Systems, 2011. Arrows were segmented via median filtering and then classified using template matching. . It can also be classified by clustering segments that satisfy geometric constraints, such as zebra junctions. Algorithms for detecting other types of markings have also been proposed, T. Marita, M. Negru, R. Danescu, and S. Nedevschi, "stop-line detection and localization method for intersection scenarios," IEEE International, focusing on stop-line detection. Conference on Intelligent Computer Communication and Processing, 2010. and Y.-W. It was also proposed in Seo and R. Rajkumar, "A vision system for detecting and tracking of stop-lines," IEEE International Conference on Intelligent Transportation Systems, 2014.

For a thorough recent review of vehicle detection, S. Sun et al., "On-road vehicle detection: a review", IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 28, no 5, may 2006, pp. 28; 694-711., all methods are based on computer vision except the method combined with a distance finder (Laser or Lidar).

The method of using the camera is based on object features (T. Rabie et al., "Active-Vision based traffic surveillance and control", Vision Interface Annual Conference (VI'01), Toronto, Canada, June 2001.) or edge lines ( R. Okada and K. Onoguchi, "Obstacle Detection Using Projective Invariant and Vanishing Lines", 9th IEEE International Conference on Computer Vision (ICCV'03), Nice, France, Oct. 2003.). In general, the segmentation of feature elements of a camera image is based on optical flowG.P. Stein and A. Shashua, "A robust method for computing vehicle ego-motion", IEEE Intelligent Vehicle Symposium (IV'00), Piscataway, NJ, USA, Oct. 2000., Dense Estimation and Color and Texture ConstraintsB. Heisele, "Motion-based object detection and tracking in color images sequences", Fourth Asian Conference on Computer Vision (ACCV'00), Taipei, China, Jan. 2000, pp 1028-1033. or parameterization of a static mapC. It can be performed by introducing Braillon et al., "Real-time moving obstacle detection using optical flow models", IEEE Intelligent Vehicle Symposium (IV’00), Tokyo, Japan, June 2006, pp 466-471.

With stereo vision, the depth of matching pixels in a pair of images can be estimated by triangulation, provided the stereo equipment is calibrated. SchreiberSchreiber, D.; Alefs, B.; Clabian, M. "Single Camera Lane Detection and Tracking", In Proceedings of IEEE Conference on Intelligent Transportation Systems, Vienna, Austria, 13-16 September 2005; pp. 302-307. proposed a robust scheme to demarcate road boundaries by using a single forward-facing camera to detect and track lane markings. McCall and Trivedi McCall, J.C.; Trivedi, M.M. "Video-Based Lane Estimation and Tracking for Driver Assistance: Survey, System, and Evaluation", IEEE Trans. Intell. Transport. Syst. 2006, July, 20-37. designed a video-based driver assistance system for estimating and tracking lane markings on the road. Liu Liu, Y.-C.; Lin, K.-Y.; Chen, Y.-S. "Bird's-Eye View Vision System for Vehicle Surrounding Monitoring.", In Proceedings of RobVis 2008, Auckland, New Zealand, 18-20 February 2008; pp. 207-218. proposed a bird's eye vision system for monitoring the vehicle's surroundings by installing six cameras on the vehicle and stitching the views from the six cameras.

In general, the proposed Inverse Perspective Mapping (IPM) methods use different conversion mechanisms. Based on linear mapping of homogeneous coordinates, Muad Mallot, H.A.; Bulthoff, H.H.; Little, J.J.; Bohrer, S. "Inverse Perspective Mapping Simplifies Optical Flow Computation and Obstacle Detection.", Biol. Cybern. 1991, 64, 177-185. treated the mathematical theory of IPM. In another study, Tan, Dale, Anderson and Johnston Tan, S.; Dale, J.; Anderson, A.; Johnston, A. "Inverse Perspective Mapping and Optic Flow: A Calibration Method and a Quantitative Analysis." Image Vision Comput. 2006, 24, 153-165. provide basic geometric transformation structures and formulas for IPM.

Bertozzi Bertozzi, M.; Broggi, A.; Fascioli, A. "Stereo Inverse Perspective Mapping: Theory and Applications." Image Vision Comput. 1998, 16, 585-590. proposed a heuristic search method for finding homologous points. Some researchers have also used the vanishing point phenomenon to calculate IPM models.

Nieto, M.; Salgado, L.; Jaureguizar, F.; Cabrera, J. "Stabilization of Inverse Perspective Mapping Images Based on Robust Vanishing Point Estimation." In Proceedings of IEEE Conference on Intelligent Vehicles Symposium, Istanbul, Turkey, 13-15 June 2007; pp. 315-320., Muad, A.M.; Hussain, A.; Samad, S.A.; Mustaffa, M.M.; Majlis, B.Y. "Implementation of Inverse Perspective Mapping Algorithm for the Development of an Automatic Lane Tracking System." In Proceedings of IEEE Region 10 Conference TENCON, Chiang Mai, Thailand, 21-24 November 2004; pp. 207-210.

IPM uses the camera position and road direction information to create a bird's-eye view image with perspective effect removed. Perspective correction allows much more efficient and robust road detection, lane marking tracking or pattern recognition algorithms to be implemented. In practice, IPM is used not only for the purpose of detecting the vehicle's position relative to the road, but also for obstacle detection (V. Kastrinaki, M. Zervakis, K. Kalaitzakis, "A survey of video processing techniques for traffic applications", Image Vision Comput. 21 (4 ) (2003) 359-381., N. Simond, M. Parent, "Obstacle detection from ipm and super-homography", in: IROS, IEEE, 2007, pp. 4283-4288.), free space estimation (P. Cerri, P. Grisleri, "Free space detection on highways using time correlation between stabilized sub-pixel precision IPM images", in: Proceedings of the IEEE International Conference on Robotics and Automation, Barcelona, Spain, 2005, pp. 2223-2228. ), detection (M. Guanglin, P. Su-Birm, S. Miiller-Schneiders, A. Ioffe, A. Kummert, "Vision based pedestrian detection-reliable pedestrian candidate detection by combining IPM and a 1D profile", in: Proceedings of the IEEE Intelligent Transportation Systems Conference, or ego-motion estimation C. Lundquist, T.B. Sch_n, "Joint ego-motion and road geometry estimation, Inform". Fusion 12 (4) (2011) 253-263.), Even in many ADAS related applications has been used

Therefore, IPM is of paramount importance in the numerous automation tasks that intelligent vehicles have to handle. IPM operates under three key assumptions. The road should be a flat surface, there should be a rigid body transformation from the camera to the road, and the road should be clear of obstacles. This is a realistic minimum assumption as roads are very often filled with other vehicles, barriers, pedestrians, etc. There are several examples in the literature supporting road sensing, e.g. J. McCall, M. Trivedi, "evaluation of a vision based lane tracker designed for driver assistance systems" 153-158., A. Muad, A. Hussain, S Samad, M. Mustaffa, B. Majlis, "Implementation of inverse perspective mapping algorithm for the development of an automatic lane tracking system," in: Proceedings of the IEEE International Conference TENCON, vol. A, Chiang Mai, Tailandia, 2004, pp. 207-210., F. Dornaika, J. Alvarez, A. Sappa, A. Lopez, "A new framework for stereo sensor pose through road segmentation and registration", IEEE Trans. Intell. Transp. Syst. 12 (4 ) (2011) 954-966.,).

Other authors have also used tunable filters for lane marking detection and retained filtering in IPM images, allowing the use of a single kernel size over the entire region of interest (J. McCall, M. Trivedi, "Performance evaluation of a vision based lane tracker designed for driver assistance systems", in: Proceedings of the IEEE Intelligent Vehicles Symposium, Las Vegas, NV, USA, 2005, pp. 153-158.). In addition, multiple cameras can be used to generate a single IPM image mosaic, as the images are mapped to a new reference system (C. Guo, S. Mita, D. McAllester, "Stereovision-based road boundary detection for intelligent vehicles in challenging scenarios", in: Proceedings of the IEEE International Conference on Intelligent Robots and Systems, St. Louis, MO, USA, 2009, pp. 1723-1728., M. Bertozzi, A. Broggi, "GOLD: a parallel real-time stereo vision system for generic obstacle and lane detection", IEEE Trans. Image Proc. 7 (1) (1998) 62-81.).

There is also a dedicated hardware system under development to compute IPM images (L. Luo, I. Koh, S. Park, R. Ahn, J. Chong, "A software-hardware cooperative implementation of bird's-eye view system for camera-on-vehicle", in: Proceedings of the IEEE International Conference on Network Infrastructure and Digital Content, Beijing, China). Pitch changes occur during brake or acceleration manoeuvres, while roll changes are expected to occur during heavy turns. This problem is discussed in P. Coulombeau, C. Laurgeau, "Vehicle yaw, pitch, roll and 3D lane shape recovery by vision", in: Proceedings of the IEEE Intelligent Vehicles Symposium, Versailles, France, 2002, pp. 619-625. R. Labayrade, D. Aubert, "A single framework for vehicle roll, pitch, yaw estimation and obstacles detection by stereovision", in: Proceedings of the IEEE Intelligent Vehicles Symposium, Columbus, OH, USA, 2003, pp. 31-36. F. Dornaika, J. Alvarez, A. Sappa, A. Lopez, "A new framework for stereo sensor pose through road segmentation and registration", IEEE Trans. Intell. Transp. Syst. 12 (4) (2011) 954-966., M. Nieto, L. Salgado, F. Jaureguizar, J. Cabrera, "Stabilization of inverse perspective mapping images based on robust vanishing point estimation", in: Proceedings of the IEEE Intelligent Vehicles Symposium, Istanbul, Turkey, 2007, pp. 315-320.

Indeed, some authors argue that even small errors in vehicle roll/pitch estimation lead to huge terrain classification errors. S. Thrun, M. Montemerlo, A. Aron, Probabilistic terrain analysis for high-speed desert driving, in: Proceedings of the Robotics Science and Systems Conference, University of Pennsylvania, Philadelphia, PA, 2006, pp. 21-28.. A. Broggi, S. Cattani, P.P. Porta, P. Zani, A laserscanner-vision fusion system implemented on the terramax autonomous vehicle, in: Proceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems, Beijing, China, 2006, pp. 111-116., an algorithm that fuses vision and laser data is proposed.

Despite the growing interest in autonomous vehicle research, perception-based positioning in complex urban environments remains challenging. The widely used GPS is sometimes limited by multipath problems caused by tall buildings. To overcome these limitations, detection sensors such as Light Detection and Ranging (LiDAR) sensor cameras have been highlighted as a solution. LiDAR Simultaneous Localization and Mapping (SLAM) J. Zhang and S. Singh, "LOAM: Lidar odometry and mapping in realtime." in Proc. Robot.: Sci. Sys. Conf., 2014, vol. 2, no. 9. and Visual SLAM R. Mur-Artal, J. M. M. Montiel, and J. D. Tardos, "ORB-SLAM: A versatile and accurate monocular SLAM system," IEEE Trans. Robot., vol. 31, no. 5, p. 1147-1163, Oct. 2015. reported surprising performance without prior knowledge of the environment. However, with prior knowledge (available prior maps), these SLAM problems can be solved.

You can also configure advance maps in LiDAR. Barsan et al. I.A. Barsan, S. Wang, A. Pokrovsky, and R. Urtasun, "Learning to localize using a LiDAR intensity map." in Proc. Conf. Robot Learn., 2018, pp. 605-616. used the intensity information of the point cloud map using deep learning techniques to estimate the vehicle attitude. Wolcott and Eustice R. W. Wolcott and R. M. Eustice, "Visual localization within lidar maps for automated urban driving," in Proc. IEEE/RSJ Intl. Conf. Intell. Robots Sys., 2014, pp. 176-183. estimated the pose of the vehicle by calculating the normalized mutual information (NMI) between the rendered image of the 3D point cloud and the real camera image in the composite view. Kim et al. performed localization on a 3D map using stereo camera images. To improve visual localization for point cloud maps, certain studies have reported methods to insert relevant visual features into maps (H. Kim, T. Oh, D. Lee, and H. Myung, "Image-based localization using prior map database and monte carlo localization," in Proc. Intl. Conf. Ubiquitous Robots Ambient Intell., 2014, pp. 308-310., N. Labs.NAVER LABS HDMap Dataset. 2019, [Online].Available: https http://hdmap.naverlabs.com/.).

Because the geometric information of the point cloud is less affected by light dispersion, it is possible to respond to visual changes in the environment. HD maps contain semantic information in the form of compressed 3D vectors constructed from point clouds obtained from high-precision Mobile Mapping System (MMS) or high-resolution aerial/satellite imagery. Since the HD map has semantic information in vector form and global coordinates, the map can represent a large area with a small capacity.

For example, N. Labs.NAVER LABS HDMap Dataset. 2019, [Online].Available: The HD map provided by https://hdmap.naverlabs.com/ encapsulates an area of 11 × 11 km within 17 MB. This is a complex urban dataset (J. Jeong, Y. Cho, Y.-S. Shin, H. Roh, and A. Kim, "Complex urban dataset with multi-level sensors from highly diverse urban environments," Intl. J. Robot. Res., vol. 38, no. 6, pp. 642-657, 2019.), it is 30 times more efficient considering that the point cloud map data size of the same area is about 500 MB.

Ma et al. W. -C. Ma et al., "Exploiting sparse semantic HD maps for self-driving vehicle localization," 2019, arXiv:1908.03274. An HD map consisting of lane and traffic sign information was used. Welte et al. A. Welte, P. Xu, P. Bonnifait, and C. Zinoune, "Estimating the reliability of georeferenced lane markings for map-aided localization," in Proc. IEEE Intell. Vehicle Symp., 2019, pp. 1225-1231. estimated vehicle positions using marks stored on the map through a Kalman smoothing process. Poggenhans et al. F. Poggenhans, N. O. Salscheider, and C. Stiller, "Precise localization in high-definition roadmaps for urban regions," in Proc. IEEE/RSJ Intl. Conf. Intell. Robots Sys., 2018, pp. 2167-2174. used HD maps with road markings and road boundary information. Each detector extracted features from the stereo image input and used an unscented Kalman filter (UKF) to match the feature to a map to estimate its location. Vivacqua et al. estimated the location by matching the lane mark points estimated by the camera with the lane information of the HD map.

Registered Patent Publication No. 10-1454153 (Announcement date: 2014.11.03.)

Through the background knowledge described above, the present invention calculates the correlation between label information included in the HD map and semantic information estimated from an image through a convolutional neural network (CNN) to determine whether or not to update the map in object units, However, the problem is to provide a system method for determining the global position of an update object with sub-meter level accuracy.

In order to solve the above-described problems, an auto-update method for automatically determining and updating a change in a road surface object through an IPM image and a precision-map fitting according to an embodiment of the present invention is one mounted on a vehicle in motion. Collecting vehicle information including a front view image of the vehicle from the above sensors, generating a corrected image by correcting distortion by a camera existing in the front view image, and converting the corrected image into an IPM image through inverse perspective mapping. obtaining a HD map from preliminary map information by using vehicle GPS positioning coordinates included in the vehicle information, extracting an IPM area from the HD map included in the preliminary map information, and generating a base map corresponding to the IPM image. Generating, correcting the pose of the IPM image through multi-directional correction of the IPM image, fitting the IPM image and the base map, and comparing the existing road surface object of the base map with the IPM image to determine whether to update It may include the step of determining.

In addition, in order to solve the above-described problems, an automatic roadmap update system for automatically determining and updating changes in road surface objects through IPM image and precision roadmap fitting according to an embodiment of the present invention is mounted on a running vehicle. A vehicle information collection unit that collects vehicle information including a front view image of the vehicle from one or more sensors, an image correction unit that generates a corrected image by correcting distortion by a camera existing in the front view image, and the corrected image. An image conversion unit that converts an IPM image through inverse perspective mapping, obtains an HD map from preliminary map information using the vehicle GPS positioning coordinates included in the vehicle information, and extracts an IPM area from the HD map to correspond to the IPM image. A map generating unit for generating a base map for the IPM image, a map fitting unit for fitting the IPM image and the base map by correcting the pose of the IPM image through multi-directional correction, and IPM for existing road surface objects in the base map. It may include a determination unit for determining whether to update through comparison with the image.

According to an embodiment of the present invention, by quickly determining whether a road facility object is updated by matching the IPM converted image with the map with precision, and calculating the spatial coordinates of the object, Top-View image conversion to IPM is complete. Since it is not an orthographic image, there is an effect of obtaining accurate 4DoF (for X, Y, Z, and Yaw) by correcting the yaw error of the IMU sensor and the latitude and error of the GPS when matching with the map with precision.

In addition, according to an embodiment of the present invention, it is possible to very effectively reduce the resource of precision map update, enable rapid automatic update, and also provide accurate positioning information with a relatively inexpensive monocamera and low-cost GPS/IMU. Since it can be obtained, it has the effect of enabling a wide range of applications, such as using it as a self-vehicle positioning solution for autonomous driving.

1 is a diagram showing a method for automatically updating a road surface object by automatically determining and updating a change in a road surface object through an IPM image and a road map fitting according to an embodiment of the present invention.
2 is a diagram for explaining a binarization method applied in a method for automatically updating a road surface object by automatically determining and updating a change in a road surface object through an IPM image and a road map fitting according to an embodiment of the present invention.
3 is a diagram illustrating a binarization result according to a method for automatically updating a road surface object by automatically determining and updating a change in a road surface object through an IPM image and a precision map fitting according to an embodiment of the present invention.
4 is a diagram for explaining a lateral direction correction method applied in a method for automatically updating a road surface object by automatically determining and updating a change in a road surface object through an IPM image and a precision road map fitting according to an embodiment of the present invention.
5 is a diagram illustrating a result of yaw correction according to a method of automatically updating a road surface object by automatically determining and updating a change of a road surface object through an IPM image and a road map fitting according to an embodiment of the present invention.
6 and 7 are diagrams for explaining a longitudinal correction method according to the method of automatically updating a road surface object by automatically determining and updating a change of a road surface object through an IPM image and a road map fitting according to an embodiment of the present invention; to be.
8 illustrates spatial coordinate data for detecting a road surface object according to a method for automatically updating a road surface object by automatically determining and updating a change in a road surface object through an IPM image and a precision road map fitting according to an embodiment of the present invention. it is a drawing
9 is a diagram for explaining a method of updating a road surface object according to the method of automatically updating a road surface object by automatically determining and updating a change of a road surface object through an IPM image and a road surface map fitting according to an embodiment of the present invention.
10 is a diagram illustrating an update result of a road surface object according to a method of automatically updating a road surface object by automatically determining and updating a change of a road surface object through an IPM image and a precision map fitting according to an embodiment of the present invention.
11 is a diagram showing an automatic updating system for updating a road surface object by automatically determining and updating a change in a road surface object through an IPM image and a precision map fitting according to an embodiment of the present invention.

The present invention as described above will be described in detail through the accompanying drawings and embodiments.

It should be noted that technical terms used in the present invention are only used to describe specific embodiments and are not intended to limit the present invention. In addition, technical terms used in the present invention should be interpreted in terms commonly understood by those of ordinary skill in the art to which the present invention belongs, unless specifically defined otherwise in the present invention, and are excessively inclusive. It should not be interpreted in a positive sense or in an excessively reduced sense. In addition, when the technical terms used in the present invention are incorrect technical terms that do not accurately express the spirit of the present invention, they should be replaced with technical terms that those skilled in the art can correctly understand. In addition, general terms used in the present invention should be interpreted as defined in advance or according to context, and should not be interpreted in an excessively reduced sense.

Also, singular expressions used in the present invention include plural expressions unless the context clearly dictates otherwise. In the present invention, terms such as "consisting of" or "comprising" should not be construed as necessarily including all of the various elements or steps described in the invention, and some of the elements or steps are included. It should be construed that it may not be, or may further include additional components or steps.

Also, terms including ordinal numbers such as first and second used in the present invention may be used to describe components, but components should not be limited by the terms. Terms are used only to distinguish one component from another. For example, a first element may be termed a second element, and similarly, a second element may be termed a first element, without departing from the scope of the present invention.

Hereinafter, a preferred embodiment according to the present invention will be described in detail with reference to the accompanying drawings, but the same or similar components are assigned the same reference numerals regardless of reference numerals, and redundant description thereof will be omitted.

In addition, in describing the present invention, if it is determined that a detailed description of a related known technology may obscure the gist of the present invention, the detailed description will be omitted. In addition, it should be noted that the accompanying drawings are only for easily understanding the spirit of the present invention, and should not be construed as limiting the spirit of the present invention by the accompanying drawings.

The present invention calculates the correlation between label information included in the HD map and semantic information estimated from an image through a convolutional neural network (CNN) to determine whether or not to update the map in object units, and to update the object with sub-meter level accuracy. It is possible to provide a global positioning method of

In detail, according to an embodiment of the present invention, the image collected from the camera mounted in the vehicle is placed in the world coordinate system through Inverse Perspective Mapping (IPM) and fitted to the same location on the HD map, thereby reducing the road surface and road facility recognized by the camera. The update of the object - hereinafter referred to as 'road surface object' - is determined. Here, the road surface objects refer to lanes and crosswalks on the road, and vehicles are deleted.

In addition, according to an embodiment of the present invention, IPM results obtained by removing the influence of shadows by using an algorithm for preventing failure of binarization due to the influence of shadows on the road surface and using image segmentation results for extracting the outlines of objects. Acquire

And, by utilizing the spatial coordinates of the HD map, the object of the map is updated by extracting the spatial coordinates of the object on the IPM.

Hereinafter, a precision roadmap auto-update method and system for automatically determining and updating a change in a road surface object through fitting an IPM image and a high-definition map (HD Map) according to an embodiment of the present invention will be described with reference to the drawings. Of course, the process of automatically determining and updating the change of the road surface object can be performed in real time.

1 is a diagram showing a method for automatically updating a road surface object by automatically determining and updating a change in a road surface object through an IPM image and a road map fitting according to an embodiment of the present invention.

Referring to FIG. 1, a method for automatically updating a road surface object by automatically determining and updating a change in a road surface object through an IPM image and a road map fitting according to an embodiment of the present invention includes one or more sensors mounted on a vehicle in motion. Collecting vehicle information including a front view image of the vehicle from (S100), generating a corrected image by correcting distortion by a camera existing in the front view image (S110), and converting the corrected image through reverse perspective mapping. Converting to an IPM image (S120), obtaining an HD map from preliminary map information using vehicle GPS positioning coordinates included in vehicle information, extracting an IPM area from the HD map, and generating a base map corresponding to the IPM image Step (S130), fitting the IPM image and the base map by correcting the pose of the IPM image through multi-directional correction of the IPM image (S140), and comparing the base map with the existing road surface object with the IPM image. It may include determining whether to update (S150).

Collecting vehicle information including a front view image of the vehicle from one or more sensors mounted on the running vehicle (S100) includes acquiring one or more sensors or modules mounted in the vehicle, for example, a front view image of the vehicle. Vehicle information including a front view image of the front of the vehicle in motion from a camera, a GNSS that generates GPS coordinates, and an inertial measurement unit (IMU) that measures the speed, direction, gravity, and acceleration of the vehicle. This is the stage of collection. In this step, the front view image captured by the vehicle camera may be recorded, and the GPS coordinates and IMU values of the vehicle may be stored together in synchronization therewith.

Next, in step S110 of generating a correction image by correcting the distortion by the camera existing in the front view image, distortion of the image due to tangential distortion or radial distortion, as an example, is corrected. In this step, internal and external parameters related to the camera are extracted, and distortion of the front view image is corrected using the extracted parameters.

Next, in the step of converting the corrected image into an IPM image through inverse perspective mapping (S120), the front view image corrected through the inverse perspective mapping technique is projected onto a road plane from which perspective is removed.

First, with respect to the camera-related distortion-corrected image, the road surface object can be accurately identified through a binarization process.

As a procedure for this, step S110 applies a deep learning recognition algorithm, such as mask R-CNN or YOLO, learned about road facilities to a front view image to recognize segmented road objects. May include recognizing. The step of recognizing a road surface object is a step of performing binarization through an adaptive binarization procedure using an average value application algorithm or a Gaussian distribution algorithm for a front view image, and a step of generating a corrected image by reconstructing divided road objects. can be subdivided.

2 is a diagram for explaining a binarization method applied in a method for automatically updating a road surface object by automatically determining and updating a change in a road surface object through an IPM image and a road map fitting according to an embodiment of the present invention.

In this regard, since the IPM image of the road surface can identify edges and vertices of road surface objects through binarization, binarization is necessarily required. The biggest problem in the binarization process is the binarization error due to the influence of shadows. When shadows exist on the road surface due to buildings around the road, the histogram areas of these images show different brightness to the human eye, but actually have the same or similar brightness. Since the values are almost similar in brightness, they are equally shaded during binarization, making it difficult to determine the threshold for binarization.

Therefore, global threshold, which is binarized with threshold values of sunny and shaded areas in the entire image, is impossible.

On the other hand, Otsu's Method, known as a representative binarization algorithm, divides pixels into two classes by arbitrarily setting a threshold and repeats the operation of obtaining the intensity distribution of the two classes. Binarization is performed by selecting the most uniform threshold. The advantage of this Otsu binarization algorithm is that it automatically finds the optimal threshold value in the entire image. On the other hand, it is not fast because it must be calculated for all image pixels, and it is difficult to obtain correct results when the brightness distribution of the original image (a) is non-uniform due to shadows (Fig. 2 (b)).

On the other hand, the Adaptive Threshold algorithm compensates for this disadvantage of the global binarization method, and in the original image (a) of FIG. Better results can be obtained in cases where it is difficult to produce an image. After dividing the image into several areas, it calculates the threshold using only the pixel values around the area. Adaptive binarization algorithms include the Adapted-Mean algorithm, which is sharper than Gaussian distribution but has noise, depending on the calculation method, and the Gaussian distribution algorithm (Adapted-Gaussian), which is less sharp than the average algorithm but has relatively less noise. .

In the method of applying the adaptive binarization algorithm, first, a target image is set into blocks with N parts, and a threshold value is determined for each block. At this time, when the threshold value is an average value, the average value of neighboring pixels is determined as the threshold value, and when the threshold value is a Gaussian distribution, the threshold value is determined by the sum of weights according to the Gaussian distribution. Then, when the threshold value is determined, binarization is performed for each block based on the determined threshold value. Therefore, sharper and smoother results can be obtained than applying binarization to the entire region. Since most images have shadows or lighting differences, it is preferable to apply adaptive binarization rather than global binarization in the map auto-update method according to an embodiment of the present invention.

2 (b) is an example of applying the Otsu binarization method, and is a result of setting the optimal threshold to 96. (c) of FIG. 2 is a result of applying the average value among the adaptive binarization methods, and has more noise than Gaussian, but is characterized in that a clear result can be obtained. In addition, (d) of FIG. 2 is the result of applying the Gaussian distribution among the adaptive binarization methods, and it can be seen that the results are less sharp although the noise is less than that of the average value method.

In addition, the road surface of an actual road may have a number of differences in light reflection and shadow due to various road conditions such as the slope of a drainage channel, weather, and the influence of buildings around the road. Accordingly, it can be said that the use of an adaptive binarization algorithm is suitable according to the difficulty of obtaining an intended result by the binarization method using a threshold value based on the image histogram and the global binarization method based on the Otsu algorithm.

3 is a diagram illustrating a binarization result according to a method of automatically updating a road surface object by automatically determining and updating a change of a road surface object through an IPM image and a precision map fitting according to an embodiment of the present invention, in FIG. 3 ( In a) and (b), the upper left corner shows an IPM image, the upper right corner shows a histogram of the IPM image, the lower left corner shows an object-tracked IPM image, and the lower left corner shows an adaptive binarization image.

As shown, when adaptive binarization is applied, a zebra pattern appears on the shadow boundary as shown in the lower right image of FIGS. 3(a) and (b). Since these zebra patterns are caused by the non-uniformity of light-dark distribution even when binarization is applied to partial blocks, in order to solve this problem, according to an embodiment of the present invention, a mask-type convolutional neural network (CNN) is used to The stain is removed by filtering through image segmentation processing of the label.

(c) of FIG. 3 illustrates an image #1 applied to an original image #2, which is subjected to filtering after the above-described adaptive binarization, and an image #3, to which the above-described general binarization is performed.

Next, the IPM image is derived by reconstructing the image in its original size using the segmented pieces of the road surface objects appearing in the image binarized through the adaptive binarization process.

In detail, as pitch, roll, and yaw of the IPM image vary depending on many factors such as the shaking of the vehicle or the condition of the road surface that acquires the front view image, the initial roll is assumed to be parallel to the ground surface, and the yaw is the IMU sensor's It is calculated using the output value.

In addition, the pitch varies depending on the condition of the road surface, for example, an uphill or downhill road, a speed bump, etc., and the related pitch angle (θ _p ) can be derived by Equation 1 below.

The X, Y coordinate calculation formula reflecting the pitch angle (θ _p ) is the same as Equation 1 above. However, since it is inaccurate and difficult to measure θp, X, and Y through the vehicle, in the embodiment of the present invention, the later-described lateral matching and longitudinal fitting methods can be applied to obtain the same IPM image as the HD map. .

Next, in step S130 of acquiring an HD map from the preliminary map information using the vehicle GPS positioning coordinates included in the vehicle information and extracting an IPM area from the HD map to generate a base map corresponding to the IPM image (S130), In order to perform a correction procedure for positioning the IPM image from which effects have been removed in the same area as the actual HD map, an HD map is obtained from prior map information, and an area corresponding to the IPM is extracted from the HD map to be a base map for fitting. (base map) is created.

Next, the step of fitting the IPM image and the base map by correcting the pose of the IPM image through multi-directional correction of the IPM image (S140) is performed by fitting between the base map generated in step S130 and the IPM image. This is the step of matching the IPM image with the preliminary map.

4 is a diagram for explaining a lateral correction method applied in a method for automatically updating a road surface object by automatically determining and updating a change in a road object through an IPM image and a precision map fitting according to an embodiment of the present invention. In the direction fitting step, a correction value is first calculated to correct the inaccurate yaw value obtained from the IMU. This lateral fit can be viewed as a movement relative to the left/right side of the plane.

In detail, the correction value of Yaw can be calculated through the following process. This can be calculated using the angles (θ ₁ , θ ₂ ) of the lane segments at both ends of the HD map, and first, the angles (θ _a , θ _b ) of the HD map lanes and the angles (θ c , θ _c , θ _d ) is obtained (FIG. 4 (a)), and the angles (θ ₁ , θ ₂ ) of the line segments can be calculated through the angle difference of each lane (FIG. 4 (b)). Here, the angle of the line segment Since the average value (θ _y ) of (θ ₁ , θ ₂ ) can be regarded as the error value of the IMU, yaw correction can be performed by rotating the lane by the average value (θ _y ) (FIG. 4(c)).

5 is a diagram illustrating the Yaw correction result according to the method of automatically updating a road surface object by automatically determining and updating a change of a road surface object through an IPM image and a precision roadmap fitting according to an embodiment of the present invention, case #1. As shown in (a) and case #2(b), it can be seen that the IPM lane before correction is corrected to match the HD map lane after correction to a certain level or more.

Meanwhile, as described above, since it is difficult to calculate the pitch angle, the lane angle of the front view image and the angle of the ROI area for performing IPM are adjusted to match the base map. In addition, the horizontal scale of the distance from the lane is adjusted in the same way as the base map, so that an image that matches the horizontal direction is created.

Next, a longitudinal correction step may be performed. 6 and 7 are diagrams for explaining a longitudinal correction method according to the method of automatically updating a road surface object by automatically determining and updating a change of a road surface object through an IPM image and a road map fitting according to an embodiment of the present invention; As , the fitting method between the IPM image and the base map in the longitudinal direction is shown. Longitudinal correction can be viewed as up and down movement on a plane.

First, in the case of longitudinal fitting, when the lanes of the IPM image and the base map are rotated in the vertical direction, the difference (n _min ) of the upper part of the road surface object is as shown in (a) and (b) of FIG. , It can be calculated through the difference between the lanes of the IPM image and the base map, and this is expressed as Equation 2 below.

Here, n _min becomes a negative value.

In addition, when the lanes of the IPM image and the base map are rotated in the vertical direction, the difference (m _min ) of the lower part of the road object is between the lanes of the IPM image and the base map, as shown in (a) and (b) of FIG. It can be calculated through the difference, and this is expressed as Equation 3 below.

In addition, the average value (l _min ) of the difference between the upper part (n _min ) and the lower part (m _min ) during fitting can be calculated by Equation 4 below.

Accordingly, when the average value (l _min ) is a negative number, the IPM image is moved to the upper side, and when it is a positive number, the longitudinal fitting is performed by moving the IPM image to the lower side.

Thereafter, in the method of automatically updating a road surface object by automatically determining and updating a change in a road surface object through fitting an IPM image and a road surface object according to an embodiment of the present invention, the edge of the road surface object through binarization processing on the fitted IPM image (edge) and vertices can be found.

Next, as a step of determining whether to update the existing road surface object of the base map by comparing it with the IPM image (S150), the system identifies the road surface object in the IPM image and uses the preliminary map to determine whether or not the road is updated. will judge

According to an embodiment of the present invention, it is possible to obtain objects around the camera from a map using the GPS positioning coordinates of the camera through prior map information, and matching is possible through comparison with objects detected in the IPM image. Depending on this, it is determined whether or not to update. This step may consist of an object detection process and an HD map object comparison and determination process.

First, as an object detection process, among object information transmitted from a module that recognizes and detects an object from a camera image, an object included in an IPM image area may be extracted from prior map information and selected as a comparison target object.

8 illustrates spatial coordinate data for detecting a road surface object according to a method for automatically updating a road surface object by automatically determining and updating a change in a road surface object through an IPM image and a precision road map fitting according to an embodiment of the present invention. As a drawing, the ROI area to be IPMed in the probe vehicle is actually measured as a distance and location away from the GPS origin of the vehicle through preliminary measurement. Then, for each object position detected in the IPM image, the proportional position (Fig. 8(a)) is calculated through the GPS reception coordinates and the IMU heading information, and the HD map object spatial coordinates of the preliminary map information (Fig. 8(b) )), the position of the detected object and all objects within the GPS error range become the objects to be compared.

And, as an HD map object comparison and determination process, a base map can be generated through the GPS error range and the heading information of the IMU sensor in the above object detection process, and the database object corresponding to the base map and the detected object in the IPM image area comparison between them is possible.

9 is a diagram for explaining a method for automatically updating a road surface object by automatically determining and updating a change in a road surface object through an IPM image and a precision road map fitting according to an embodiment of the present invention, and a system for providing detection objects and map information. It is possible to determine whether there is no change, deletion, change, or creation of an object through comparison between objects in the database.

At this time, if the distance between the two comparison objects is within a certain distance within the error range by the RTK method GPS, it can be determined as the same object, and even when it is determined that the object is the same object based on the distance between the objects, objects such as lanes and arrows It is possible to determine whether the corresponding object has been changed by comparing the types of . In addition, if the detected object does not exist within a certain distance from the central point of the object in the database, it may be determined that the corresponding database object has been deleted.

Conversely, if an object in the database that is matched within a certain distance of the center point of the detected object does not exist, the corresponding detected object may be determined to be new (created).

10 is a diagram illustrating the update result of a road surface object according to the method of automatically updating a road surface object by automatically determining and updating a change of a road surface object through an IPM image and a precision road map fitting according to an embodiment of the present invention; The central arrow detected in (a) and (b) of Fig. 10 may be an object that does not exist in the base map of (c), and in this case, it may be determined as a new object. The spatial coordinates of the newly determined object may be calculated from a spatial coordinate relationship with the spatial coordinates or image coordinates of surrounding objects. That is, the spatial coordinates of the new judgment object may be calculated and obtained from a proportional expression for mutual coordinate distances using the spatial coordinates or image coordinates of surrounding objects.

Hereinafter, with reference to the drawings, the structure of the automatic updating system for automatically determining and updating road surface objects through IPM image and precision map fitting according to an embodiment of the present invention will be described in detail.

11 is a diagram showing an automatic updating system for updating a road surface object by automatically determining and updating a change in a road surface object through an IPM image and a precision map fitting according to an embodiment of the present invention.

Referring to FIG. 11, an automatic road map updating system for automatically determining and updating changes in road surface objects through IPM image and precision road map fitting according to an embodiment of the present invention includes one or more sensors mounted on a running vehicle. A vehicle information collection unit 110 that collects vehicle information including a front view image of the vehicle from the vehicle information collection unit 110, an image corrector 120 that generates a corrected image by correcting distortion by a camera existing in the front view image, and a corrected image. The image conversion unit 130 converts the image into an IPM image through inverse perspective mapping, obtains an HD map from the preliminary map information using the vehicle GPS positioning coordinates included in the vehicle information, extracts the IPM area from the HD map, and converts the IPM image to the IPM image. A map generator 140 that generates a corresponding base map, a map fitting unit 150 that fits the IPM image and the base map by correcting the pose of the IPM image through multi-directional correction of the IPM image, and the existing base map of the base map. It may include a determination unit 160 for determining whether or not to update the road surface object through comparison with the IPM image.

The vehicle information collection unit 110 is mounted on a vehicle and connected to a front view camera and GNSS, etc., and obtains information about the road by photographing the road while driving, thereby changing the map with precision provided by the map information providing system 200, etc. It is to automatically check and update the details, and various vehicle information can be collected from the camera, GNSS, and IMU of the vehicle in motion.

The image correction unit 120 may perform correction for the front view image included in the vehicle information, in particular, correction for distortion such as radial or tangential distortion occurring in the image according to the characteristics of the camera, internal and external related thereto. Parameters can be extracted and used for calibration.

The image conversion unit 130 may convert the image corrected through the adaptive binarization process into an IPM image by performing an IPM procedure of binarizing and projecting the image onto a plane.

The map generator 140 may generate a base map for an area corresponding to the IPM image. The collected vehicle information includes vehicle GPS positioning coordinates provided from GNSS, and the map generator 140 obtains an HD map from prior map information, extracts an area corresponding to the IPM image, and projects similarly to the IPM. You can create a base map.

The map fitting unit 150 may correct an image pose by comparing the IPM image to the base map and rotating it in a yaw direction, a horizontal direction, a vertical direction, and the like. At this time, Yaw is obtained from the IMU and corrected through lateral fitting, and longitudinal fitting may be performed by rotating the IPM image based on the difference between the lanes of the IPM image and the base map.

The determination unit 160 compares the IPM image and the base map, determines that road surface objects existing within the error range are the same object when the distance between the two compared objects is within a certain distance, and changes according to the type and existence of the object. , deletion or creation can be determined.

In addition, the automatic map renewal system 100 according to an embodiment of the present invention, when a new road surface object exists according to the comparison with the IPM image of the existing road surface object of the base map by the determination unit 160 It may further include a map updating unit that acquires the spatial coordinates of a new road surface object from the spatial coordinate relationship with surrounding road surface objects and adds the new road surface object to preliminary map information.

Such a map updating unit allows the map information built in the database built in the map information providing system 200 or the like to be rapidly and automatically updated according to the identification of the road surface object.

In particular, when the system according to the embodiment of the present invention is mounted on a plurality of vehicles, the reliability of automatic map renewal can be increased by collecting and updating information on road surface objects of the same spatial coordinates from a plurality of systems.

Although many details are specifically described in the above description, this should be interpreted as an example of a preferred embodiment rather than limiting the scope of the invention. Therefore, the invention should not be defined by the described examples, but should be defined by what is equivalent to the claims and claims.

100: Map automatic renewal system
110: vehicle information collection unit
120: image correction unit
130: image conversion unit
140: map generator
150: map fitting unit
160: judgment unit
200: Map information provision system

Claims (9)

As a map automatic renewal method by an automatic map renewal system,
Collecting vehicle information including a front view image of the vehicle from one or more sensors mounted on the running vehicle;
generating a corrected image by correcting distortion by a camera existing in the front view image;
converting the corrected image into an IPM image through inverse perspective mapping;
obtaining an HD map from preliminary map information using vehicle GPS positioning coordinates included in the vehicle information, extracting an IPM area from the HD map included in the preliminary map information, and generating a base map corresponding to the IPM image;
correcting the pose of the IPM image through multi-directional correction of the IPM image and fitting the IPM image and the base map; and
A precision road map that automatically determines and updates changes in road objects through IPM image and precision map fitting, including the step of determining whether to update through comparison with the IPM image of the existing road surface object of the base map. renewal method. According to claim 1,
Converting the corrected image into an IPM image through inverse perspective mapping,
Automatically determining and updating changes in road surface objects through IPM image and precision map fitting, including the step of recognizing segmented road surface objects by applying a deep learning recognition algorithm learned about road facilities to the front view image Map auto-update system with precision. According to claim 2,
Recognizing divided road surface objects by applying a deep learning recognition algorithm learned about road facilities to the front view image,
performing binarization on the front-view image through an adaptive binarization procedure using an average value application algorithm or a Gaussian distribution algorithm; and
Generating the IPM image that reconstructs the segmented road surface object by filtering only the outline-extracted object area, and automatically determining and updating the change of the road surface object through the IPM image and precision map fitting. renewal system. According to claim 1,
The step of fitting the IPM image to the base map by correcting the pose of the IPM image through multi-directional correction of the IPM image,
calculating yaw using an output value of an IMU sensor corresponding to the IPM image;
A first angle of a segment of the lane on both sides of the HD map (

₁ ,

₂ ) calculating;
The first angle (

₁ ,

₂ ) of the mean (

calculating _y );
The average value of the IPM image (

correcting the yaw by rotating as much as _y ); and
Yaw adjusts the angle and spacing of the lanes appearing in the calibrated IPM image to the lateral scale to perform lateral fitting, automatically determining the change of the road surface object through map fitting with precision with the IPM image, and Map auto-update system with renewing precision. According to claim 4,
After the step of performing lateral fitting by adjusting the angle and spacing of lanes appearing in the yaw-corrected IPM image with a lateral scale,
calculating a difference between an upper part (n _min ) and a lower part (m _min ) of a road surface object when the lanes of the IPM image and the base map are rotated in a vertical direction; and
Automatically determine the change of the road surface object through the IPM image and precision map fitting, including the step of performing longitudinal fitting using the calculated upper part difference (n _min ) and lower part difference (m _min ), Map auto-update system with renewing precision. According to claim 5,
The difference between the upper part of the road surface object (n _min ) and the lower part of the road surface object (m _min ) is expressed by the following equations,

It is calculated as (where n is a negative number), and the movement distance for performing the longitudinal fitting (

) is the following equation,

It is an auto-update map system that automatically determines and renews changes in road surface objects through IPM image and precision map fitting. According to claim 1,
After the step of determining whether to update the existing road surface object of the base map through comparison with the IPM image,
If there is a new road surface object, obtaining the spatial coordinates of the new road surface object from the spatial coordinate relationship with the surrounding road surface objects and adding them to the preliminary map information, through the IPM image and precision map fitting Map auto-update system with precision that automatically judges and updates changes. a vehicle information collection unit that collects vehicle information including a front view image of the vehicle from one or more sensors mounted on the running vehicle;
an image correction unit generating a correction image by correcting distortion caused by a camera existing in the front view image;
an image converter converting the corrected image into an IPM image through inverse perspective mapping;
a map generator that obtains an HD map from preliminary map information using vehicle GPS positioning coordinates included in the vehicle information, extracts an IPM area from the HD map, and generates a base map corresponding to the IPM image;
a map fitting unit for fitting the IPM image and a base map by correcting the pose of the IPM image through multi-directional correction; and
With respect to the existing road surface object of the base map, a precision road map that automatically determines and updates the change of the road surface object through the IPM image and precision map fitting, including a determination unit that determines whether to update through comparison with the IPM image renewal system. According to claim 8,
According to the comparison of the existing road surface object of the base map with the IPM image by the determination unit, if a new road surface object exists, the spatial coordinates of the new road surface object are obtained from the spatial coordinate relationship with the surrounding road surface objects, and the preliminary map An automatic map updating system with precision that automatically determines and updates changes in road surface objects through an IPM image and precision map fitting, further comprising a map update unit that adds information.

Images