KR102797025B1 - System and method for providing parking information based on 3d positioning through deep learning model - Google Patents

Abstract

According to the present disclosure, a system and method for providing parking information based on three-dimensional positioning through a deep learning model are provided. The system may include an image receiving unit configured to extract a still image frame from a parking lot capture image received from an image capturing device; a three-dimensional positioning inference unit configured to obtain three-dimensional positioning information about a vehicle in the image by performing deep learning inference on the still image frame; a positioning information correction unit configured to obtain corrected three-dimensional positioning information by performing correction on the three-dimensional positioning information based on optical correction information; and a parking information providing unit configured to provide parking information including occupancy information of a parking space in the parking lot based on ground occupancy coordinates obtained from the corrected three-dimensional positioning information.
According to the present disclosure, in providing parking information using a parking lot shooting video, it is possible to provide accurate information on not only hidden parking spaces but also vehicle locations through three-dimensional positioning and optical correction using a deep learning model.

Description

{SYSTEM AND METHOD FOR PROVIDING PARKING INFORMATION BASED ON 3D POSITIONING THROUGH DEEP LEARNING MODEL}

The present invention relates to a system and method for providing parking information based on three-dimensional positioning through a deep learning model.

Recently, attempts have been made to determine whether a parking lot is parked based on images captured by a camera device. These conventional parking determination methods generally identify the portion of the image occupied by the vehicle shape using deep learning or color/luminance processing, and determine whether or not a parking space is parked by calculating whether or not it intersects with each parking space boundary square or polygon.

In the case of these conventional methods of determining whether a vehicle is parked, a prerequisite is required that the shape of the vehicle does not encroach on adjacent parking spaces other than the parking space where the vehicle is parked.

However, in order to meet these prerequisites, a top-view type of captured image is required, which requires the presence of a fairly tall building in the parking lot, the installation of a large number of camera installation poles, or the operation of a filming drone for 24 hours, which is inefficient in terms of cost. In other words, the conventional parking judgment method using camera captured images has limitations in popularization because it is difficult to accept in terms of cost compared to the effect.

Meanwhile, as another method for determining the exact location of an object, a method of developing a deep learning model capable of estimating a 3D image from a 2D image can be considered. This deep learning model is a method of configuring object feature point location information and meta information obtained from a 2D image, and the actual 3D feature point location and meta information of the object as a training data set for the deep learning model, thereby allowing deep learning to estimate 3D feature point locations from a 2D image.

However, this method has a weak configuration that requires linking multiple models step by step and combining them up to the correction stage, and has an obvious limitation on the number of target objects that can be positioned (about 1 to 2), and can only be operated under ideal shooting conditions where the target object is fully exposed without being obscured by other objects.

Therefore, the aforementioned method also has clear limitations in analyzing images from parking control systems, which must film dozens of vehicles per camera and inevitably film significant portions of the vehicle bodies overlapping each other.

Korean Patent Registration No. 10-2161948 (2020.09.25)

To solve these problems, the present disclosure aims to provide a system and method for providing parking information based on three-dimensional positioning through a deep learning model.

According to the present disclosure, a system and method are provided that calculates a three-dimensional coordinate and direction of a vehicle by linking coordinates of feature points of a vehicle shape obtained through a deep learning model from a two-dimensional shooting image captured through an optical lens and optical correction information, and thereby precisely identifies vehicle occupancy coordinates on a parking lot ground plane, thereby identifying even hidden parking spaces and vehicle locations, and at the same time accurately determining whether a large number of vehicles are parked with a minimum number of cameras.

According to one embodiment of the present disclosure, a system for providing parking information based on three-dimensional positioning through a deep learning model is provided. The system may include an image receiving unit configured to extract a still image frame from a parking lot capture image received from an image capturing device; a three-dimensional positioning inference unit configured to obtain three-dimensional positioning information about a vehicle in the image by performing deep learning inference on the still image frame; a positioning information correction unit configured to obtain corrected three-dimensional positioning information by performing correction on the three-dimensional positioning information based on optical correction information; and a parking information providing unit configured to provide parking information including occupancy information of a parking space in the parking lot based on ground occupancy coordinates obtained from the corrected three-dimensional positioning information.

Additionally, the three-dimensional positioning information may include point or length information for representing spatial coordinates of the three-dimensional shape of the vehicle, or point or length information for representing two-dimensional coordinates in which the three-dimensional spatial coordinates are projected onto the still image frame.

Additionally, the optical correction information may include an object length change ratio or an object angle change ratio calculated based on a pan angle and a tilt angle of the still image frame.

In addition, the 3D positioning inference unit may be additionally configured to infer the optical correction information from the still image frame. Alternatively, the optical correction information may be obtained from a property of the image capturing device. Alternatively, the optical correction information may be determined based on both the inference of the 3D positioning inference unit and the property of the image capturing device.

In addition, the parking information providing unit may be configured to determine whether the vehicle occupies a corresponding parking space by comparing the intersection level of a polygon representing a portion occupied by a floor surface of a three-dimensional shape of the vehicle determined based on the ground occupancy coordinates or a polygon representing a position where the floor surface of the three-dimensional shape of the vehicle is projected onto the still image frame with a parking space boundary polygon.

In addition, corrected 3D positioning information for each vehicle in the still image frame can be acquired by the 3D positioning inference unit and the positioning information correction unit. The parking information providing unit can be additionally configured to provide parking information including parking lot congestion information, double-parked vehicle information, passageway moving vehicle information, temporarily stopped vehicle information, and illegally parked vehicle information, based on the corrected 3D positioning information for each vehicle in the still image frame.

Additionally, the 3D positioning inference unit may be configured to use a detection model or a segmentation model for deep learning inference.

According to one embodiment of the present disclosure, a method for providing parking information based on three-dimensional positioning through a deep learning model can be provided. The method can include the steps of: extracting a still image frame from a parking lot capture image received from an image capture device; performing deep learning inference on the still image frame to obtain three-dimensional positioning information on a vehicle in the image; performing correction on the three-dimensional positioning information based on optical correction information to obtain corrected three-dimensional positioning information; and providing parking information including occupancy information of a parking space in the parking lot based on ground occupancy coordinates obtained from the corrected three-dimensional positioning information.

Additionally, the three-dimensional positioning information may include point or length information for representing spatial coordinates of the three-dimensional shape of the vehicle, or point or length information for representing two-dimensional coordinates in which the three-dimensional spatial coordinates are projected onto the still image frame.

Additionally, the optical correction information may include an object length change ratio or an object angle change ratio calculated based on a pan angle and a tilt angle of the still image frame.

Additionally, the optical correction information may be inferred from the still image frame through the deep learning inference. Alternatively, the optical correction information may be obtained from the properties of the image capturing device. Alternatively, the optical correction information may be determined based on both the deep learning inference and the properties of the image capturing device.

In addition, the step of providing the parking information may include a step of determining whether the vehicle occupies the corresponding parking space by comparing the intersection level of a polygon representing a portion occupied by a floor surface of a three-dimensional shape of the vehicle determined based on the ground occupancy coordinates or a polygon representing a position where the floor surface of the three-dimensional shape of the vehicle is projected onto the still image frame with a parking space boundary polygon.

In addition, the step of acquiring the three-dimensional positioning information may include a step of acquiring three-dimensional positioning information for each vehicle in the still image frame, and the step of acquiring the corrected three-dimensional positioning information may include a step of acquiring corrected three-dimensional positioning information for each vehicle in the still image frame. The step of providing the parking information may further include a step of providing parking information including congestion information of the parking lot, information on double-parked vehicles, information on vehicles moving in an aisle, information on vehicles temporarily stopped, and information on illegally parked vehicles, based on the corrected three-dimensional positioning information for each vehicle in the still image frame.

Additionally, the deep learning inference can be performed using a detection model or a segmentation model.

According to one embodiment of the present disclosure, a computer program stored on a computer-readable medium including computer-executable instructions for executing a method for providing parking information based on three-dimensional positioning using a deep learning model can be provided.

According to the present disclosure, in providing parking information using a parking lot shooting video, it is possible to provide accurate information on not only hidden parking spaces but also vehicle locations through three-dimensional positioning and optical correction using a deep learning model.

In addition, according to the present disclosure, by utilizing images from cameras installed in a parking lot where vehicles inevitably overlap each other and are photographed, while recognizing concealed vehicle bodies and parking spaces, the guidance accuracy of a parking information provision system that provides guidance on the location of empty parking spaces and corresponding routes can be improved in a situation where, for example, there are few empty parking spaces and the remaining empty parking spaces are located between parked vehicles and are not easily visible, significantly increasing the driver's wandering time.

In addition, according to the present disclosure, the cost of constructing a parking information provision system can be minimized as the need for securing a high-altitude structure to secure a high shooting angle so that empty parking spaces are not concealed, the need for installing a large number of camera installation poles, and the need for densely installing cameras on installation poles, indoor parking lot pillars, ceiling raceways, etc. are eliminated.

In addition, according to the present disclosure, it is possible to provide three-dimensional positioning information and optical correction information for a captured image regardless of the type or model of the capturing device without generating a dewarping image in real time, and thus, existing image detection equipment such as a crime prevention CCTV can be utilized as is instead of expensive CCTV or a dedicated model CCTV, thereby having the advantage of being able to cost-effectively construct a parking information provision system in an existing parking lot while ensuring performance.

FIG. 1 is an exemplary drawing showing a parking lot to which a parking information providing system according to one embodiment of the present disclosure is applied.
FIG. 2 is a schematic diagram illustrating components of a parking information providing system according to one embodiment of the present disclosure.
FIG. 3 is an exemplary diagram showing the difference in parking judgment accuracy between a conventional two-dimensional inference model and a three-dimensional inference model according to the present disclosure.
FIG. 4 is an exemplary diagram showing how to provide parking information by calculating three-dimensional positioning information and optical correction information of a vehicle in cases where the angle of view, resolution, installation height, and parking space concealment ratio are different in a parking information providing system according to one embodiment of the present disclosure.
FIG. 5a is an exemplary drawing showing that the width length of a projected object shape is changed according to changes in the pan angle and tilt angle of an image capturing device.
Figure 5b is an exemplary graph showing the projection deformation ratio of the width length of an object shape according to changes in the pan angle and tilt angle of the video capturing device.
FIG. 6a is an exemplary drawing showing that the corner angle of a projected object shape is changed according to changes in the pan angle and tilt angle of the video capturing device.
Figure 6b is an exemplary graph showing the projection deformation ratio of the corner angle of the object shape according to changes in the pan angle and tilt angle of the video capturing device.
FIG. 7 is an exemplary diagram showing the result of matching the three-dimensional positioning correction result by the system according to one embodiment of the present disclosure to the ground coordinates of the analysis target parking lot.
FIG. 8 is an exemplary diagram showing parking information produced by a system according to one embodiment of the present disclosure.
FIG. 9 is an exemplary flow chart illustrating a method for providing parking information based on three-dimensional positioning via a deep learning model in a system according to one embodiment of the present disclosure.

Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the attached drawings. First, when adding reference numerals to components in each drawing, it should be noted that the same components are given the same numerals as much as possible even if they are shown in different drawings. In addition, when describing the present invention, if it is determined that a specific description of a related known configuration or function may obscure the gist of the present invention, the detailed description will be omitted.

Various aspects of the present invention are described below. It is to be understood that the inventions disclosed herein may be embodied in a wide variety of forms and that any specific structure, function, or both disclosed herein are merely exemplary. Based on the inventions disclosed herein, one skilled in the art will appreciate that one aspect disclosed herein may be implemented independently of any other aspects, and that two or more of these aspects may be combined in various ways. For example, an apparatus may be implemented or a method may be practiced using any number of the aspects disclosed herein. Furthermore, such an apparatus may be implemented or such a method may be practiced using other structures, functions, or structures and functions in addition to or other than one or more of the aspects disclosed herein.

FIG. 1 is an exemplary drawing showing a parking lot to which a parking information providing system according to one embodiment of the present disclosure is applied.

As illustrated in FIG. 1, the parking information providing system (200) according to the present disclosure receives parking lot shooting images from video recording devices (170-1, 170-2, 170-N) installed in the parking lot (100), and provides parking information within the parking lot (100) generated based on the parking lot shooting images to a corresponding parking lot management system, an external organization (300) (e.g., an upper control system, a linked service system, etc.). At this time, the parking information may include occupancy information of parking spaces (150-1, 150-2, 150-N) within the parking lot (100).

The video recording device (170) may be configured to record a video of a parking lot and provide the recorded video information to the parking information providing system (200). For example, the video recording device (170) may be a closed circuit television (CCTV) surveillance camera installed in the parking lot (100), but is not limited thereto and may be another type of device having a video recording function, such as a smart phone. The video recording device (170) may provide the recorded video to the parking information providing system (200) through a wired or wireless communication method, a method using a video signal, or a file storage medium.

The wired/wireless communication method between the parking information provision system (100) and the video recording device (170) may be a network communication method using a standardized video protocol such as RTSP (Real-Time Streaming Protocol), RTMP (Real-Time Messaging Protocol), HLS (Hypertext transfer protocol Live Streaming), etc., or a dedicated protocol provided by the video recording device (170) itself.

When transmitted over a network, for efficient traffic operation, the video data may be encoded using a standardized video compression codec such as H.264, H.265, MJPEG (Motion Joint Photographic Experts Group), or another video compression codec that has modified or added some technology thereto, or another alternative video compression codec, and transmitted to the parking information provision system (200).

When transmitting by video signal, the video data can be transmitted to the parking information provision system (200) through a video cable of an interface such as HDMI (High-Definition Multimedia Interface), DVI (Digital Visual Interface), HD-SDI (High Definition Serial Digital Interface), DisplayPort, VGA (Video Graphics Array), S-Video (Separate Video), etc.

FIG. 2 is a schematic diagram illustrating components of a parking information providing system according to one embodiment of the present disclosure.

As illustrated in FIG. 2, a parking information providing system (200) according to the present disclosure may be configured to include an image receiving unit (210), a preprocessing unit (220), a 3D positioning inference unit (250), a postprocessing unit (260), a positioning information correction unit (270), and a parking information providing unit (290).

The image receiving unit (210) may be configured to receive a captured image of the parking lot (100) from the image capturing device (170) and extract a still image frame from the received captured image of the parking lot (100). As described above, the image receiving unit (210) may be connected to the image capturing device (170) via wired or wireless communication, or may be connected via a video signal cable, in order to receive the captured image of the parking lot. Alternatively, the image receiving unit (210) may receive the captured image of the parking lot (100) from the image capturing device (170) via a file storage medium.

When receiving an image via a video signal cable, the image receiving unit (210) can directly receive the image of the parking lot (100) from the image shooting device (170) through the capture input port processing function. In addition, the image receiving unit (210) can separately designate the identifier of the input port used for capture. In addition, the image receiving unit (210) that has received the image of the parking lot (100) can change the brightness, contrast ratio, and sharpness values of the image of the parking lot (100) and then extract a still image frame. In addition, the image receiving unit (210) that has received the image of the parking lot (100) can individually exclude a certain number of pixels from the top, bottom, left, and right of the image of the parking lot (100) and then extract a still image frame.

When receiving an image through a network, the image receiving unit (210) can operate a buffering function for stable reception. When using the buffering function, the criterion can be selected based on the size of the time stored in the buffer or the timestamp of the received packet, etc. In addition, the image receiving unit (210) can designate an ID and password for network connection authentication with the image shooting device (170). Furthermore, when receiving an image from the image shooting device (170) through a relay device such as a proxy server, the image receiving unit (210) can also designate an ID and password for proxy communication, etc., along with the network address of the proxy device required for the relay.

When receiving an image through a network, the image receiving unit (210) can select whether to notify the image capturing device (170) of whether to maintain the connection, specify whether to request retransmission in case of a lost packet when receiving an image captured in the parking lot (100), and set a criterion for discarding the image when the reception of the image captured in the parking lot (100) is excessively delayed. In addition, the image receiving unit (210) can set information necessary for connection to the image capturing device (170) and path search of the image captured in the parking lot (100). This information can include information indicating the location of the device on the network, such as an IP address. In addition, the image receiving unit (210) can specify a detailed protocol to be used for receiving data through the network, such as TCP (Transmission Control Protocol), UDP (User Datagram Protocol), Unicast, and Multicast. In addition, the image receiving unit (210) can specify a path to a time server for aligning the time standard of the parking information providing system (200). In addition, the video receiving unit (210) can specify whether to use a standard specification for searching and communicating with a video capturing device, such as ON-VIF (Open Network Video Interface Forum). In addition, the video receiving unit (210) can specify a reference value for reconnection attempts when a communication connection is disconnected or a problem occurs.

When receiving an image through a file storage medium, the image receiving unit (210) can specify the path of the file to be received.

As described above, the image receiving unit (210) can extract still image frames from the image captured of the parking lot (100). At this time, if the received image captured of the parking lot (100) is an encoded compressed image, the image receiving unit (210) can perform decoding, and if it is not an encoded image, the decoding step can be omitted and the still image frame can be extracted. For decoding, a central processing unit such as a CPU (Central Processing unit) can be used, or a graphic processing unit advantageous for parallel processing such as a GPU (Graphics Processing unit) can be used. When a GPU is used, the image receiving unit (210) can use a GPU identifier such as GPU-ID to be used for decoding. In addition, the image receiving unit (210) can perform an automatic codec determination function so that the codec used for compression encoding can be extracted from the received information rather than the user's settings and automatically determined so that the corresponding decoding routine can be used. Additionally, the video receiving unit (210) can specify a range or interval period of compressed video frames not to be decoded when performing decoding.

The preprocessing unit (230) may be configured to preprocess the image frame extracted from the parking lot (100) shooting image so that it is suitable for processing by the 3D positioning inference unit (250). To this end, the preprocessing unit (230) may change the resolution of the image frame to a resolution suitable for 3D positioning inference. In addition, when the 3D positioning inference unit (250) performs inference using a GPU, the preprocessing unit (230) may determine the required memory size by using the number of input image streams to be processed simultaneously, the horizontal and vertical resolution sizes of the input image to be resized for inference, the number of color channels of the input image, etc., and may then secure memory space in the GPU equivalent to the corresponding memory size. In addition, when the 3D positioning inference unit (250) performs inference using a GPU, the preprocessing unit (230) may perform a task of changing the color format if the color format of the image transmitted in the previous step is different from the color format to be transmitted in the next step. Here, the change in color format may mean changing the format for displaying the colors of the image, such as changing the Red, Green, Blue, Alpha values to Blue, Green, Red values. Alternatively, the change in color format may mean changing the size of the data used, such as changing a 32-bit format to a 24-bit format. Alternatively, the change in color format may mean changing the frame image storage method, such as changing an interleaved format to a planar format.

The 3D positioning inference unit (250) can be configured to obtain 3D positioning information about a vehicle in an image based on a (preprocessed) still image frame. Specifically, the 3D positioning inference unit (250) can obtain 3D positioning information about a vehicle in an image by performing deep learning inference through a deep learning model (255).

In one embodiment, the three-dimensional positioning information may include point or length information for representing spatial coordinates of a three-dimensional shape of the vehicle (e.g., a rectangular solid), or point or length information for representing two-dimensional coordinates into which three-dimensional spatial coordinates are projected onto a still image frame.

For example, the three-dimensional positioning information of a vehicle may include specific points of the three-dimensional shape of the vehicle; or attribute values or specific points of a three-dimensional rectangular solid including the vehicle and having the length, width, and height of the vehicle as the lengths of each edge; or two-dimensional points in the form of three-dimensional space coordinates of the specific three-dimensional points projected onto a two-dimensional photographed image; or two-dimensional image points from which three-dimensional space coordinates can be obtained when combined with one or more of the angle of view or tilt values or the photographed position coordinates of the photographed image; or length values or angle values of specific line segments from which three-dimensional space coordinates can be obtained when combined with one or more of the angle of view or tilt values or the photographed position coordinates of the photographed image; or points from which a part of the list of coordinates of the points can be replaced with a combination of numbers indicating a direction or a size.

The 3D positioning inference unit (250) can perform inference on the received still image frame using a deep learning model (255) and store the result in the memory of the system or the memory of the GPU. In addition, the 3D positioning inference unit (250) can provide the user with an optional function that allows the user to select the type of deep learning model (255) to be applied. The deep learning model (255) that can be selected here may be a deep learning model that detects and distinguishes a specific object, such as a detection model, or a deep learning model that divides image or video information into multiple sets of pixels, such as a segmentation model, but is not limited thereto and may be another type of artificial neural network model that can perform deep learning inference. For example, the 3D positioning inference unit (250) may provide the user with a deep learning model (255) that can be selected, such as a YoLo (You only Look once) series deep learning model that divides an image frame into grid areas and calculates the probability of an estimated area of a proposed object in the image for each grid among detection models. In addition, for example, the 3D positioning inference unit (250) may provide the user with an instance segmentation model that obtains pixel set information in the form of a 2D prediction mask among segmentation models, such as a deep learning model (255).

In addition, the 3D positioning inference unit (250) may apply a method for reducing the redundancy of gradients in the optimization process of the deep learning neural network so as to operate the deep learning model in lightweight devices such as edge equipment by reducing the amount of inference computation. In addition, as another method for reducing the amount of inference computation, the 3D positioning inference unit (250) may apply a deep learning model improvement method for evenly distributing the amount of computation of each layer of the neural network to improve the bottleneck phenomenon of the computation. In addition, in order to reduce the amount of inference computation, the 3D positioning inference unit (250) may use a value obtained by quantifying the shape deformation characteristics that occur when a 3D object (e.g., a rectangular solid) for the shape of a vehicle is projected onto a 2D image as the output value of the deep learning model, thereby reducing the types of model output values.

Here, the value that quantifies the shape deformation characteristic may be a single value or a combination of multiple values such as an array. In addition, the quantified value of the shape deformation characteristic or some of its values may be a ratio value in which the angles between three edges that meet at a specific vertex of a three-dimensional solid surrounding the vehicle are perpendicular to each other, but when projected onto a two-dimensional image, the angles of the three edges are each transformed to a specific value between 0 and 180 degrees. In addition, the quantified value of the shape deformation characteristic or some of its values may be a mutual length ratio value between two specific edges among three edges in the length direction, height direction, and width direction of the vehicle in the three-dimensional solid surrounding the vehicle. In addition, the quantified value of the shape deformation characteristic or some of its values may be a unit vector or angle value in a two-dimensional space when the front direction of the vehicle in a three-dimensional space that can be expressed as a three-dimensional unit vector or an array of three-dimensional angle values is projected onto a two-dimensional image.

The post-processing unit (260) may be configured to post-process the inference result of the 3D positioning inference unit (250) to obtain 3D positioning information or optical correction information of a plurality of vehicle shapes existing within a still image frame.

The post-processing unit (260) can copy the inference results of the deep learning model (255) from the GPU to the host. In addition, the post-processing unit can perform appropriate post-processing according to the deep learning model (255).

For example, if the deep learning model (255) is a method for calculating the probability of an estimated region of an object in an image, such as the YoLo series, the processing of the post-processing unit (260) may include a process of leaving the most suitable estimated region through NMS (Non Maximum Suppression) processing using the IoU (Intersection over Union) value for the given estimated region. In addition, for example, if the deep learning model (255) is a method for obtaining pixel set information in the form of a two-dimensional prediction mask, such as the instance segmentation model, the processing of the post-processing unit (260) may include a process of converting the pixel set information into a binary mask method by processing it in a thresholding method or a CRF (Conditional Random Field) method.

In addition, the processing of the post-processing unit (260) may include a process of converting the estimated boundary line information into coordinates of a three-dimensional rectangular solid representing the three-dimensional position of the vehicle by utilizing binary mask information. In addition, the processing of the post-processing unit (260) may include a process of calculating an estimated type probability value of the target vehicle by utilizing the result value used for inference in the deep learning model (255) as a code value. In addition, the post-processing unit (260) may obtain a three-dimensional vector representing the direction of the vehicle, a direction angle value for a three-dimensional axis, or a two-dimensional vector or a direction angle value for a two-dimensional reference axis when the corresponding direction vector is photographed and projected onto a two-dimensional image.

The positioning information correction unit (270) may be configured to obtain corrected 3D positioning information by performing correction on the 3D positioning information based on the optical correction information. In one embodiment, the optical correction information may include an object length change ratio or an object angle change ratio calculated based on a pan angle and a tilt angle of a still image frame (see FIGS. 5 and 6 ). Depending on the implementation, the 3D positioning inference unit (250) may infer optical correction information from an input image frame and provide it to the positioning information correction unit (270), or the optical correction information may be obtained from properties of the image capturing device (170) (e.g., installation location, angle, field of view information, etc.) and provided to the positioning information correction unit (270), or the optical correction information may be determined based on both the inference of the 3D positioning inference unit (250) and the properties of the image capturing device (170).

In addition, the positioning information correction unit (270) can utilize the parking space (150) block coordinate information inferred by the deep learning model (255) or the parking space (150) block coordinate information set by the user to correct the 3D positioning information of the vehicle. In addition, the positioning information correction unit (270) can utilize the length or width or position information of the parking space (150) slot among the parking space (150) block coordinate information for correction. In addition, the positioning information correction unit (270) can utilize the length or width or position information of an arbitrary square where the parking space appears in the captured image among the parking space (150) block coordinate information for correction.

The following is an example of a 3D positioning information correction process that can be performed by the positioning information correction unit (270).

The length (Wc) of the minimum rectangular solid containing the vehicle in a three-dimensional space in the direction of the width of the vehicle perpendicular to the direction of the vehicle can be expressed as mathematical expression 1, which has as variables the distance ( _dc ) from the photographing device (170) to the subject, the frontal photographing ratio of the vehicle ( _rc ), the tilt amount ( _tc ) of the photographing device (170) when photographing the subject vehicle, and the zoom magnification ( _zc ) of the photographing device (170).

In addition, the width (W _s ) of the slot of the parking lot (150) in the three-dimensional space can be expressed in the same manner as the mathematical expression 2 having the distance (d _s ) from the shooting device (170) to the parking lot (150), the front shooting ratio (r _s ) of the parking lot (150) as the subject, the tilt amount (t _s ) of the shooting device (170) when shooting the subject parking lot (150), and the zoom magnification (Z _s ) of the shooting device (170).

Here, since the width of the vehicle and the width of the parking space (150) in the shooting image have a certain proportional relationship (k), the length in the vehicle width direction (W _c ) can be expressed as in mathematical expression 3.

In the function F(d,r,t,z), the distance d to the subject to be photographed is an independent variable unrelated to other variables, and the value of F(d,r,t,z) can change inversely proportional to the distance to the subject. In addition, the magnification z of the photographing device (170) is also an independent variable unrelated to other variables, and can cause the value of F(d,r,t,z) to change in direct proportion. Therefore, F(d,r,t,z) can be inferred as in mathematical expression 4.

Here, since the magnification of the photographing device (170) does not change, z _s = z _c is established, and J(z _s ) = J(z _c ) is established. In addition, since G(d) is a one-dimensional inverse proportional function, it can be expressed as G(d) = m/d through the constant m and the distance d.

Reflecting these results, the length in the width direction (Wc) can be re-expressed as in mathematical expression 5.

Here, the distance (d _c ) from the shooting device (170) to the subject and the distance (d _s ) from the shooting device (170) to the parking lot (150) can be calculated as (d s /d c ) using the position (x _p , y _p ) of the ground pole on which the shooting device (170) is installed, the position coordinates (x _s , y _s ) of the parking lot (150) _, and the position pixel coordinates ( _{x c ,} y _c ) of the vehicle, as in mathematical expression 6.

Here, the function H(r,t) is a function representing the change in the size of the object according to the pan and tilt angles (as shown in FIGS. 5a and 5b). The function H(r,t) can be expressed as a projection angle according to the pan angle (r) and the tilt angle (t) as in mathematical expression 7.

By combining mathematical expressions 5 to 7, it can be seen that information capable of correcting the width of a vehicle among 3D positioning information can be obtained by using optical correction information expressed by the following mathematical expression 8.

The following is an example of calculating a correction formula for a corner angle of a rectangular solid projected on a drawing among three-dimensional positioning information using the pan angle (r) and tilt angle (t) of an image capture device (170) that can be obtained from optical correction information (see FIGS. 6a and 6b).

The angle between each edge forming the vertex of a three-dimensional rectangular solid is vertical, but when the rectangular solid is photographed and projected onto a two-dimensional image, the angle between the line segments representing the corners is not vertical unless the shooting center and the angle match the vector expressing the corner. At this time, the rate at which the angle changes can be obtained through the pan angle (r) and the tilt angle (t) which can be obtained from the pixel information on the shooting image frame of the shooting device (170). The positioning information correction unit (270) can obtain the corrected position of the vertex of the three-dimensional rectangular solid from among the three-dimensional positioning information calculated by using this rate.

At this time, the rate at which the angle changes P(r,t) can be expressed as in mathematical expression 9 using the fan angle (r) and the tilt angle (t) (as confirmed in FIGS. 6a and 6b).

When the corrected 3D positioning information is acquired in the manner described above, the parking information providing unit (290) can be configured to acquire ground occupancy coordinates from the corrected 3D positioning information and provide parking information including occupancy information of a parking space (150) in the parking lot based on the acquired ground occupancy coordinates.

To this end, the parking information provider (290) can calculate coordinates projected on the ground plane of the parking lot from the corrected 3D positioning coordinates (i.e., ground occupancy coordinates). This ground plane may be a plane representing the floor of the parking lot (100). For example, the ground plane may be a ground plane in the case of a parking lot parked on an outdoor flat surface. In addition, in the case of an indoor parking structure consisting of multiple parking floors, the ground plane may be the floors of each parking floor. Accordingly, in the case of such an indoor parking structure, there may be multiple ground planes corresponding to multiple parking floors.

Additionally, the ground occupancy coordinates can be coordinates where the height of the object is set to 0, such as (x,y,0) in the corrected 3D positioning coordinates (x,y,z).

The parking information providing unit (290) can determine whether the vehicle occupies the corresponding parking space by comparing the intersection level of a polygon representing a portion occupied by the floor surface of a three-dimensional shape of the vehicle determined based on ground occupancy coordinates and a parking space boundary line polygon located on the floor surface of the parking lot. In addition, the parking information providing unit (290) can determine whether the vehicle occupies the corresponding parking space by comparing the intersection level of a polygon representing a position where the floor surface of the three-dimensional shape of the vehicle is projected onto a still image frame and a polygon representing a position where the parking space boundary line polygon is captured within a two-dimensional still image frame.

In addition, if the ground occupancy coordinates are derived from image frames captured by two or more capturing devices (170) for the captured vehicle, the shape indicated by the ground occupancy coordinates, such as some rectangles exemplified in FIG. 7, may also be displayed in duplicate since it is actually the same vehicle.

In addition, the corrected 3D positioning information for each vehicle in the still image frame can be obtained by the 3D positioning inference unit (250) and the positioning information correction unit (270) as described above. The parking information providing unit (290) can obtain various parking information in the parking lot through filtering according to the judgment of the same vehicle position and cross-comparing the filtered vehicle with the parking space (150) based on the corrected 3D positioning information for each vehicle in the still image frame. This various parking information can include parking lot congestion information, double-parked vehicle information, passageway moving vehicle information, temporarily stopped vehicle information, illegally parked vehicle information, etc. (see FIG. 8).

FIG. 3 is an exemplary diagram showing the difference in parking judgment accuracy between a conventional two-dimensional inference model and a three-dimensional inference model according to the present disclosure.

As illustrated in FIG. 3, it can be confirmed that the accuracy of parking information provided by the parking information providing system (200) using a deep learning model according to the present disclosure is greatly improved compared to the accuracy of parking information when a conventional two-dimensional area recognition deep learning model is applied. As described below, the system (200) according to the present disclosure can produce much more accurate results when performing occupancy judgment for each parking space and judgment of whether or not to park in a concealed parking space.

In conventional systems that provide parking information in a parking lot based on images captured by a shooting device, there has always been a problem in which vehicles in the captured images overlap each other and obscure each other.

The left side (310, 320, 330) of Fig. 3 shows a process of determining whether a vehicle is parked or not using a conventional two-dimensional area recognition deep learning model. In this conventional model, when matching a rectangle representing a vehicle area with each parking space (150), the exposed boundary line of the vehicle or the minimum rectangle including the exposed boundary line of the vehicle may occupy an adjacent parking space (150) other than the parking space (150) where the vehicle is parked, and in reality, it can be confirmed that a considerable number of empty parking spaces (150) are misjudged as being occupied by vehicles, and the error rate of determining whether a vehicle is parked or not exceeds 40% for each parking space (150).

On the other hand, the right side (340, 350, 360) of Fig. 3 shows a process of determining a full space by using three-dimensional positioning information in a system (200) according to the present disclosure. This system (200) can perform a judgment of an occupied parking space (150) and an empty parking space (150) for each parking space (150) by using three-dimensional positioning information, and further, can perform a judgment of whether a parking space (150) concealed by a vehicle is parked. As shown on the right side of Fig. 3, it can be confirmed that the parking information providing system (200) can accurately determine a full space even for a concealed parking space (150) by matching the floor surface of the corresponding vehicles and the parking space (150) after deriving three-dimensional positioning information of the vehicles.

FIG. 4 is an exemplary diagram showing how to provide parking information by calculating three-dimensional positioning information and optical correction information of a vehicle in cases where the angle of view, resolution, installation height, and parking space concealment ratio are different in a parking information providing system according to one embodiment of the present disclosure.

As illustrated in FIG. 4, it can be confirmed that the parking information providing system (200) according to the present disclosure can produce 3D positioning information and correction information even under conditions where shooting conditions change.

In conventional parking systems, when the angle of view of the shooting device is wide, optical distortion occurs in the shooting image, and a dewarping function is essential to correct this. If this dewarping function is applied, it increases the unit price of the shooting device, and there is a problem that both the original shooting image before dewarping processing and the shooting image after dewarping processing, which is the basis image for parking judgment, must be stored and managed.

On the other hand, the parking information providing system (200) according to the present disclosure can calculate the positioning information and the correction information by reflecting the change in the pan angle, tilt angle, and zoom, which are the shooting variables of the shooting device (170), in the 3D positioning inference unit (255) and the positioning information correction unit (270). Therefore, even for an image in which significant warping exists without dewarping processing, such as the first example image (410) and the fourth example image (440) illustrated in FIG. 4, the parking information providing system (200) according to the present disclosure can calculate the 3D positioning information and the correction information without having to perform the dewarping processing.

FIG. 5a is an exemplary drawing showing that the width length of a projected object shape is transformed according to changes in the pan angle and tilt angle of an image capturing device, and FIG. 5b is an exemplary graph showing a projection transformation ratio of the width length of an object shape according to changes in the pan angle and tilt angle of an image capturing device.

FIG. 6a is an exemplary drawing showing that a corner angle of a projected object shape is transformed according to changes in the pan angle and tilt angle of an image capturing device, and FIG. 6b is an exemplary graph showing a projection transformation ratio of a corner angle of an object shape according to changes in the pan angle and tilt angle of an image capturing device.

As shown in FIGS. 5a, 5b, 6a, and 6b, it can be confirmed that the size and angle of a photographed object (e.g., a vehicle) are transformed when projected from three dimensions to two dimensions according to changes in the pan angle (Pan(r)) and tilt angle (Tilt(t)) of the photographing device (170) that photographs the parking lot (100).

Furthermore, as illustrated in FIGS. 5b and 6b, it can be seen that as the pan angle (Pan(r)) increases between 0 and 90 degrees, the projection deformation ratio of the vehicle width decreases, but the projection deformation ratio of the corner angle does not significantly change. In addition, as illustrated in FIGS. 5b and 6b, it can be seen that as the tilt angle (Tilt(t)) increases between 0 and 90 degrees, the projection deformation ratio of the vehicle width decreases, and the projection deformation ratio of the corner angle increases.

FIG. 7 is an exemplary diagram showing the result of matching the three-dimensional positioning correction result by the system according to one embodiment of the present disclosure to the ground coordinates of the analysis target parking lot.

As illustrated in FIG. 7, the parking information providing system (200) according to the present disclosure can determine whether a parking space (150) in a parking lot (100) is occupied based on ground occupancy coordinates obtained based on captured images (i.e., still image frames) received from one or more image capturing devices (170).

For example, the parking information providing system (200) may display an area representing each parking space (150) on a display screen (710) for displaying the status of the target parking lot, and a polygon representing a portion occupied by the floor surface of the three-dimensional shape of the vehicle, or a polygon representing a position where the floor surface of the three-dimensional shape of the vehicle is projected onto a corresponding still image frame. At this time, a parking space (150) corresponding to a parking space area (730) where a polygon is not located may be determined as an unoccupied parking space (150), and a parking space (150) corresponding to a parking space area where a polygon (750) is located may be determined as an occupied parking space (150). Furthermore, although the same vehicle may be displayed as two polygons (750-1, 750-2) by two photographing devices (170) whose photographing areas overlap, the parking information providing system (200) may determine and process them as the same vehicle through filtering.

FIG. 8 is an exemplary diagram showing parking information produced by a system according to one embodiment of the present disclosure.

As illustrated in FIG. 8, the parking information providing system (200) according to the present disclosure can provide parking information of a corresponding parking lot based on the ground coordinate matching result as illustrated in FIG. 7. The parking information providing system (200) can provide parking information including not only occupancy information of each parking space in the corresponding parking lot, but also congestion information of the parking lot, double-parked vehicle information, passageway moving vehicle information, temporarily stopped vehicle information, illegally parked vehicle information, etc.

For example, a display screen (810) (e.g., a parking status display board, etc.) providing parking information of the corresponding parking lot (100) may display information on empty parking spaces (830) and occupied parking spaces (850), and information on congestion of the parking lot may be provided based on the ratio of occupied parking spaces to the total number of parking spaces, the number of vehicles located in passageways and entrances, etc. In addition, the display screen (810) may display information on vehicles (870) that are not located in the parking space (150) (e.g., in passageways, etc.). Such information on vehicles (870) located outside the parking space (170), etc. may be used to determine whether the vehicle is a double-parked vehicle, a vehicle moving in the passageway, a vehicle temporarily stopped, or an illegally parked vehicle by analyzing consecutive preceding and following still image frames.

For example, if a vehicle (870) A located outside a parking space (170) is moving along a passageway as a result of comparing still image frames from previous and previous time periods, the parking information providing system (200) may determine that vehicle A is a vehicle moving along the passageway. In addition, if a vehicle (870) B located outside a parking space (170) is stopped for a specific period of time (e.g., 20 minutes) or longer as a result of comparing still image frames from previous and previous time periods, the parking information providing system (200) may determine that vehicle B is an illegally parked vehicle.

FIG. 9 is an exemplary flow chart illustrating a method for providing parking information based on three-dimensional positioning via a deep learning model in a system according to one embodiment of the present disclosure.

As illustrated in FIG. 9, the parking information providing system (200) according to the present disclosure can receive a captured image of a parking lot (100) from a capturing device (170) (901). The parking information providing system (200) can extract a still image frame from the received captured image (902). The parking information providing system (200) can obtain three-dimensional positioning information about a vehicle in the image by preprocessing the extracted still image frame, performing deep learning inference on the still image frame, and performing postprocessing (903). The parking information providing system (200) can obtain corrected three-dimensional positioning information by performing correction on the inferred three-dimensional positioning information based on optical correction information (904). The parking information providing system (200) can obtain ground occupancy coordinates from the corrected three-dimensional positioning information (905). The parking information provision system (200) can provide parking information including occupancy information of a parking space (150) in a parking lot (100) based on acquired ground occupancy coordinates (906).

The method for providing parking information based on the three-dimensional positioning and optical correction information through the deep learning model described above can be implemented by a computer program. The computer program can include not only functional modules for providing parking information based on the three-dimensional positioning and optical correction information through the deep learning model according to the present disclosure, but also computer-executable instructions therefor. The computer-executable instructions of the computer program, when executed by a processor, can cause the processor to perform the method for providing parking information based on the three-dimensional positioning through the deep learning model of the present disclosure.

It is to be understood that any particular order or hierarchy of steps in any of the presented processes is an example of exemplary approaches. It is to be understood that, based on design priorities, the particular order or hierarchy of steps in the processes may be rearranged within the scope of the present invention. The appended method claims provide elements of the various steps in an exemplary order, but are not meant to be limited to the particular order or hierarchy presented.

The terms "component," "unit or part," "module," "system," and the like, as used herein, may refer to a computer-related entity, hardware, firmware, software, a combination of software and hardware, or an execution of software. For example, a component may be, but is not limited to, a process running on a processor, a processor, an object, a thread of execution, a program, and/or a computer. For example, both an application running on a computing device and the computing device may be a component. One or more components may reside within a processor and/or thread of execution, and a component may be localized on one computer or distributed between two or more computers. Additionally, these components may execute from various computer-readable media having various data structures stored therein.

The description of the disclosed embodiments is provided to enable any person skilled in the art to make or use the invention. Various modifications to these embodiments will be apparent to those skilled in the art, and the generic principles defined herein may be applied to other embodiments without departing from the scope of the invention. Thus, the invention is not intended to be limited to the disclosed embodiments, but is to be construed in the widest scope consistent with the principles and novel features disclosed herein.

100 : Parking lot
150 : Parking space
170 : Shooting device
200: Parking Information System
210 : Video receiver
230 : Preprocessing section
250: 3D positioning inference unit
255: Deep Learning Model
260 : Post-processing section
270: Positioning information correction unit
290: Parking Information Provider
300 : External agency

Claims (15)

A system for providing parking information based on 3D positioning through a deep learning model.
An image receiving unit configured to extract a still image frame from a parking lot image captured from a video capturing device;
A 3D positioning inference unit configured to obtain 3D positioning information about a vehicle in an image by performing deep learning inference on the still image frame;
A positioning information correction unit configured to obtain corrected 3D positioning information by performing correction on the 3D positioning information based on optical correction information; and
A parking information providing unit configured to provide parking information including occupancy information of a parking space within the parking lot based on ground occupancy coordinates obtained from the above-mentioned corrected 3D positioning information,
The optical correction information includes a length change ratio or an angle change ratio of each vehicle within the still image frame, which is calculated based on a pan angle and a tilt angle of the still image frame.
Parking information provision system. In paragraph 1,
The above three-dimensional positioning information includes point or length information for representing the spatial coordinates of the three-dimensional shape of the vehicle, or point or length information for representing the two-dimensional coordinates projected onto the still image frame of the three-dimensional shape.
Parking information provision system. delete In paragraph 1,
The above 3D positioning inference unit is additionally configured to infer the optical correction information from the still image frame, or
The above optical correction information is obtained from the properties of the image capturing device, or
The above optical correction information is determined based on both the inference of the 3D positioning inference unit and the properties of the image capturing device.
Parking information provision system. In paragraph 1,
The above parking information provider is,
A method configured to determine whether the vehicle occupies a parking space by comparing the intersection level of a polygon representing a portion occupied by the floor surface of the three-dimensional shape of the vehicle based on the ground occupancy coordinates or a polygon representing a position where the floor surface of the three-dimensional shape of the vehicle is projected onto the still image frame with a parking space boundary polygon,
Parking information provision system. In paragraph 5,
Corrected 3D positioning information for each vehicle in the still image frame is obtained by the 3D positioning inference unit and the positioning information correction unit,
The above parking information provider is,
Based on the corrected 3D positioning information for each vehicle in the still image frame, the system is additionally configured to provide parking information including parking lot congestion information, double-parked vehicle information, passageway moving vehicle information, temporarily stopped vehicle information, and illegally parked vehicle information.
Parking information provision system. In paragraph 1,
The above 3D positioning inference unit is configured to use a detection model or a segmentation model for deep learning inference.
Parking information provision system. A method for providing parking information based on three-dimensional positioning using a deep learning model.
A step of extracting still image frames from parking lot footage received from a video capturing device;
A step of obtaining three-dimensional positioning information about a vehicle in an image by performing deep learning inference on the still image frame;
A step of obtaining corrected 3D positioning information by performing correction on the 3D positioning information based on optical correction information; and
A step of providing parking information including occupancy information of a parking space within the parking lot based on ground occupancy coordinates obtained from the above-mentioned corrected 3D positioning information ,
The optical correction information includes a length change ratio or an angle change ratio of each vehicle within the still image frame, which is calculated based on a pan angle and a tilt angle of the still image frame.
How to provide parking information. In Article 8,
The above three-dimensional positioning information includes point or length information for representing the spatial coordinates of the three-dimensional shape of the vehicle, or point or length information for representing the two-dimensional coordinates projected onto the still image frame of the three-dimensional shape.
How to provide parking information. delete In Article 8,
The optical correction information is inferred from the still image frame through the deep learning inference, or
The above optical correction information is obtained from the properties of the image capturing device, or
The above optical correction information is determined based on both the deep learning inference and the properties of the image capturing device.
How to provide parking information. In Article 8,
The steps for providing the above parking information are:
A step of determining whether the vehicle occupies the parking space by comparing the intersection level of a polygon representing a portion occupied by the floor surface of the three-dimensional shape of the vehicle determined based on the ground occupancy coordinates or a polygon representing a position where the floor surface of the three-dimensional shape of the vehicle is projected onto the still image frame with a parking space boundary polygon,
How to provide parking information. In Article 12,
The step of obtaining the above 3D positioning information includes the step of obtaining 3D positioning information for each vehicle within the still image frame,
The step of obtaining the corrected 3D positioning information includes the step of obtaining the corrected 3D positioning information for each vehicle within the still image frame,
The steps for providing the above parking information are:
Further comprising a step of providing parking information including parking lot congestion information, double-parked vehicle information, passageway moving vehicle information, temporarily stopped vehicle information, and illegally parked vehicle information based on the corrected three-dimensional positioning information for each vehicle in the still image frame.
How to provide parking information. In Article 8,
The above deep learning inference is performed using a detection model or a segmentation model.
How to provide parking information. A computer program stored on a computer-readable medium, comprising computer-executable instructions for executing a method according to any one of claims 8, 9 and 11 to 14.