KR102310446B1 - Apparatus for identifying landmarks irrespective of location change based on deep learning model and method therefor - Google Patents

Abstract

Provided is a device for identifying a landmark. The device includes: a data processing unit for receiving an image and location information at which the image was photographed from a user device; and an identification unit which calculates a probability indicating whether a learned landmark exists in the image through an identification model including an identification network corresponding to the location information, identifies the landmark according to the probability, and transmits name and description of the identified landmark to the user device.

Description

Apparatus for identifying landmarks irrespective of location change based on deep learning model and method therefor

The present invention relates to a technique for identifying a landmark in an image, and more particularly, to an apparatus for identifying a landmark regardless of a location change based on a deep learning model, and a method therefor.

One of New York City's landmarks, the Brooklyn Bridge connects the Manhattan and Brooklyn Boroughs. However, near the Brooklyn Bridge, there is the Manhattan Bridge that connects the Manhattan and Brooklyn areas, and the shape of the bridge is also similar. Many cases exist.

Korea Patent Publication No. 2015-0026535 (published on March 11, 2015)

It is an object of the present invention to provide an apparatus and a method for identifying a landmark regardless of a change in location based on a deep learning model.

A method for identifying a landmark regardless of a change in location according to a preferred embodiment of the present invention for achieving the object as described above includes the steps of: Calculating, by an identification unit, a probability indicating whether a learned landmark exists in the image through an identification model including an identification network corresponding to the location information, an identification unit identifying the landmark according to the probability, and transmitting the name and description of the identified landmark to the user device.

Calculating the probability includes: generating a feature map by the backbone network of the identification model through a convolution operation of a plurality of convolutional layers on the image; Deriving one or more regions of interest through the step of, the identification unit loading the learned identification network to identify the landmark located in the shortest distance from the location information, the identification network is the feature map and the region box and calculating a probability indicating whether a learned landmark exists in the area box through a plurality of calculations in which a plurality of inter-layer weights are applied to the feature map.

의 하이퍼파라미터를 설정하는 단계와, 상기 학습용 영상 배치 각각에 포함된 학습용 영상에 대한 레이블을 설정하는 단계와, 상기 모델생성부가 학습용 영상을 식별모델에 입력하는 단계와, 상기 식별모델이 상기 학습용 영상에 대해 복수의 계층 간의 가중치가 적용되는 복수의 연산을 수행하여 출력값을 산출하는 단계와 상기 모델생성부가 상기 손실 함수를 통해 산출된 출력값과, 상기 레이블과의 차이인 손실이 최소가 되도록 식별모델의 가중치를 수정하는 단계를 포함하며, 상기 L은 손실이고, 상기

및 상기

는 하이퍼파라미터이며,

이고, 상기 i는 학습용 영상에 대응하는 인덱스이고, 상기 yi는 i번째 학습용 영상에 대한 레이블이고, 상기 f(xi)는 i번째 학습용 영상에 대한 상기 식별망의 출력값이고, 상기

및 상기

는 학습용 영상 배치에 포함된 복수의 학습용 영상에서 상기 데이터처리부에 의해 수집된 영상 및 사용자장치가 전송한 영상 각각이 차지하는 비율에 따라 값이 설정되는 것을 특징으로 한다. Before the step of receiving the image and the location information at which the image was taken, the step of the model generating unit providing a training image arrangement including a plurality of learning images, and the model generating unit losing function

setting hyperparameters of, setting a label for a training image included in each arrangement of the training image, and inputting the training image by the model generator into an identification model, wherein the identification model is the training image Calculating an output value by performing a plurality of calculations to which a weight between a plurality of layers is applied to , and a loss that is a difference between the output value calculated through the loss function by the model generator and the label and the label is minimized. modifying the weights, wherein L is the loss, and

and said

is a hyperparameter,

, wherein i is an index corresponding to the training image, yi is a label for the i-th training image, and f(xi) is an output value of the identification network for the i-th training image, and

and said

It is characterized in that the value is set according to the ratio of each of the image collected by the data processing unit and the image transmitted by the user device in the plurality of learning images included in the training image arrangement.

An apparatus for identifying a landmark according to a preferred embodiment of the present invention for achieving the object as described above includes a data processing unit for receiving an image and location information at which the image is taken from a user device, and identification corresponding to the location information Calculate a probability indicating whether or not a learned landmark exists in the image through an identification model including a network, identify the landmark according to the probability, and transfer the name and description of the identified landmark to the user device It includes an identification unit to transmit.

According to the present invention, it is possible to precisely identify a landmark regardless of where the user takes the picture.

1 is a view for explaining the configuration of a system for identifying a landmark regardless of a change in location based on a deep learning model according to an embodiment of the present invention.
2 is a view for explaining the configuration of an apparatus for identifying a landmark regardless of a change in location based on a deep learning model according to an embodiment of the present invention.
3 is a view for explaining the detailed configuration of an apparatus for identifying a landmark regardless of a location change based on a deep learning model according to an embodiment of the present invention.
4 is a view for explaining the overall configuration of the identification model for identifying landmarks regardless of location changes based on the deep learning model according to an embodiment of the present invention.
5 and 6 are diagrams for explaining the detailed configuration of an identification model for identifying a landmark regardless of a change in location based on a deep learning model according to an embodiment of the present invention.
7 is a flowchart illustrating a method of preparing an image arrangement for learning according to an embodiment of the present invention.
8 is a diagram for explaining a vector space for preparing an image arrangement for learning according to an embodiment of the present invention.
9 is a flowchart illustrating a method of generating an identification model using an image arrangement for training according to an embodiment of the present invention.
10 is a flowchart for explaining a method for identifying a landmark regardless of a location change based on a deep learning model according to an embodiment of the present invention.
11 and 12 are diagrams for explaining a method for identifying a landmark regardless of a location change based on a deep learning model according to an embodiment of the present invention.

Prior to the detailed description of the present invention, the terms or words used in the present specification and claims described below should not be construed as being limited to their ordinary or dictionary meanings, and the inventors should develop their own inventions in the best way. For explanation, it should be interpreted as meaning and concept consistent with the technical idea of the present invention based on the principle that it can be appropriately defined as a concept of a term. Accordingly, the embodiments described in this specification and the configurations shown in the drawings are only the most preferred embodiments of the present invention, and do not represent all of the technical spirit of the present invention, so various equivalents that can be substituted for them at the time of the present application It should be understood that there may be water and variations.

Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings. In this case, it should be noted that the same components in the accompanying drawings are denoted by the same reference numerals as much as possible. In addition, detailed descriptions of well-known functions and configurations that may obscure the gist of the present invention will be omitted. For the same reason, some components are exaggerated, omitted, or schematically illustrated in the accompanying drawings, and the size of each component does not fully reflect the actual size.

First, a system for identifying landmarks regardless of location changes based on a deep learning model according to an embodiment of the present invention will be described. 1 is a view for explaining the configuration of a system for identifying a landmark regardless of a change in location based on a deep learning model according to an embodiment of the present invention.

Referring to Figure 1, a system for identifying a landmark regardless of a change in location based on a deep learning model according to an embodiment of the present invention (hereinafter, abbreviated as 'identification system') is an identification server 10 and a plurality of of the user device 20 .

The identification server 10 basically, when receiving the image and the location information of the place where the image was shot from the user device 20, identifies the landmark in the received image, and includes the name of the landmark Related information is provided to the user device 20 . In addition, the identification server 10 may provide information about a nearby tourist destination, a restaurant, etc. based on the identified landmark.

The user device 20 is a device used by a user who has subscribed to the service provided by the identification server 10 . The user device 20 captures an image of an object estimated as a landmark by the user according to the user's manipulation, and obtains location information (eg, GPS signal) at the time of shooting to identify the captured image and the location information to the identification server ( 10) is uploaded. In addition, information about the landmark included in the image and information around the landmark may be received from the identification server 10, and the information may be output for the user to read. The user device 20 may be exemplified by a mobile phone, a mobile phone, a mobile communication terminal, a smart phone, a PDA, a tablet, a phablet, and the like.

Then, the configuration of the above-described identification server 10 will be described in more detail. 2 is a view for explaining the configuration of an apparatus for identifying a landmark regardless of a change in location based on a deep learning model according to an embodiment of the present invention. 3 is a view for explaining the detailed configuration of an apparatus for identifying a landmark regardless of a location change based on a deep learning model according to an embodiment of the present invention. Referring to FIG. 2 , the identification server 10 according to an embodiment of the present invention includes a communication module 11 , a storage module 12 , and a control module 13 .

The communication module 11 is for communicating with the user device 20 through a network. The communication module 11 may transmit/receive data to and from the user device 20 . The communication module 11 may include a modem that modulates a signal to be transmitted and demodulates a received signal in order to transmit/receive data through a network. The communication module 11 may transmit data received from the control module 13 through the network. In addition, the communication module 11 may transmit data received through the network to the control module 13 .

The storage module 12 serves to store programs and data necessary for the operation of the identification server 10 . For example, the storage module 12 may store an image for learning according to an embodiment of the present invention. These learning images may be received from the user device 20 or collected on the web. Various types of data stored in the storage module 12 may be registered, deleted, changed, or added according to the operation of the administrator.

The control module 13 may control the overall operation of the identification server 10 and the signal flow between internal blocks of the identification server 10, and may perform a data processing function of processing data. The control module 13 may be a central processing unit (CPU), a graphic processing unit (GPU), a digital signal processor (DSP), or the like.

Referring to FIG. 3 , the control module 13 includes a data processing unit 100 , a model generation unit 200 , and an identification unit 300 .

The data processing unit 100 may access various sites on the web and collect images interposed with the names of landmarks. In addition, the data processing unit 100 may access a search site, search for an image using the name of the landmark as a search word, and collect a plurality of images according to the order of reliability or accuracy. The data processing unit 100 classifies the collected images by landmarks and stores them in the storage module 12 .

The model generator 200 is to generate an identification model RM, which is a deep learning model (DLM), through deep learning. The operation of the model generating unit 200 will be described in more detail below.

The identification unit 300 is for identifying the landmark included in the image transmitted by the user device 20 using the identification model RM generated by the model generation unit 300 . The operation of the identification unit 300 will be described in more detail below.

Next, an identification model for identifying a landmark regardless of a change in location based on the deep learning model according to an embodiment of the present invention will be described. 4 is a view for explaining the overall configuration of the identification model for identifying landmarks regardless of location changes based on the deep learning model according to an embodiment of the present invention. 5 and 6 are diagrams for explaining the detailed configuration of an identification model for identifying a landmark regardless of a change in location based on a deep learning model according to an embodiment of the present invention.

5 and 6 , when an image is input, the identification model RM determines whether a learned landmark exists in the image through a plurality of operations in which the learned weights of a plurality of layers are applied to the image. output as This identification model (RM) includes a backbone network (Back Bone network: BN), a region detection network (Region Proposal Network: RPN), and an identification network (RN: Recognition Network).

The backbone network (BN) includes a plurality of convolution layers, and generates and outputs a feature map (FM) through convolution operation of a plurality of convolution layers (CL) for an input image. .

The region detection network (RPN) derives at least one region of interest (ROI) indicating a region in which the probability of the existence of an object is greater than or equal to a predetermined value in the feature map (FM) through a bounding box (BB). The area box BB is a rectangle, and is expressed by a center coordinate, a width w and a height h based on the center coordinates (x, y).

The identification network (RN) receives the feature map (FM) from the backbone network (BN), receives the region box (BB) representing the region of interest from the region detection network (RPN), and the land learned in the region box (BB) Calculate the probability of whether a mark exists or not, and output it.

The identification network (RN) may typically be exemplified by a Convolutional Neural Network (CNN). As shown in FIG. 5 , the identification network RN sequentially includes an input layer (IL), a convolution layer (CL), a pooling layer (PL), and a fully-connected layer. layer: FL) and an output layer (OL). Here, each of the convolutional layer CL, the pooling layer PL, and the fully connected layer FL may be two or more. The convolutional layer CL and the pooling layer PL are composed of at least one feature map (FM). The feature map FM is derived as a result of receiving a value applied with a weight and a threshold to the operation result of the previous layer, and performing an operation on the input value. These weights are applied through a filter or kernel that is a weight matrix of a predetermined size. In the embodiment of the present invention, the first filter W1 is used for the convolution operation of the convolutional layer CL, and the second filter W2 is used for the pooling operation of the pooling layer PL.

When the feature map FM is input from the backbone network BN to the input layer IL, the convolution layer CL performs convolution using the first filter W1 on the feature map FM of the input layer IL. At least one first feature map FM1 is derived by performing a (convolution) operation and an operation by an activation function. Subsequently, the pooling layer PL performs a pooling or sub-sampling operation using the second filter W2 on at least one first feature map FM1 of the convolutional layer CL to obtain at least one A second feature map FM2 is derived.

The final connection layer FL includes a plurality of operation nodes E1 to En. Referring to FIG. 6 , the plurality of operation nodes E1 to En of the final interconnection layer FL are calculated by an activation function for at least one second feature map FM2 of the pooling layer PL. Calculate the operation node value E[e1, e2, e3, ..., en].

As shown in FIG. 6 , the output layer OL includes two output nodes O1 and O2. Each of the plurality of operation nodes E1 to En of the fully connected layer FL is connected to the output nodes O1 and O2 of the output layer OL through a channel (indicated by a dotted line) having a weight w: weight. In other words, the plurality of operation node values e1, e2, e3, ..., en of the plurality of operation nodes E1 to En are inputted to the output nodes O1 and O2, respectively. Accordingly, the output nodes O1 and O2 of the output layer OL are calculated by the activation function for a plurality of operation node values e1, e2, e3, ..., en to which the weight is applied to the input layer IL. ), in response to the input feature map (FM), it is calculated with a probability whether the object in the area box is a landmark.

Each of the two output nodes O1 and O2 of the output layer OL corresponds to a case in which an object in an area box indicating a range of the region of interest ROI is a landmark (S) and a case where it is not a landmark (F). Accordingly, the output value of the first output node O1 represents the probability that an object in the region box indicating the range of the region of interest (ROI) is a landmark, and the output value of the second output node O2 is the corresponding region of interest (ROI). It represents the probability that the object in the area box representing the range of is not a landmark. For example, each of the output nodes O1, O1 has a weight w=[w1, w2, ... , wn], and an activation function is applied to the result to calculate an output value. For example, if the respective output values of the first and second output nodes O1 and O2 are 0.122 and 0.878, respectively, the probability of being a landmark is 12%, and the probability of not being a landmark is 88%. Output values of the first and second output nodes O1 and O2 of the output layer OL become output values of the identification network RN and the identification model RM. If the identification network (RN) outputs the probability (0.122, 0.878), the identification unit 300 knows that the probability of being a landmark is 12% and the probability that it is not a landmark is 88% through these probabilities (0.122, 0.878). It can be determined that the object in the area box is not a landmark according to the probability.

Activation functions used in the above-described convolutional layer (CL), final connection layer (FL) and output layer (OL) are Sigmoid, Hyperbolic tangent (tanh), Exponential Linear Unit (ELU), and ReLU. (Rectified Linear Unit), Leakly ReLU, Maxout, Minout, Softmax, and the like can be exemplified. Any one of these activation functions may be selected and applied to the convolutional layer CL, the final connection layer FL, and the output layer OL.

Next, a method of generating the identification model RM as described above through deep learning will be described. First, it is necessary to prepare data for learning for deep learning. A method of preparing such training data will be described. 7 is a flowchart illustrating a method of preparing an image arrangement for learning according to an embodiment of the present invention. 8 is a diagram for explaining a vector space for preparing an image arrangement for learning according to an embodiment of the present invention.

Referring to FIG. 7 , the model generator 200 collects a plurality of images including a landmark to be learned among a plurality of images stored in the storage module 12 in step S110 . As described above, the data processing unit 100 may access various sites on the web and collect images interposed with the names of landmarks. In addition, the data processing unit 100 may access a search site, search for an image using the name of the landmark as a search word, and collect a plurality of images according to the order of reliability or accuracy. The data processing unit 100 classifies the collected images by landmarks and stores them in the storage module 12 . In addition, an image classified as including a corresponding landmark among images uploaded by the user is also stored in the storage module 12 . Accordingly, in step S110 , the model generator 200 may collect a plurality of images including a landmark to be learned among a plurality of images stored in the storage module 12 . Then, the model generator 200 performs a plurality of convolution operations on the image extracted in step S120 to embed the plurality of convolutional images in the multidimensional vector space VS to generate a plurality of image vectors.

Next, the model generator 200 randomly selects a center vector of a predetermined number of clusters from among a plurality of image vectors in the multidimensional vector space VS in step S130 . Next, the model generator 200 clusters a plurality of image vectors according to the distance from the center vector of a predetermined number of clusters in the multidimensional vector space VS as shown in FIG. 8 in step S140 to form a plurality of clusters. create That is, in step S140 , the model generator 200 calculates a distance to each center vector of a predetermined number of clusters of a plurality of image vectors in the vector space VS, and divides each of the plurality of image vectors with the center vector of the cluster. Include the cluster with the smallest distance. Then, the model generator 200 re-calculates the center vector of the cluster for each of the plurality of clusters generated in step S150 . For example, in step S150 , the model generator 200 may select an image vector having an intermediate value among a plurality of image vectors of each cluster as a center vector.

Next, the model generator 200 checks whether all the image vectors are included in the cluster having the minimum distance from the center vector of the cluster calculated above ( S150 ) in step S160 .

As a result of checking in step S160, if all the image vectors are not included in the cluster having the minimum distance to the center vector of the cluster calculated above (S150), the model generator 200 repeats steps S140 to S160 described above. .

That is, the data processing unit 100 repeats steps S140 to S160 until the following Equation 1 becomes the minimum.

는 n번째 영상벡터가 k번째 클러스터에 속하면 1, 그렇지 않으면 0을 가지는 플래그 변수를 나타낸다. In Equation 1, k is the index of the cluster, and Xk is the center vector of the k-th cluster. n is the index of the image vector, and Un means the nth image vector.

denotes a flag variable having 1 if the n-th image vector belongs to the k-th cluster, and 0 otherwise.

On the other hand, as a result of checking in step S160, if all the image vectors are included in the cluster having the minimum distance from the center vector of the cluster calculated previously (S150), the model generator 200 determines the current cluster as the optimized cluster. do.

Then, as shown in FIG. 8 for each of the plurality of clusters optimized in step S180 , the model generator 200 generates a plurality of images included within a predetermined radius R from the center CX of the cluster within the same cluster. Extract the vector. That is, the model generator 200 forms an area FA having a predetermined radius R from the center CX of the cluster in any one cluster, and extracts a plurality of image vectors in the formed area. Then, the model generation unit 200 extracts a plurality of images corresponding to the plurality of image vectors extracted in step S190 as a plurality of images for learning, and generates a batch of images for learning including the plurality of images for learning. do. As described above, outliers can be stably removed by extracting a plurality of images corresponding to the image vectors in the area FA having a predetermined radius R from the center CX of the cluster as training images.

Then, a method for learning the identification network (RN) using the training image arrangement will be described. 9 is a flowchart illustrating a method of learning an identification network (RN) using a batch of images for training according to an embodiment of the present invention. In the embodiment of the present invention of FIG. 9 , the model generator 200 performs learning through a loss function as shown in Equation 2 below.

및

는 하이퍼파라미터이며,

이다. 특히, 하이퍼라미터

및

는 학습용 영상 배치에 포함된 복수의 학습용 영상에서 데이터처리부(100)에 의해 수집된 영상 및 사용자장치(20)가 전송한 영상 각각이 차지하는 비율에 따라 그 값이 설정된다. 또한, i는 학습용 영상에 대응하는 인덱스이다. yi는 i번째 학습용 영상에 대한 레이블이고, f(xi)는 i번째 학습용 영상에 대한 식별망(RN)의 출력값이다. In Equation 2, L means loss.

and

is a hyperparameter,

am. In particular, hyperparameters

and

The value of is set according to the ratio of each of the images collected by the data processing unit 100 and the images transmitted by the user device 20 in the plurality of learning images included in the training image arrangement. Also, i is an index corresponding to an image for training. yi is a label for the ith training image, and f(xi) is an output value of the identification network (RN) for the ith training image.

Referring to FIG. 9 , the model generator 200 prepares a batch of images for training in step S210 . A specific method of preparing an image arrangement for learning has been previously described with reference to FIGS. 7 and 8 .

및

의 값을 설정한다. 손실함수의 첫 번째 항은 특이치(Outlier)에 강인한(robust) 특성을 가지며, 두 번째 항은 특이치(Outlier)에 민감한(sensitive) 특성을 가진다. 이에 따라, 데이터처리부(100)에 의해 수집된 영상의 경우, 검증이 이루어진 영상이 아니기 때문에 특이치(Outlier)가 존재할 가능성이 높다. 따라서 학습용 영상 중 데이터처리부(100)에 의해 수집된 영상의 비율이 높을수록

값을 높게 설정한다. 반면, 사용자장치(20)가 전송한 영상의 경우, 검증이 한번 이루어지기 때문에 특이치(Outlier)가 존재할 가능성이 희박하다. 따라서 학습용 영상 중 사용자장치(20)가 전송한 영상의 비율이 높을수록

값을 높게 설정한다. Next, the model generation unit 200 sets the hyperparameter in step S220. The model generating unit 200 uses a hyperparameter according to a ratio of each of the image collected by the data processing unit 100 and the image transmitted by the user device 20 in the plurality of training images included in the training image arrangement.

and

set the value of The first term of the loss function is robust to outliers, and the second term is sensitive to outliers. Accordingly, in the case of the image collected by the data processing unit 100 , since the image is not verified, there is a high possibility that an outlier exists. Therefore, the higher the ratio of the images collected by the data processing unit 100 among the learning images, the more

Set the value high. On the other hand, in the case of an image transmitted by the user device 20, since verification is performed once, the possibility that an outlier exists is slim. Therefore, the higher the ratio of the image transmitted by the user device 20 among the learning images, the more

Set the value high.

Next, the model generator 200 extracts necessary information from the storage module 12 in step S230 and sets a label corresponding to the training image included in each training image arrangement. Labels are set using one-hot encoding. For example, if a landmark exists in the learning image, it may be [1, 0], and vice versa, it may be [0, 1].

Then, when the model generating unit 200 inputs the training image to the identification model RM, the identification model RM performs a plurality of operations in which weights between a plurality of layers are applied to the training image input in step S240. Calculate the output value. More specifically, the backbone network (BN) generates a feature map (FM) by performing a plurality of convolution operations. The generated feature map FM is input to a region detection network RPN, and the region detection network RPN derives one or more regions of interest from the feature map FM through a region box. And the identification network (RN) receives information on the region box defining the region of interest from the region detection network (RPN) and the feature map (FM) from the backbone network (BN). Here, the information on the area box includes the central coordinates (x, y), the width (w) and the height (h) based on the central coordinates. The identification network (RN) calculates a probability indicating whether a learning target landmark exists in the area box by performing a plurality of operations to which a plurality of inter-layer weights are applied on the feature map (FM), and outputs the calculated probability as an output value output as

Then, the model generation unit 200 through a back-propagation algorithm so that the loss that is the difference between the output value and the label through the loss function as in Equation 2 in step S250, the identification network (RN) and the backbone network (BN) ), an optimization is performed to correct the weight w.

Steps S210 to S250 described above are performed for all training images in the training image arrangement. Then, the accuracy may be calculated through the evaluation index, and it may be repeatedly performed through a plurality of different training image arrangements until a target accuracy is reached.

Next, as described above, after learning is completed, a method for identifying landmarks regardless of location changes based on the deep learning model will be described. 10 is a flowchart for explaining a method for identifying a landmark regardless of a location change based on a deep learning model according to an embodiment of the present invention. 11 and 12 are diagrams for explaining a method for identifying a landmark regardless of a location change based on a deep learning model according to an embodiment of the present invention.

Referring to FIG. 10 , the user device 20 may connect to the identification server 10 according to a user's operation and transmit an image captured by the user and location information indicating a location at which the image was captured to the identification server 10 . . For example, as shown in FIG. 11 , it is assumed that the user is located between the Manhattan Bridge and the Brooklyn Bridge.

When the data processing unit 100 receives the image and location information through the communication module 11 in step S310 , the data processing unit 100 provides the received image and location information to the identification unit 300 . Accordingly, the identification unit 300 inputs the image provided from the data processing unit 100 into the identification model RM. Then, the backbone network (BN) of the identification model (RM) generates a feature map by performing a convolution operation of a plurality of convolutional layers on the input image in step S320.

The generated feature map (FM) is input to the region detection network (RPN), and the region detection network (RPN) is one through the region box (BB: x, y, w, h) in the feature map (FM) in step S330. A region of interest (ROI) above is derived.

Then, the identification unit 300 loads the learned identification network (RN) to identify the landmark according to the order close to the input location information. For example, assuming that the landmark closest to the user's location (U) in FIG. 11 is the Manhattan Bridge (B), the identification unit 300 is an identification network (RN) learned to identify the Manhattan Bridge (B) as a landmark. load

When the identification network (RN) is loaded, the backbone network (BN) provides a feature map (FM) to the identification network (RN), and the region detection network (RPN) creates a region box defining the region of interest. provided to Here, the area box represents the central coordinates (x, y), and the width (w) and height (h) based on the central coordinates. In this case, the region detection network (RPN) may sequentially provide information on the region of interest in the order of increasing the size of the region box in step S350. For example, referring to FIG. 12 , since the size of the third area box BB3 is larger than the size of the fourth area box BB4, first, the area detection network RPN determines the center coordinates of the third area box BB3, The width and height (x, y, w, h) are provided to the identification network (RN).

Then, the identification network RN performs a plurality of operations in which a plurality of inter-layer weights are applied to the feature map FM in step S360 so that a learning target landmark (eg, Manhattan Bridge) exists in the area box BB3. A probability indicating whether or not to do so is calculated, and the calculated probability is output as an output value.

Then, the identification unit 300 determines whether the landmark is identified according to the output value in step S370. For example, if the identification network (RN) outputs a probability (0.122, 0.878), the identification unit 300 uses these probabilities (0.122, 0.878) to generate a landmark ( For example, it can be seen that the probability of the existence of the Manhattan Bridge) is 12% and the probability that the landmark does not exist is 88%. Conversely, if the identification network RN outputs probabilities (0.746, 0.254), the identification unit 300 uses these probabilities (0.746, 0.254) to generate landmarks (eg, within the area box, for example, the third area box BB4). , Manhattan Bridge), the probability of existence is 75%, and the probability that the landmark does not exist is 25%, so it can be determined that the object in the area box is a landmark according to the probability.

As a result of the determination in step S370 , when the landmark is identified, the identification unit 300 transmits the name and description of the landmark identified in step S380 to the user device 20 through the communication module 11 . In addition, the identification unit 300 may transmit additional information such as restaurants and tourist attractions around the landmark to the user device 20 .

On the other hand, as a result of the determination in step S370, if landmark identification is not made, the identification unit 300 determines whether landmark identification has failed in all regions of interest. As a result of this determination, if landmark identification in all regions of interest does not fail, the identification network (RN) receives a region box defining another region of interest having the next-order size from the region detection network (RPN) and receives the above-described step S350 to S370 are repeated. For example, the fourth area box BB4, which is the larger area box next to the previously transmitted third area box BB3, is used.

On the other hand, if landmark identification fails in all regions of interest, the identification unit 300 proceeds to step S400 to identify each other landmark located within a predetermined radius from the location information U transmitted by the user device 20. In all of the learned identification networks (RN), it is checked whether the user device 20 fails to identify the landmark in the transmitted image.

As a result of the check in step S400, if all identification networks (RN) do not fail to identify the landmark, the identification unit 300 proceeds to step S340 and the landmark whose distance from the location information transmitted by the user device 20 is the next priority. Repeat the above procedure by replacing it with a learned identification network (RN) to identify . For example, referring to FIG. 11 , the Brooklyn Bridge is replaced with an identification network RN that can identify the Manhattan Bridge, and the first area box BB1 and the second area box BB2 are connected to the corresponding identification network RN. ) can be provided sequentially to identify landmarks (eg, Brooklyn Bridge).

On the other hand, as a result of the confirmation of step S400, if all identification networks (RN) have failed to identify landmarks, the identification unit 300 proceeds to step S410 and sends the land from the image to the user device 20 through the communication module 11. Indicates that the mark cannot be identified.

Meanwhile, the method according to the embodiment of the present invention described above may be implemented in the form of a program readable by various computer means and recorded in a computer readable recording medium. Here, the recording medium may include a program command, a data file, a data structure, etc. alone or in combination. The program instructions recorded on the recording medium may be specially designed and configured for the present invention, or may be known and available to those skilled in the art of computer software. For example, the recording medium includes magnetic media such as hard disks, floppy disks and magnetic tapes, optical recording media such as CD-ROMs and DVDs, and magneto-optical media such as floppy disks ( magneto-optical media) and hardware devices specially configured to store and execute program instructions such as ROM, RAM, flash memory, and the like. Examples of program instructions may include not only machine language wires such as those generated by a compiler, but also high-level language wires that can be executed by a computer using an interpreter or the like. Such hardware devices may be configured to operate as one or more software modules to perform the operations of the present invention, and vice versa.

Although the present invention has been described above using several preferred embodiments, these examples are illustrative and not restrictive. As such, those of ordinary skill in the art to which the present invention pertains will understand that various changes and modifications can be made in accordance with the doctrine of equivalents without departing from the spirit of the present invention and the scope of rights set forth in the appended claims.

10: identification server 11: communication module
12: storage module 13: control module
20: user device 100: data processing unit
200: model generation unit 300: identification unit

Claims (4)

In the method for identifying a landmark regardless of a change in location,
receiving, by a data processing unit, an image and location information at which the image was captured from the user device;
calculating, by an identification unit, a probability indicating whether a learned landmark exists in the image through an identification model including an identification network corresponding to the location information; and
identifying the landmark according to the probability by an identification unit, and transmitting a name and description of the identified landmark to the user device;
includes,
The step of calculating the probability is
generating, by the backbone network of the identification model, a feature map through a convolution operation of a plurality of convolutional layers with respect to the image;
determining, by the region detection network of the identification model, a region box representing the region of interest in the feature map;
Loading the identification network learned to identify the landmark based on the identification unit the location information; and
The identification network receives the feature map and the region box, and calculates a probability indicating whether a learned landmark exists in the region box through a plurality of operations in which a plurality of inter-layer weights are applied to the feature map to do;
including,
The identification unit loads the identification network learned to identify landmarks according to the order close to the location information,
The identification unit loads the identification network learned to identify the landmark closest to the location information, and sequentially in each of the plurality of area boxes in the order of the size of the area box among the plurality of area boxes based on the identification network. Determining whether the nearest landmark is identified,
The identification unit determines that the closest landmark is identified when the nearest landmark is identified in all of the plurality of area boxes,
When the identification unit succeeds in identification of the nearest landmark based on some of the area boxes among the plurality of area boxes, and fails to identify the nearest landmark based on the remaining area boxes, the identification network Receiving another area box defining another area of interest having a next-order size from the area detection network and re-determining whether the nearest landmark is identified based on the other area box,
When the identification unit fails to identify the nearest landmark based on each of the plurality of area boxes, the identification network is based on another identification network for identifying another landmark located within a predetermined radius from the location information. A method for identifying a landmark, comprising defining a plurality of different area boxes to perform identification of the different landmarks based on each of the different plurality of area boxes. delete According to claim 1,
Before receiving the image and the location information at which the image was taken,
Further comprising the step of providing a training image arrangement including a plurality of training images by the model generator,
The training video arrangement is
collecting a plurality of images including the landmark;
generating a plurality of image vectors by embedding the plurality of images in a multidimensional vector space through a plurality of convolution operations;
determining a center vector of a cluster randomly on the vector space;
clustering the plurality of image vectors according to a distance from the center vector;
recrystallizing a center vector of a cluster for each cluster generated based on the clustering;
determining whether all of the plurality of image vectors are included in a cluster having a minimum distance from the center vector of the re-determined cluster;
When all of the plurality of image vectors are not included in the cluster having the minimum distance from the center vector of the recrystallized cluster, the plurality of image vectors are again clustered according to the distance from the center vector, and the plurality of images determining a current cluster as an optimization cluster when all vectors are included in a cluster having a minimum distance from the center vector of the cluster recrystallized;
extracting a plurality of image vectors within a predetermined radius from the center of the optimization cluster; and
Method for identifying a landmark, characterized in that determined based on the step of determining a plurality of images corresponding to the extracted plurality of image vectors as the learning image arrangement. delete

Images