KR101320247B1 - Apparatus and method for image matching in augmented reality service system - Google Patents

Abstract

The present invention relates to an apparatus and method for image matching in a system supporting an augmented reality service. In accordance with another aspect of the present invention, an apparatus for image matching includes: a database for storing feature points of a plurality of reference images for each object in a reference image group of a corresponding object, a communication unit for receiving a photographed image of the object and location information of the terminal, and the input unit; Based on the location information of the terminal, the feature points of a plurality of reference images for each object existing in the region where the terminal is located are obtained from the database, the feature points of the received captured image are acquired, and the feature points of the obtained captured image. And an image matching unit for comparing the feature points of the plurality of reference images for each detected object, the number of feature point matchings of the plurality of reference images for each reference image group, and the number of feature points of the plurality of reference images for each reference image group. Based on a ratio of the feature point match number sum result to the feature point sum sum result Chapter select one reference image group having the largest matching degree of similarity, and wherein the object includes a determination unit determining an object corresponding to the selected reference image group as the object of the photographed image.

Description

Apparatus and method for image matching in a system supporting augmented reality service {APPARATUS AND METHOD FOR IMAGE MATCHING IN AUGMENTED REALITY SERVICE SYSTEM}

The present invention relates to image matching, and more particularly, to an apparatus and method for image matching in a system supporting an augmented reality service.

Recently, with the development of digital image processing technology, a so-called Augmented Reality (Augmented Reality) technology is becoming a commercial trend.

Augmented reality is a technology for simultaneously outputting additional information about a target object in real time on a scene displayed on the screen when the target object is illuminated through the camera module of the terminal. Here, the additional information means related information about the target object included in the scene displayed on the screen. For example, the information as shown in Table 1 below may belong to the additional information according to the target object.

Target object Additional Information restaurant Menu, price, rating Convenience Event items, discount cards Offices Business Hours, Available Services

As such, the augmented reality technology may reinforce additional information that is difficult to obtain only in the real world and provide the information realistically. Accordingly, augmented reality technology can be applied to a variety of service environments, in particular, it is in the spotlight as a next-generation display technology suitable for ubiquitous environment.

In order to implement the service by applying the augmented reality technology, it is premised that the user accurately identifies the target object that the user watches through the camera module of the terminal. Conventional augmented reality technology to identify the target object based on the location and direction information of the terminal, the sensor (for example, GPS (GPS) sensor and compass used to obtain the location and direction information) (Compass sensor) error (approximately ± 50m) may cause an error in object identification. In detail, other objects existing within an error range, other than the object that the user watches through the camera module of the terminal, may be misidentified as the target object. In this case, additional information of the misidentified other object may be displayed on the screen, and thus there is a problem in that an appropriate service cannot be provided to the user.

Accordingly, an object of the present invention is to provide an apparatus and method for accurately identifying a front object in a system supporting an augmented reality service.

Another object of the present invention is to provide a new image matching device and method for accurately identifying a front object through matching between a photographed image of a terminal and a reference image in a database of an augmented reality providing server in a system supporting an augmented reality service. .

Another object of the present invention is to provide an apparatus and method for determining a view point and distance for a front object based on image matching information in a system supporting augmented reality service.

According to a first aspect of the present invention for achieving the above object, an apparatus for image matching includes a database for storing feature points of a plurality of reference images for each object in a reference image group of the object, a captured image of the object, and a terminal. Detects feature points of a plurality of reference images for each object existing in a region where the terminal is located in a database based on the communication unit receiving position information of the terminal and the received position information of the terminal, and detects the characteristic points of the received captured image. And an image matching unit comparing the acquired feature points of the captured image with the feature points of the plurality of reference images for each detected object, the number of feature point matchings of the plurality of reference images for each reference image group, and adding a reference image group. Summing the number of feature points of the plurality of reference images for each of the plurality of reference images, and the feature of the summing result of the number of feature points And an object determiner configured to select one reference image group having the largest match similarity based on a ratio of the matching result sum and determine an object corresponding to the selected reference image group as the object of the captured image. It is characterized by.

According to a second aspect of the present invention, an apparatus for image matching includes receiving a database storing feature points of a plurality of reference images for each object in a reference image group of an object, receiving a captured image of the object, and position information of the terminal. Based on the communication unit and the location information of the received terminal, detects feature points of a plurality of reference images for each object existing in a region where the terminal is located in a database, obtains feature points of the received captured image, and obtains The image matching unit for comparing the feature points of the photographed image and the feature points of the plurality of reference images for each detected object, and the largest matching based on the ratio of the number of feature point matches to the number of feature points of the plurality of reference images for each object. Selecting a reference picture having a similarity degree, the customer corresponding to the reference picture group to which the selected reference picture belongs The object is characterized by comprising a determination unit determining an object of the photographed image.

According to a third aspect of the present invention, an apparatus for image matching includes a database storing feature points of one reference image for each object, a communication unit for receiving a captured image of the object and location information of the terminal, and the received terminal. On the basis of the position information of the, from the database to detect the feature points of one reference image for each object existing in the area where the terminal is located, to obtain the feature points of the input image, and for each feature of the input image And comparing the feature points of one reference image among the feature points of the detected one reference image for each object, and, in the same manner, the feature points of the detected remaining reference images with respect to each feature point of the received captured image. Image matching unit for comparison to target and feature points for the number of feature points of one reference image per object And an object determination unit that selects one reference image having the largest matching similarity based on a ratio of the sum number, and determines an object corresponding to the selected reference image as the object of the captured image. .

According to a fourth aspect of the present invention, an apparatus for image matching includes a database storing feature points of one reference image for each object, a communication unit configured to receive a photographed image of the object and location information of the terminal, and the received terminal. On the basis of the position information of the, from the database to detect the feature points of one reference image for each object existing in the area where the terminal is located, to obtain the feature points of the input image, and for each feature of the input image The image matching unit for comparing the feature points of one reference image for every object detected by the object, and the largest matching similarity based on a ratio of the number of feature point matches to the number of feature points of one reference image for each object An object corresponding to the selected reference image, and an object corresponding to the selected reference image It characterized in that it includes a decision to determine an object.

The present invention can accurately identify the front object by applying a new image matching technology between the photographed image of the terminal and the reference images in the database of the augmented reality providing server in a system supporting the augmented reality service, and accordingly, There is an advantage that can accurately output the additional information on the screen. In addition, the present invention may determine the type and size of the additional information displayed on the screen of the terminal according to the determined view point and distance information by determining the view point and distance for the front object based on the image registration information.

1 is a block diagram schematically showing a system supporting an augmented reality service according to the present invention;
2 is a block diagram schematically showing the device configuration of a portable terminal in a system supporting an augmented reality service according to the present invention;
3 is a block diagram schematically showing the device configuration of the augmented reality providing server in the system supporting the augmented reality service according to the present invention;
4 is an augmented reality providing server for matching a captured image and a reference image when a database is implemented by storing feature points of one reference image for each object in a system supporting an augmented reality service according to the first embodiment of the present invention. A flowchart illustrating a method of operation of
5 is an augmented reality providing server for matching a captured image and a reference image when a database is implemented by storing feature points of one reference image for each object in a system supporting an augmented reality service according to the first embodiment of the present invention. An illustration showing the operation method of,
FIG. 6 is an augmented reality providing server for matching a captured image and a reference image when a database is implemented by storing feature points of one reference image for each object in a system supporting an augmented reality service according to a second embodiment of the present invention. A flowchart illustrating a method of operation of
7 is an augmented reality providing server for matching a captured image and a reference image when a database is implemented by storing feature points of one reference image for each object in a system supporting an augmented reality service according to a second embodiment of the present invention. An illustration showing the operation method of,
8 is an augmented reality providing server for matching a captured image and a reference image when a database is implemented by storing feature points of a plurality of reference images for each object in a system supporting an augmented reality service according to a third embodiment of the present invention. A flow chart illustrating a method of operation; and
9 is an augmented reality providing server for matching a captured image and a reference image when a database is implemented by storing feature points of a plurality of reference images for each object in a system supporting an augmented reality service according to a fourth embodiment of the present invention. Flow chart illustrating a method of operation.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Reference will now be made in detail to the preferred embodiments of the present invention, examples of which are illustrated in the accompanying drawings. In the following description, a detailed description of known functions and configurations incorporated herein will be omitted when it may make the subject matter of the present invention rather unclear.

Hereinafter, an apparatus and method for accurately identifying a front object in a system supporting an augmented reality service will be described. In particular, the present invention describes a new image registration device and method between the captured image and the reference image for accurately identifying the front object in the system supporting the augmented reality service. Additionally, an apparatus and method for determining a viewpoint and a distance for a front object based on image registration information will be described.

In the following description, the photographed image refers to an image photographed through the camera module, and the reference image refers to an image photographed in advance and used to match the photographed image.

In the following description, one object is included in one image as an example, but it is applicable to a case in which a plurality of objects are included in one image. In this case, 'object' used in the following description is one image. It is assumed that it refers to an object located at the center of the screen among a plurality of objects included in.

In the following description, an image matching technique is a technique of searching a reference image including a scene identical to a scene included in a photographed image, and comparing the photographed image with reference images in a database to obtain the highest matching similarity. A reference image having a ratio) is determined as the final matched image for the captured image.

In the image matching technique according to the present invention, after feature points are extracted from the captured image and the reference image, the image matching technique is performed using a method of comparing the nearest neighbors between the feature points of the extracted image and the feature image of the reference image. Do this. Here, the 'feature point' is a point that is characteristic of an image and is formed of a high-dimensional (eg, 64-dimensional) vector value of a real type. For example, the shortest distance comparison method calculates a sum of absolute difference (SAD) between two feature points in the same dimension for each pair of feature points, and sets a pair of feature points at which the calculated SAD becomes the minimum. The matching method can be used.

Here, the SAD can be calculated using Equation 1 below.

[Equation 1]

는 n차원을 갖는 촬영 영상과 참조 영상의 특징점에 대하여, 동일 차원에서의 두 특징점 값 간 차이의 합을 나타내며, 여기서, 특징점의 차원 수는 달라질 수 있다.

는 촬영 영상의 특징점을 나타내고,

는 참조 영상의 특징점을 나타낸다. here,

Represents the sum of the differences between two feature point values in the same dimension with respect to the feature point of the captured image and the reference image having n dimensions, where the number of dimensions of the feature point may vary.

Indicates a feature point of the captured image,

Denotes a feature point of the reference image.

On the other hand, both the captured image and the reference image has a different number of feature points according to the characteristics of the image information. In general, many feature points are extracted from an image of a complex scene and a small number of feature points are extracted from an image of a relatively simple scene. This means that the number of feature points matched by each pair (captured image-reference image) cannot be interpreted as the match similarity of the pair, and thus the number of feature point matches cannot determine the best match image pair. In addition, even though the SAD value of one pair of feature points has a minimum value compared to the SAD values of other pairs of feature points, two feature points having the minimum SAD value may or may not actually be the same feature point. Since only two feature points are the same feature point, the number of feature points must be counted. Therefore, the pair of feature points is considered to be the same feature point only when two feature points having the minimum SAD value satisfy the following Equation 2.

&Quot; (2) "

가 참조 영상의 특징점

와 갖는 최소 SAD값이, 다른 특징점

와 갖는 두 번째로 작은 SAD값보다 훨씬 작아야만, 최소 SAD값을 가지는 촬영 영상의 특징점과 참조 영상의 특징점

를 동일한 특징점으로 결정한다는 것을 의미한다.here, Is a discriminant ratio and may have a value between 0.5 and 0.9. Equation 2 is a characteristic point of the photographed image.

Feature of the reference image

Has a minimum SAD value

The feature point of the captured image with the minimum SAD value and the feature point of the reference image must be much smaller than the second smallest SAD value

Means to determine the same feature point.

As described above, based on Equation 1 and Equation 2, for each feature point in the captured image, the feature point in the corresponding reference image may be determined.

1 is a block diagram schematically showing a system supporting an augmented reality service according to the present invention.

As shown, the system supporting the augmented reality service according to the present invention comprises a portable terminal 100, wired and wireless communication network 110 and augmented reality providing server 120.

The portable terminal 100 is a terminal for performing a general voice call and data communication by wireless communication in conjunction with the wired and wireless communication network 110, in particular, a variety of providing augmented reality service to the user based on the captured image according to the present invention Means appliance. For example, the portable terminal 100 may be a cellular phone, a smart phone, a personal computer (PC), a notebook computer, a personal digital assistant (PDA), a portable multimedia. A player may be a player (PMP: Portable Multimedia Player), a PlayStation Portable (PSP), or the like.

The portable terminal 100 provides the captured image obtained through the camera module to the augmented reality providing server 120 through the wired / wireless communication network 110, and photographed from the augmented reality providing server 120 through the wired / wireless communication network 110. Additional information about the target object in the image is obtained, and the obtained additional information is merged with the captured image and output on the screen. Here, the portable terminal 100 provides the position and direction information of the terminal itself to the augmented reality providing server 120 together with the captured image. Accordingly, the augmented reality providing server 120 does not match the reference images of all objects in the database with respect to the captured image, but matches only the reference images of the objects in the region where the portable terminal 100 is located. You can do In addition, the portable terminal 100 obtains view and distance information on the object from the augmented reality providing server 120 together with additional information on the target object in the captured image, and displays it on the screen according to the view and distance information. The type and size of the additional information may be set differently.

The wired / wireless communication network 110 refers to a network connecting the portable terminal 100 and the augmented reality providing server 120.

The augmented reality providing server 120 provides additional information on the object in the captured image to the portable terminal 100 in conjunction with the wired and wireless communication network (110). To this end, the augmented reality providing server 120 stores the feature points of the reference images and the exact position information (latitude / longitude) of each reference image in a database in advance, photographed through the camera module of the portable terminal 100 The searched image is matched with reference images in a database to search for the same reference image, and additional information about an object in the searched reference image is obtained and transmitted to the portable terminal 100. In this case, the augmented reality providing server 120 targets reference images having a position within a GPS error range based on the position and orientation information of the portable terminal 100 and the position information for each reference image. Also, the augmented reality providing server 120 may determine a viewpoint and a distance of a corresponding object based on the matched reference image, and transmit other additional information to the portable terminal 100 according to the viewpoint and distance information.

2 is a block diagram schematically illustrating a device configuration of a portable terminal in a system supporting an augmented reality service according to the present invention.

As illustrated, the portable terminal includes a control unit 200, a camera module 202, an image processing unit 204, an additional information merging unit 206, a display unit 208, a position measuring unit 210, and a direction measuring unit ( 212, a storage unit 214, an input unit 216, and a communication unit 218.

Referring to FIG. 2, the controller 200 controls the overall operation of the portable terminal, and controls and processes a function for providing an augmented reality service to a user based on the captured image according to the present invention. In particular, the control unit 200 provides a captured image obtained through the camera module 202 to the augmented reality providing server through the communication unit 218, and the object in the captured image through the communication unit 218 from the augmented reality providing server. Acquiring additional information about an object, merging the obtained additional information with the captured image through the additional information merging unit 206, and outputting the merged captured image and the additional information to the screen through the display unit 208. Control and handle the functions. Here, the control unit 200 provides the position and direction information of the terminal with the captured image to the augmented reality providing server through the communication unit 218.

The camera module 202 includes a camera sensor for converting an optical signal detected at the time of capturing an image into an electrical signal, and a signal processor for converting an analog image signal photographed from the camera sensor into digital data. The converted video signal is output to the video processor 204.

The image processor 204 processes the image signal from the camera module 202 in units of frames and outputs the image signal to the display unit 208 or the additional information merger 206 according to the characteristics and size of the display unit 208. In addition, the image processor 204 outputs the captured image to the controller 200 to obtain additional information of the object in the captured image. The image processor 204 includes an image codec, and the image codec functions to code an image signal in a set manner or to decode the coded frame image data into original frame image data.

The additional information merging unit 206 acquires additional information on the target object in the captured image from the augmented reality providing server through the communication unit 218 and merges it with the captured image from the image processing unit 204, and the merged image. The video and additional information are output to the display unit 208.

The display unit 208 displays status information generated during operation of the portable terminal and limited numbers and letters. In particular, the display unit 208 displays an image from the image processing unit 204 or the additional information merging unit 206 on the screen.

The position measuring unit 210 measures the position of the terminal, for example, may be implemented by a GPS sensor.

The direction measuring unit 212 measures the direction of the terminal, for example, may be implemented as a compass sensor.

The storage unit 214 stores a program and various data for the overall operation of the portable terminal.

The input unit 216 includes a plurality of numeric keys, character keys, and function keys, and provides data corresponding to a key pressed by the user to the controller 200.

The communication unit 218 performs a function of transmitting and receiving a radio signal of data input and output through an antenna.

3 is a block diagram schematically illustrating a device configuration of an augmented reality providing server in a system supporting an augmented reality service according to the present invention.

As shown, the augmented reality providing server, the control unit 300, the image matching unit 302, the database 304, the object determination unit 306, the viewpoint determination unit 308, the distance determination unit 310, the addition The information acquisition unit 312 and the communication unit 314 is configured.

Referring to FIG. 3, the controller 300 controls the overall operation of the augmented reality providing server, and controls and processes a function for providing additional information about an object in a captured image to a portable terminal according to the present invention. In particular, the controller 300 searches for the same reference image by matching the captured image from the portable terminal with the reference images in the database, and acquires additional information about the object in the searched reference image and transmits it to the portable terminal. Control and handle the functions. In addition, the controller 300 determines a viewpoint and distance of the corresponding object based on the finally matched reference image, and controls a function for transmitting other additional information to the portable terminal according to the determined viewpoint and distance information. Process.

The image matching unit 302 receives the captured image and the position and direction information of the terminal from the portable terminal through the communication unit 314, and based on the received position and direction information of the terminal, the terminal in the database 304 The feature points of the reference image for the objects in the located area are determined. The image matching unit 302 obtains feature points of the received captured image and matches the feature points of the determined reference image with respect to each feature point of the received captured image. In the present invention, a matching method between a captured image and a reference image according to a database implementation method is proposed as four embodiments, which will be described in detail later.

The database 304 stores feature points of reference images and accurate position information (latitude / longitude) of each reference image.

The object determiner 306 selects one reference image to determine a final matched image based on the number of feature point matchings of the reference images, and determines an object corresponding to the selected reference image.

The viewpoint determination unit 308 determines the viewpoint of the corresponding reference image with respect to the determined object.

The distance determiner 310 determines a distance of the captured image with respect to the determined object.

The additional information obtaining unit 312 obtains additional information of the determined object, and the obtained additional information is transmitted to the terminal through the communication unit 314. In this case, the additional information obtaining unit 312 may set different types and sizes of additional information displayed on the screen of the terminal and transmit them to the terminal based on the determined time and distance of the object.

The communication unit 314 performs a function of transmitting and receiving a radio signal of input and output data.

Meanwhile, the present invention considers two methods in implementing a database based on the feature points of the object-specific reference image. That is, a method of implementing a database with feature points of one reference image for each object and a method of implementing a database with feature points of a plurality of reference images for each object are considered. The method of matching the captured image and the reference image may vary according to the method of implementing the database. The present invention proposes a matching method between a captured image and a reference image according to a database implementation method, and will be described in detail with reference to FIGS. 4 to 9. The four embodiments may be used independently, or two or more embodiments may be used in combination.

4 is an augmented reality providing server for matching a captured image and a reference image when a database is implemented by storing feature points of one reference image for each object in a system supporting an augmented reality service according to the first embodiment of the present invention. A flowchart illustrating a method of operation.

Referring to FIG. 4, in operation 401, the AR server stores feature points of one reference image for each object in a database.

In step 403, the augmented reality providing server receives a captured image of an object and position and direction information of the terminal from the terminal.

In step 405, the AR providing server determines the feature points of the reference image for the objects in the region where the terminal is located in the database, based on the received position and direction information of the terminal. That is, the feature points of the reference image for the objects existing in the position and the direction of the terminal are determined.

In step 407, the AR providing server acquires feature points of the received captured image. Here, various feature point extraction algorithms such as Scale Invariant Feature Transform (SIFT) and Speeded Up Robust Features (SURF) may be used to obtain the feature points.

Thereafter, the AR providing server matches each feature point of the received captured image with the feature points of one reference image among the determined reference image feature points through steps 409 to 413. Each feature point of the received captured image is matched with the feature points of the remaining reference image. That is, for each feature point of the received captured image, a feature point that satisfies Equation 2 is determined based on the feature points of one reference image among the determined reference image feature points.

In detail, the augmented reality providing server selects the feature points of one reference image that is not selected among the determined reference image feature points in step 409, and selects the selected feature points of the received captured image in step 411. After matching the feature points of one reference image to an object, it is checked whether the feature points of all the determined reference images are selected in step 413. If it is determined in step 413 that the feature points of all the determined reference images are not selected, the augmented reality providing server returns to step 409 and repeats the following steps, and the feature points of all the determined reference images If it is determined that the selection is made, the flow proceeds to step 415.

For example, as shown in FIG. 5, the SAD is calculated between the first feature point of the photographed image and each feature point of the reference image of the first object, using <Equation 1>, and the <math The pair of feature points having the minimum SAD, that is, the first feature point of the captured image and the corresponding feature point of the reference image of the first object corresponding to Equation 2 > In addition, the same method is applied to the remaining feature points of the captured image to match each feature point of the captured image with the feature points of the reference image of the first object. Also, by applying the same method, each feature point of the captured image is matched with feature points of the reference image of the remaining object.

In step 415, the AR providing server determines match similarity of each reference image based on the number of feature point matches of the reference images.

Here, the matching similarity may be determined using Equation 3 below.

&Quot; (3) &quot;

는 i번째 참조 영상의 정합 유사도를 나타내고,

는 i번째 참조 영상의 특징점 개수,

는 i번째 참조 영상의 특징점 정합 개수를 나타낸다. here,

Represents the match similarity of the i th reference image,

Is the number of feature points in the i reference image,

Denotes the number of feature point matches of the i-th reference image.

In step 417, the augmented reality providing server selects a reference image having the largest matching similarity based on the matching similarity of the respective reference images as the final matching image for the captured image.

In step 419, the augmented reality providing server determines an object corresponding to the selected reference image.

In step 421, the augmented reality providing server acquires additional information of the determined object and transmits the additional information to the terminal. As a result, the terminal may provide the augmented reality service to the user by merging the captured image and the additional information of the object on the screen.

Thereafter, the AR providing server terminates the algorithm according to the present invention.

FIG. 6 is an augmented reality providing server for matching a captured image and a reference image when a database is implemented by storing feature points of one reference image for each object in a system supporting an augmented reality service according to a second embodiment of the present invention. A flowchart illustrating a method of operation.

Referring to FIG. 6, in operation 601, the AR server stores feature points of one reference image for each object in a database.

In step 603, the augmented reality providing server receives a captured image of an object and position and direction information of the terminal from a terminal.

In step 605, the augmented reality providing server determines feature points of reference images of objects in a region where a terminal is located in a database, based on the input position and direction information of the terminal. That is, the feature points of the reference image for the objects existing in the position and the direction of the terminal are determined.

In step 607, the AR providing server acquires feature points of the received captured image. Here, various feature point extraction algorithms such as Scale Invariant Feature Transform (SIFT) and Speeded Up Robust Features (SURF) may be used to obtain the feature points.

In step 609, the AR providing server matches each feature point of the received captured image with the feature points of all the determined reference images. In other words, a feature point that satisfies Equation 2 is determined with respect to each feature point of the determined reference images with respect to each feature point of the received captured image.

For example, as shown in FIG. 7, the SAD is calculated between the first feature point of the photographed image and each feature point of the reference image of all objects by using Equation 1, and the SAD is calculated for the calculated SADs. A pair of feature points having a minimum SAD satisfying 2> are regarded as the same feature point and matched. In addition, the same method is applied to the remaining feature points of the captured image to match each feature point of the captured image with the feature points of the reference images of all objects.

In step 611, the augmented reality providing server determines matching similarity degree of each reference image based on the number of feature point matchings of the reference images.

Here, the matching similarity may be determined using Equation 4 below.

&Quot; (4) &quot;

는 i번째 참조 영상의 정합 유사도를 나타내고,

는 비교 대상으로 검출된 모든 참조 영상의 특징점 총 개수,

는 촬영 영상의 특징점 개수,

는 i번째 참조 영상의 특징점 개수,

는 i번째 참조 영상의 특징점 정합 개수를 나타낸다. here,

Represents the match similarity of the i th reference image,

Is the total number of feature points of all reference images detected as comparison targets,

Is the number of feature points in the captured image,

Is the number of feature points in the i reference image,

Denotes the number of feature point matches of the i-th reference image.

In step 613, the augmented reality providing server selects a reference image having the largest matching similarity based on the matching similarity of the respective reference images as the final matching image for the captured image.

In step 615, the augmented reality providing server determines an object corresponding to the selected reference image.

In step 617, the AR server acquires additional information of the determined object and transmits the additional information to the terminal. As a result, the terminal may provide the augmented reality service to the user by merging the captured image and the additional information of the object on the screen.

Thereafter, the AR providing server terminates the algorithm according to the present invention.

8 is an augmented reality providing server for matching a captured image and a reference image when a database is implemented by storing feature points of a plurality of reference images for each object in a system supporting an augmented reality service according to a third embodiment of the present invention. A flowchart illustrating a method of operation.

Referring to FIG. 8, in operation 801, the AR server stores feature points of a plurality of reference images for each object in a reference image group of a corresponding object in a database.

In step 803, the augmented reality providing server receives a captured image of an object and position and direction information of the terminal from a terminal.

In step 805, the AR providing server determines the feature points of the plurality of reference images for the objects in the region where the terminal is located in the database, based on the input position and direction information of the terminal. That is, the feature points of the plurality of reference images for the objects existing in the location and direction of the terminal are determined.

In step 807, the AR providing server acquires feature points of the received captured image. Here, various feature point extraction algorithms such as Scale Invariant Feature Transform (SIFT) and Speeded Up Robust Features (SURF) may be used to obtain the feature points.

Thereafter, the augmented reality providing server matches each feature point of the received captured image with the feature points of one reference image among the feature points of the plurality of reference images for each object determined through steps 809 to 813, and the same. In this way, each feature point of the received captured image is matched with feature points of the determined remaining reference image. That is, for each feature point of the received captured image, a feature point that satisfies Equation 2 is determined based on the feature points of one reference image among the feature points of the plurality of reference images for each object.

In detail, the augmented reality providing server selects feature points of one reference image that are not selected among the feature points of the plurality of reference images for each object determined in step 809, and each feature point of the received captured image in step 811. In step 813, the feature points of the selected reference image are matched with each other, and then, in step 813, the feature points of all the determined reference images are checked. If it is determined in step 813 that the feature points of all the determined reference images are not selected, the augmented reality providing server returns to step 809 to repeat the following steps, and the feature points of all the determined reference images If it is determined that the selection is made, the flow proceeds to step 815.

In step 815, the augmented reality providing server determines matching similarity of each reference image based on the number of feature point matchings of the plurality of reference images for each object.

Here, the matching similarity may be determined using Equation 5 below.

&Quot; (5) &quot;

는 i번째 참조 영상의 정합 유사도를 나타내고,

는 i번째 참조 영상의 특징점 개수,

는 i번째 참조 영상의 특징점 정합 개수를 나타낸다. 상기 <수학식 6>은, 하나의 참조 영상의 특징점 정합 개수를, 해당 참조 영상의 특징점 개수로 나눔으로써, 해당 참조 영상의 정합 유사도를 결정한다는 것을 의미한다. here,

Represents the match similarity of the i th reference image,

Is the number of feature points in the i reference image,

Denotes the number of feature point matches of the i-th reference image. Equation (6) means that the matching similarity of the reference image is determined by dividing the number of feature point matchings of one reference image by the number of feature points of the reference image.

In step 817, the augmented reality providing server selects a reference image having the largest matching similarity based on the matching similarity of the respective reference images as the final matching image for the captured image.

In step 819, the augmented reality providing server determines an object corresponding to the reference picture group to which the selected reference picture belongs.

In step 821, the augmented reality providing server acquires additional information of the determined object and transmits the additional information to the terminal. As a result, the terminal may provide the augmented reality service to the user by merging the captured image and the additional information of the object on the screen.

Thereafter, the AR providing server terminates the algorithm according to the present invention.

9 is an augmented reality providing server for matching a captured image and a reference image when a database is implemented by storing feature points of a plurality of reference images for each object in a system supporting an augmented reality service according to a fourth embodiment of the present invention. A flowchart illustrating a method of operation.

Referring to FIG. 9, in operation 901, the AR server stores feature points of a plurality of reference images for each object in a reference image group of a corresponding object in a database.

In step 903, the augmented reality providing server receives a captured image of the object and position and direction information of the terminal from the terminal.

In step 905, the AR providing server determines the feature points of the plurality of reference images for the objects in the region where the terminal is located in the database, based on the received position and direction information of the terminal. That is, the feature points of the plurality of reference images for the objects existing in the position and the direction of the terminal are determined.

In operation 907, the AR providing server acquires feature points of the received captured image. Here, various feature point extraction algorithms such as Scale Invariant Feature Transform (SIFT) and Speeded Up Robust Features (SURF) may be used to obtain the feature points.

Thereafter, the augmented reality providing server matches each feature point of the received captured image with the feature points of one reference image among the feature points of the plurality of reference images for each object determined in steps 909 to 913, and the same. In this way, each feature point of the received captured image is matched with feature points of the determined remaining reference image. That is, for each feature point of the received captured image, a feature point that satisfies Equation 2 is determined based on the feature points of one reference image among the feature points of the plurality of reference images for each object.

In detail, the augmented reality providing server selects feature points of one reference image that are not selected from among the feature points of the plurality of reference images for each object determined in step 909, and each feature point of the received captured image in step 911. In step 913, the feature points of the selected reference image are matched with each other, and then, in operation 913, it is determined whether the feature points of all the determined reference images are selected. If it is determined in step 913 that the feature points of all the determined reference images are not selected, the augmented reality providing server returns to step 909 and repeats the following steps, and the feature points of all the determined reference images If it is determined that the selection is made, the flow proceeds to step 915.

In step 915, the augmented reality providing server determines match similarity for each reference image group based on the number of feature point matches of the plurality of reference images for each object.

Here, the matching similarity may be determined using Equation 6 below.

&Quot; (6) &quot;

는 g번째 참조 영상 그룹의 정합 유사도를 나타내고,

은 g번째 참조 영상 그룹의 첫 번째 참조 영상의 인덱스,

은 g번째 참조 영상 그룹의 마지막 번째 참조 영상의 인덱스,

는 g번째 참조 영상 그룹의 i번째 참조 영상의 특징점 개수,

는 g번째 참조 영상 그룹의 i번째 참조 영상의 특징점 정합 개수를 나타낸다. 상기 <수학식 6>은, 하나의 참조 영상 그룹 내 다수의 참조 영상의 특징점 정합 개수를 합산하고, 상기 참조 영상 그룹 내 다수의 참조 영상의 특징점 개수를 합산하며, 특징점 정합 개수의 합산 결과를 특징점 개수의 합산 결과로 나눔으로써, 해당 참조 영상 그룹의 정합 유사도를 결정한다는 것을 의미한다. here,

Represents the match similarity of the g th reference picture group,

Is the index of the first reference picture of the g-th reference picture group,

Is the index of the last reference picture of the g-th reference picture group,

Is the number of feature points of the i th reference picture of the g th reference picture group,

Denotes the number of feature point matches of the i th reference picture of the g th reference picture group. Equation 6 includes summing the number of feature point matches of the plurality of reference images in one reference picture group, summing the number of feature points of the plurality of reference pictures in the reference picture group, and adding the result of the sum of the number of feature point matches. By dividing by the sum result, it means that the matching similarity of the corresponding reference picture group is determined.

In step 917, the augmented reality providing server selects a reference image group having the largest matching similarity based on the matching similarity for each reference image group as a final matching image group for the captured image.

In operation 919, the augmented reality providing server determines an object corresponding to the selected reference image group.

In step 921, the augmented reality providing server acquires additional information of the determined object and transmits the additional information to the terminal. As a result, the terminal may provide the augmented reality service to the user by merging the captured image and the additional information of the object on the screen.

Thereafter, the AR providing server terminates the algorithm according to the present invention.

Meanwhile, when the final matched image for the captured image is selected based on the first to fourth embodiments, a viewpoint (ie, an angle at which the camera module views the object) of the reference image with respect to the front object is determined, and Only the additional information corresponding to the determined time point among the additional information of the object corresponding to the reference image may be transmitted to the terminal. That is, other additional information may be transmitted to the terminal according to the determined time point.

Here, the viewpoint of the corresponding reference image with respect to the front object may be determined based on, for example, a projection matrix of reference feature points of the reference image and feature points of the photographed image corresponding thereto. As already mentioned, in the database in the augmented reality providing server, feature points of the reference image for each object and accurate position information (latitude / longitude) of each reference image are stored. In addition, the database is repeatedly extracted for each feature of the reference image, that is, feature points having high repeatability in the external environment, that is, changes in the external environment (ie, photographing angle, rotation, lighting, etc.). The feature points to be divided into reference feature points may be defined in advance. Knowing the reference feature points of the reference image and the feature points of the photographed image corresponding thereto, a projection transformation matrix for converting the reference image into the photographed image at a predetermined time can be determined based on the feature points, and the inverse of the determined projection transformation matrix can be determined. Based on this, the viewpoint of the corresponding reference image for the front object may be determined.

In addition, when the final matched image for the captured image is selected based on the first to fourth embodiments, the captured image corresponding to the distance (ie, the distance from the viewpoint of the camera module) of the reference image with respect to the front object is selected. The distance may be determined and the size of the additional information displayed on the screen of the terminal may be set differently based on the determined distance information of the captured image.

The distance information of the captured image of the front object may be determined based on, for example, distance information of the corresponding reference image of the front object, reference feature points of the reference image, and feature points of the corresponding captured image. The database may store distance information from a viewpoint of a camera module for each reference image. After correcting the perspective distortion between the reference feature points of the reference image and the feature points of the photographed image corresponding thereto, the bounding box generated by connecting the reference feature points of the reference image, and corresponding By comparing the bounding boxes generated by connecting the feature points of the captured image, determining the size ratio between the two bounding boxes, and calculating the product of the determined size ratio and distance information of the corresponding reference image, distance information of the captured image with respect to the front object. Can be determined.

While the present invention has been described in connection with what is presently considered to be the most practical and preferred embodiment, it is to be understood that the invention is not limited to the disclosed embodiments, but is capable of various modifications within the scope of the invention. Therefore, the scope of the present invention should not be limited by the illustrated embodiments, but should be determined by the scope of the appended claims and equivalents thereof.

Portable terminal 100, wired / wireless communication network 110, augmented reality server 120

Claims (30)

A communication unit configured to receive a captured image of the object and location information of the terminal;
Based on the received location information of the terminal, reference images of objects existing in the region where the terminal is located among the previously stored reference images are obtained, feature points of the received captured image are acquired, and the obtained captured image is obtained. An image matching unit for comparing the feature points of the feature points with the feature points of the detected object-specific reference image; And
Determine the match similarity of each reference image based on the ratio of the number of feature point matches to the number of feature points, select one reference image having the largest match similarity based on the match similarity of the determined reference images, and select And an object determiner configured to determine an object corresponding to a reference image as an object of the captured image. The method of claim 1,
And a database for storing feature points of one reference image for each object. 3. The method of claim 2,
The image matching unit detects feature points of one reference image for each object existing in the region where the terminal is located in the database based on the received position information of the terminal, and detects the feature points of the obtained captured image. After comparing the feature points of one reference image that is not selected among the feature points of one reference image per object, it is determined whether the comparison of the feature points of one reference image for every detected object is performed. And comparing feature points of one reference image for every detected object. The method of claim 3,
The object determining unit is a device for image registration, characterized in that for determining the matching similarity using the following equation.

(here,

Represents the match similarity of the i th reference image,

Is the number of feature points in the i reference image,

Represents the number of feature point matches of the i th reference image.) 3. The method of claim 2,
The image matching unit detects feature points of one reference image for each object existing in an area in which the terminal is located in the database based on the received position information of the terminal, and for each feature point of the received captured image, An apparatus for image registration in which feature points of one reference image of every detected object are compared with each other. The method of claim 5,
The object determining unit is a device for image registration, characterized in that for determining the matching similarity using the following equation.

(here,

Represents the match similarity of the i th reference image,

Is the total number of feature points of all reference images detected as comparison targets,

Is the number of feature points in the captured image,

Is the number of feature points in the i reference image,

Represents the number of feature point matches of the i th reference image.) In the first aspect,
And a database for storing feature points of a plurality of reference images for each object in a reference image group of the corresponding object. The method of claim 7, wherein
The image matching unit detects feature points of a plurality of reference images for each object existing in an area in which the terminal is located in the database based on the received position information of the terminal, and detects the feature points of the received captured image. An apparatus for image registration, comprising comparing feature points of one reference image among feature points of a plurality of reference images for each object. 9. The method of claim 8,
The object determiner may add the number of feature point matchings of the plurality of reference pictures for each reference picture group, add the number of feature points of the plurality of reference pictures for each reference picture group, and add a ratio of the result of the sum of the feature point match numbers to the result of the number of feature points. and selecting one reference picture group having the largest matching similarity based on a ratio, and determining an object corresponding to the selected reference picture group. 10. The method of claim 9,
And the object determining unit determines the matching similarity using the following equation.

(here,

Represents the match similarity of the g th reference picture group,

Is the index of the first reference picture of the g-th reference picture group,

Is the index of the last reference picture of the g-th reference picture group,

Is the number of feature points of the i th reference picture of the g th reference picture group,

Denotes the number of feature point matches of the i th reference picture of the g th reference picture group.) 9. The method of claim 8,
And the object determiner determines an object corresponding to a reference picture group to which the selected reference picture belongs. 12. The method of claim 11,
And the object determining unit determines the matching similarity using the following equation.

(here,

Represents the match similarity of the i th reference image,

Is the number of feature points in the i reference image,

Represents the number of feature point matches of the i th reference image.) The method of claim 1,
And an additional information obtaining unit which obtains the additional information of the determined object and transmits the additional information of the determined object to the terminal. Step 1 of receiving a photographed image of the object and the location information of the terminal from the terminal;
Detecting at least one feature point of at least one reference image of an object existing in an area in which the terminal is located among feature points of a previously stored reference image based on the received position information of the terminal;
Obtaining a feature point from the input photographed image;
Comparing feature points of the detected reference image with respect to feature points of the obtained captured image;
Determining match similarity of each reference image with respect to the captured image based on a ratio of the number of feature point matches to the number of feature points according to the comparison;
Selecting one reference image having the largest matching similarity based on the matching similarity of the respective reference images; And
And determining an object corresponding to the selected reference image as an object of the captured image. 7. 15. The method of claim 14,
Before the step 1, storing the feature points of one reference image for each object in a database;
The operation of the apparatus for image registration, characterized in that for detecting the feature points of one reference image for the object existing in the area where the terminal is located among the feature points of the pre-stored reference image. 16. The method of claim 15,
The fourth step may include comparing feature points of one reference image, which is not selected, among feature points of the detected one reference image with respect to each feature point of the obtained captured image; And
Determining whether the comparison of the feature points of one reference image for every detected object is performed; or otherwise, comparing the feature points of one reference image for every detected object; Method of operation of the device for registration. 17. The method of claim 16,
The operation of the device for image registration, characterized in that the step of determining the matching similarity using the following equation.

(here,

Represents the match similarity of the i th reference image,

Is the number of feature points in the i reference image,

Represents the number of feature point matches of the i th reference image.) 16. The method of claim 15,
Wherein the step 4, the method for operating the device for image registration, characterized in that for comparing the feature points of one reference image for each object detected for each feature point of the obtained captured image. 19. The method of claim 18,
The fifth step is a step of determining the matching similarity using the following equation.

(here,

Represents the match similarity of the i th reference image,

Compare The total number of feature points of all reference images detected as a target,

Is the number of feature points in the captured image,

Is the number of feature points in the i reference image,

Represents the number of feature point matches of the i th reference image.) 15. The method of claim 14,
Before the step 1, storing feature points of a plurality of reference images for each object in a database as a reference image group of the corresponding object;
In the step 2, the method of the device for image registration, characterized in that for detecting the feature points of a plurality of reference image for the object existing in the area where the terminal is located among the feature points of the pre-stored reference image. 21. The method of claim 20,
The step 4 may include comparing feature points of the obtained captured image with feature points of one reference image that is not selected among the feature points of the plurality of reference images for each detected object; And
Determining whether the comparison of the feature points of the plurality of reference images of all detected objects has been performed; or otherwise, comparing the feature points of the plurality of reference images of all detected objects with each other; Method of operation of the device for registration. The method of claim 21,
Step 5 is a step of determining the matching similarity using the following equation.

(here,

Represents the match similarity of the i th reference image,

Is the number of feature points in the i reference image,

Represents the number of feature point matches of the i th reference image.) The method of claim 22,
Step 7 is a method of operating an apparatus for image registration, characterized in that determining the object corresponding to the reference image group to which the selected reference image belongs. The method of claim 21,
The step 5 may include summing the number of feature point matchings of the plurality of reference pictures for each reference picture group;
Summing the number of feature points of the plurality of reference pictures for each reference picture group; And
And determining matching similarity from the ratio of the feature point number summation result to the feature point number summation result. 25. The method of claim 24,
And the matching similarity is determined by using the following equation.

(here,

Represents the match similarity of the g th reference picture group,

Is the index of the first reference picture of the g-th reference picture group,

Is the index of the last reference picture of the g-th reference picture group,

Is the number of feature points of the i th reference picture of the g th reference picture group,

Denotes the number of feature point matches of the i th reference picture of the g th reference picture group.) 26. The method of claim 25,
Step 6 is a method of operating the device for image registration, characterized in that the step of selecting one reference image group having the largest matching similarity. The method of claim 26,
Step 7 is an operation method of the device for image registration, characterized in that the step of determining the object corresponding to the selected reference image group. 15. The method of claim 14,
And obtaining the additional information of the determined object and transmitting the additional information to the terminal. 8. 29. The method of claim 28,
The eighth step may include obtaining additional information of the determined object;
Determining a projection transformation matrix based on predetermined reference feature points of the selected reference image and feature points of the photographed image corresponding thereto;
Determining a viewpoint of a corresponding reference image for the determined object based on the inverse of the determined projection transformation matrix; And
And transmitting the additional information corresponding to the determined view point information to the terminal among the obtained additional information. 29. The method of claim 28,
The eighth step may include obtaining additional information of the determined object;
Determining a size ratio between the two bounding boxes by comparing a bounding box generated by connecting the predetermined reference feature points of the selected reference image and a bounding box generated by connecting the feature points of the photographed image corresponding thereto; ;
Determining a distance of a captured image with respect to the determined object based on the determined size ratio and distance information of the selected reference image; And
And setting the screen display size of the acquired additional information differently according to the determined distance information and transmitting the different screen display size to the terminal.

Images