KR20140103058A - Electronic device, method and computer readable recording medium for operating the electronic device - Google Patents

Abstract

A method of operating an electronic device in accordance with an embodiment of the present invention includes recognizing at least one object from a digital image, wherein the act of recognizing the object comprises: generating a descriptor; Determining an object on the digital image based at least in part on identification data corresponding to the at least one descriptor and the at least one reference object; And determining an orientation of the object based at least in part on the reference descriptors corresponding to the determined object and the at least one descriptor.
Further, the embodiments of the present invention are capable of other embodiments.

Description

BACKGROUND OF THE INVENTION 1. Field of the Invention [0001] The present invention relates to an electronic device, a method of operating the electronic device, and a computer readable recording medium.

Various embodiments of the present invention are directed to a method of operating an electronic device.

Virtual Reality (VR) generally refers to an environment or situation created by computer graphics that has an actual environment similar to that of the real world. It refers to an interface that makes people feel and interact with each other through the senses. The user can interact with the virtual reality in real time through manipulation of the device, and can experience a sensory experience similar to the real one.

Augmented Reality (AR) is a field of virtual reality, which is a computer graphics technique that combines virtual objects or information into a real environment to make them appear as objects in the original environment. This augmented reality is a technique of superimposing virtual objects on the real world seen by the user. It is also referred to as mixed reality (MR) since the additional information and the virtual world are combined into one image in real time in the real world.

In addition, as the mobile devices (e.g., smart phones, tablet PCs, etc.) become generalized, the virtual reality technology can be frequently accessed in various services such as education, game, navigation, advertisement, or blog. Recently wearable devices that can be worn on the body have been commercialized, and researches on virtual reality technology are becoming more active.

In order to provide the augmented reality, a feature and a descriptor are calculated on the image data, and an object on the image data is recognized using the feature and a descriptor, and an initial posture (pose) can do.

However, a considerable amount of computation is required for the object recognition and initial posture calculation. Therefore, when the object recognition and the initial posture calculation are performed in the electronic device, the processing speed may be slowed due to the limited memory size, and accurate object recognition may be difficult. For example, in a mobile terminal such as a smart phone, since there is a memory limitation, it may be important to perform accurate object recognition with a small size memory.

Accordingly, various embodiments of the present invention can be applied to a method for performing a query process performed in an object recognition process and a localization process for calculating an initial posture of an object output as a result of a query process It is possible to provide an electronic device, a method of operating an electronic device, and a computer-readable recording medium that can perform accurate and quick object recognition even with a small-sized memory by processing only the data necessary for each step is loaded into the memory.

According to an embodiment of the present invention, a method of operating an electronic device includes recognizing at least one object from a digital image, wherein the act of recognizing the object comprises: An operation for generating a descriptor; Determining an object on the digital image based at least in part on identification data corresponding to the at least one descriptor and the at least one reference object; And determining an orientation of the object based at least in part on the reference descriptors corresponding to the determined object and the at least one descriptor.

Also according to another embodiment of the present invention, an electronic device comprises: a memory configured to store a digital image; And a processor configured to process the digital image, wherein the processor is configured to generate at least one descriptor from the digital image and to generate at least one descriptor corresponding to the at least one descriptor and the at least one reference object A recognition unit for determining an object on the digital image based on a part thereof; And an initial posture calculator for determining the posture of the object based at least in part on the reference descriptors corresponding to the determined object and the at least one descriptor.

On the other hand, information on how to operate the electronic device can be stored in a computer-readable recording medium. Such a recording medium includes all kinds of recording media in which programs and data are stored so that they can be read by a computer system. Examples of the storage medium include a ROM, a Random Access Memory, a CD, a DVD, a magnetic tape, a floppy disk, an optical data storage device, an embedded multimedia card (eMMC) And the like, and may also be implemented in the form of a carrier wave (for example, transmission over the Internet). Such a recording medium may also be distributed over a networked computer system so that computer readable code in a distributed manner can be stored and executed.

According to an embodiment of the present invention, when a minutiae is detected in image data to perform object recognition and initial posture calculation, only the data necessary for each step is loaded into a memory to be operated, Object recognition is possible.

According to the embodiment of the present invention, when a feature point is detected from image data and a descriptor is generated from the detected feature points, only the identification data of the same attribute is loaded based on the attribute according to the brightness relative to the adjacent pixels The calculation performance can be improved.

Also, according to the embodiment of the present invention, accurate feature point matching can be performed with high probability by performing matching with respect to feature points in the object recognized in the initial posture calculation step.

1 is a block diagram illustrating an electronic device in accordance with various embodiments of the present invention.
2 is a block diagram showing a structure of an AR processing unit according to an embodiment of the present invention.
3 is a block diagram showing a detailed structure of an AR processing unit according to an embodiment of the present invention.
4 is a diagram of a system according to an embodiment of the present invention.
5 is a flowchart illustrating a procedure for operating an electronic device according to an embodiment of the present invention.
6 is a flow diagram illustrating a procedure for operating an electronic device in accordance with various embodiments of the present invention.
7A is a diagram showing minutiae with dark attributes according to an embodiment of the present invention.
7B is a view showing feature points having bright attributes according to an embodiment of the present invention.
8 is a diagram illustrating fields of data stored in a memory according to an embodiment of the present invention.
9 is a diagram illustrating a KD tree structure according to an embodiment of the present invention.
10 is a diagram illustrating a KD tree structure in which a nearest neighbor is indicated according to an embodiment of the present invention.
11 is a block diagram showing a detailed structure of an electronic device according to an embodiment of the present invention.
12 is a block diagram illustrating a software structure of a computer apparatus to which embodiments of the present invention may be applied.

The present invention can be variously modified and may have various embodiments, and specific embodiments will be described in detail with reference to the drawings. It is to be understood, however, that the invention is not to be limited to the specific embodiments, but includes all modifications, equivalents, and alternatives falling within the spirit and scope of the invention.

Terms including ordinals, such as first, second, etc., may be used to describe various elements, but the elements are not limited to these terms. The terms are used only for the purpose of distinguishing one component from another. For example, without departing from the scope of the present invention, the first component may be referred to as a second component, and similarly, the second component may also be referred to as a first component. And / or < / RTI > includes any combination of a plurality of related listed items or any of a plurality of related listed items.

The terminology used in this application is used only to describe a specific embodiment and is not intended to limit the invention. The singular expressions include plural expressions unless the context clearly dictates otherwise. In the present application, the terms "comprises" or "having" and the like are used to specify that there is a feature, a number, a step, an operation, an element, a component or a combination thereof described in the specification, But do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, or combinations thereof.

Unless defined otherwise, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs. Terms such as those defined in commonly used dictionaries are to be interpreted as having a meaning consistent with the contextual meaning of the related art and are to be interpreted as either ideal or overly formal in the sense of the present application Do not.

Embodiments of the present invention disclose a method of using multiple reference image recognition by feature sequential classification for implementation of augmented reality (AR).

In the following description of the embodiments of the present invention, an 'electronic device' may be any device having at least one processor and may be a camera, a portable device, a mobile terminal, a communication terminal, a portable communication terminal, A terminal, and the like. For example, the electronic device may be a digital camera, a smart phone, a mobile phone, a game machine, a television (TV), a display device, a car head unit, a notebook computer, a laptop computer, a tablet computer, Personal Digital Assistants), a navigation device, a bank ATM, a POS device in a store, and the like. Further, the electronic device in the embodiments of the present invention may be a flexible device or a flexible display device. Further, the electronic device in the embodiments of the present invention may be a wearable device (e.g., a watch device, a spectacles device, a clothes device, etc.).

Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings, so that those skilled in the art can easily carry out the present invention.

First, a structure of a system and an apparatus according to an embodiment of the present invention will be described with reference to FIG. 1 to FIG. 4, and then a procedure according to an embodiment of the present invention will be described in detail with reference to FIG. 5 and FIG.

1 is a block diagram illustrating an electronic device in accordance with various embodiments of the present invention. Referring to FIG. 1, an electronic device 100 according to an embodiment of the present invention may include an AR processing unit 101 or an AR content management unit 102. The electronic device 100 according to the embodiment of the present invention further includes a reference information database 103, a content information DB 104, a storage unit 105, a CPU 106, a GPU 107, .

The system allows the AR processing unit 101 to receive data from several inputs, such as cameras, media, audio, and multiple sensor input modules. The sensor inputs may include inputs from, for example, accelerometers, gyroscopes, magnetic sensors, temperature sensors, gravity sensors, and the like.

The AR processing unit 101 can use the storage unit 105, the CPU 106 and the GPU 107 for necessary processing for input data. The AR processing unit 101 can also use the reference information DB 103 to identify and recognize objects. The output of the AR processing unit 101 may include, for example, identification information and localization information.

The posture information may be used to determine a 2D / 3D pose of the target object. The identification information may be used to determine what the object is. The AR content manager 102 may be used to organize the final video / audio output into an output from the AR processor 101 and content from a remote or local content information DB 104. [

FIG. 2 is a diagram showing a configuration of the AR processing unit 101 shown in FIG. 1 according to an embodiment of the present invention. 2, an AR processing unit 101 according to an embodiment of the present invention includes a control unit 210, a recognition unit 220, an initialization unit 230, a tracking unit unit (240).

The control unit 210 can determine the branch to the recognition process through the recognition unit 220 or the tracking process through the tracking unit 240. [ While the processing of the recognition unit 220 and the tracking unit 240 can be executed in parallel, this control unit 210 makes the best decision for the optimized processing using the given input. For example, the main processing through the AR processing unit 101 may include three steps of 'recognition', 'initialization', and 'tracking'.

The recognition unit 220 may perform identification for the target object based at least in part on reference information provided through the local reference information DB 103 or the remote reference information DB if necessary.

In some examples, the recognition unit 220 may require reference information on the recognized specific object. The reference information may be a local reference information DB 103 provided inside the electronic device 100 as shown in FIG. 1, or a reference information DB 440 configured at a remote location as shown in FIG. For example, in the case of face recognition, it is possible to recognize authenticated faces and refer to an external reference face database to identify different faces. On the other hand, a QR (Quick Response) code generally requires only some specific rules for recognizing QR codes in the database, and is not usually updated dynamically in a normal case, so that it can have internal reference data of an electronic device have.

The initial posture calculation unit 230 may calculate the initial posture of the recognized target objects. Subsequently, the tracking unit 240 can dynamically calculate the posture of the object to continue tracking of the object, and the initial information for measuring the posture of the object is obtained from the output of the initial posture calculation unit 230 . Lastly, the tracking unit 240 may have a basic output of recognition information and initial posture information including an object posture.

3 is a block diagram showing an AR processing unit according to an embodiment of the present invention in more detail. For example, the detailed functions for each processing block included in the AR processing unit 101 shown in FIG. 2 may be shown in FIG.

3, the recognition unit 220 includes detailed steps of a feature detection unit 221, a descriptor calculation unit 222, and an image query unit 223 .

The feature point detector 221 can detect a feature point in the input image data when the image data is input. The feature point detector 221 may transmit the detected feature points to the descriptor calculator 222.

The descriptor calculator 222 may calculate a descriptor using the detected minutiae received from the minutia matching detector 221, generate the descriptor, and transmit the generated descriptor to the image searcher 223.

The descriptor calculator 222 may be configured to recognize one or more objects on the digital image according to various embodiments of the present invention and may determine a descriptor to be used for recognition of the object.

The descriptor calculator 222 may use at least some of the position, direction, and / or scale of the feature on the image to determine a descriptor to be used for recognition of the object, The above descriptor can be determined.

The descriptor calculator 222 may determine the brightness gradients of the pixels located in the area around the feature point so as to determine a descriptor to be used for recognition of the object. The descriptor calculator 222 may determine the brightness gradients of the pixels based on two or more non-orthogonal fixed directions that are different from the direction of the feature point. The descriptor calculator 222 may convert the brightness changes determined based on the fixed directions to the direction of the feature point.

The descriptor calculator 222 sets an area around the feature point, and an area around the feature point includes divided areas based on a direction of the feature point and a direction orthogonal thereto, They can overlap each other.

The image retrieval unit 223 retrieves the reference information corresponding to the input image data from the local reference information DB 103 or the remote reference information DB 440 using the calculated descriptor received from the descriptor calculation unit 222, The image data may be detected and an object on the input image data may be recognized through the detected at least one reference image data.

The initial posture calculation unit 230 may calculate an initial posture for an object identified through the minutiae detection in the input image data. The initial posture calculation unit 230 may include a feature matching unit 231 and an initial pose estimation unit 232.

The feature point matching unit 231 performs a matching operation on the feature points using the calculated descriptor received from the recognition unit 220 and outputs matching information about the feature points to the initial posture estimation unit 232 Lt; / RTI >

The initial posture estimating unit 232 may estimate the initial posture of the object included in the input image data through the matching information on the minutiae received from the minutiae matching unit 231. [

The tracking unit 240 can dynamically track changes in the posture with respect to the object in sequentially input image data.

The tracking unit 240 acquires initial information that can estimate an initial posture of the object of the input image data from the initial posture calculation unit 230, , And can dynamically calculate a pose change for the object. The tracking unit 240 may output recognition information indicating the type of the object and local information indicating the posture of the object in each of the sequentially received input image data.

The tracking unit 240 includes an object pose prediction unit 241, a feature detection unit 242, a descriptors calculation unit 243, a feature point matching unit 242, a matching unit 244 and a pose estimation unit 245. [

The posture predicting unit 241 can predict the posture of the object on the next input image data through the posture of the object estimated in each of the previously input at least one input image data.

The minutia detection unit 242 may detect minutiae points in the input image data sequentially received after the initial posture estimation for the object included in the input image data and transmit the detected minutiae points to the descriptor calculator 243 .

The descriptor calculator 243 may calculate descriptors using the feature points of the input image data received from the feature point detector 242 and may transmit the calculated descriptors to the feature point matching unit 244. [

The feature point matching unit 244 performs a matching operation on the feature points using the calculated descriptor received from the descriptor calculator 243 and transmits the matching information about the feature points to the attitude estimator 245 .

The posture estimating unit 245 dynamically estimates the posture change of the object in each of at least one input image data sequentially received using the matching information on the minutiae points received from the minutiae matching unit 244 And output recognition information indicating the type of object included in each input image data and local information indicating the posture of the object.

4 is a diagram of a system according to an embodiment of the present invention. Referring to FIG. 4, a system according to an embodiment of the present invention may include an electronic device 410, a communication network 420, and a content server 430.

The electronic device 400 may include at least some or all of the functions of the electronic device 100 described above with reference to FIG.

The communication network 420 may be configured without regard to its communication mode such as wired and wireless, and may be a personal area network (PAN), a local area network (LAN), a metropolitan area Network), a wide area network (WAN), and the like. Also, the communication network 420 may be a known World Wide Web (WWW) and may use a wireless transmission technology used for short-distance communication such as Infrared Data Association (IrDA) or Bluetooth It is possible. In addition, the communication network 420 may include a cable broadcasting communication network, a terrestrial broadcasting communication network, a satellite broadcasting communication network, or the like for receiving broadcast signals.

The content server 430 may perform at least one of recognition of an object included in an image, calculation of an initial posture of a recognized object, and tracking of an object according to an embodiment of the present invention. For example, various functions according to an embodiment of the present invention, which may be processed in the electronic device 100 of FIG. 1, may be distributed or distributed by the electronic device 410 and the content server 430 of FIG. 4 .

The content server 430 may include a reference information DB 440 and a content information DB 450. The content server 450 may include a reference information DB 440 and a content information DB 450, And provide the electronic device 410 with reference information or content information stored in the electronic device 410.

Meanwhile, in the embodiment of the present invention, each functional unit and module may mean a functional and structural combination of hardware for performing the technical idea of the embodiment of the present invention and software for driving the hardware. For example, each functional unit may refer to a logical unit of a predetermined code and a hardware resource for executing the predetermined code, and may be a code physically connected to the functional unit, But can be easily deduced to the average expert in the field of embodiments of the invention.

Further, the implementations described in connection with one or more of the above FIGS. 1-4 may be at least a part of the system or apparatus shown in FIG. 11 and / or FIG. The figures and descriptions provide various implementations and are not intended to limit the scope of the disclosure. The above description and the illustrated implementations may be adapted for application to the various embodiments disclosed herein. Those skilled in the art will recognize that one or more of the elements described and / or illustrated above may be omitted or changed in the embodiments disclosed herein. In addition, other elements may be added as needed to the embodiments described herein. In the embodiments described herein, one or more methods, steps or algorithms may be implemented using one or more of the components described and / or shown in the above described implementations.

5 to 10, an object is recognized through the recognition unit 220 and / or the initial posture calculation unit 230 according to an embodiment of the present invention, and an initial posture of an object included in the image is determined The calculation method will be described in detail.

The recognition process in the augmented reality includes a query process of finding out which object an image includes and an initial process of calculating an initial posture of the object recognized as a result of the query process And a localization step. According to the embodiment of the present invention, only data necessary for each step are loaded into a memory, and a method of quickly and accurately recognizing an object and attitude estimation using a small-sized memory is disclosed.

5 is a flowchart illustrating a procedure for operating an electronic device according to an embodiment of the present invention. Referring to FIG. 5, at operation 501, when an image is input, at operation 503, a feature point on the input image may be detected and a descriptor may be generated.

In operation 505, the efficiency of memory use can be increased by loading only the identification data into the memory in order to determine the target object. In operation 507, the target object can be determined by recognizing the object from the detected feature point and the loaded identification data (e.g., KD tree data).

In operation 509, a reference descriptor is loaded into the memory. In operation 511, the loaded reference descriptor may be used to determine a posture of the target object.

6 is a flow diagram illustrating a procedure for operating an electronic device in accordance with various embodiments of the present invention. Figure 6 may be a more detailed embodiment of Figure 5.

Referring to FIG. 6, an electronic device according to an embodiment of the present invention may initially load a KD tree for a query step from a local repository or server (e.g., a content server) into memory at an operation 603. At this time, KD tree for dark feature points and KD tree for bright feature points may be separately loaded according to various embodiments of the present invention. Meanwhile, according to the embodiment of the present invention, the descriptor for the reference features may not be loaded at this stage. By doing so, the efficiency of memory use can be increased and the processing speed can be improved.

If image data is input in operation 601, a feature may be detected from the image data input in operation 605. The method for detecting the feature points may be FAST, AGAST, Moravec corner detector, Harris & Stephens corner detector, SUSAN, and the present invention is not limited thereto. Further, a descriptor may be generated by performing a description on the detected minutiae. Various methods such as BRIEF, SIFT, SURF, and FREAK may be applied as the method for the description, and the present invention is not limited to the above methods.

In addition, according to various embodiments of the present invention, for each detected feature point, a bright attribute may be assigned to a bright feature point, and a dark attribute may be assigned to a dark feature point relative to surrounding pixels, . FIG. 7A is a view showing feature points having dark attributes according to an embodiment of the present invention, and FIG. 7B is a view showing feature points having bright attributes according to an embodiment of the present invention.

In operation 607, the feature points may be masked before being used in the query step. Since the query step is for finding a new object, it is possible to exclude an area for an object that is already recognized and tracked on the entire image. Therefore, among the minutiae, those detected in the area of the tracked object can be masked and excluded in the subsequent matching process.

The unmasked feature points are matched with the feature points of the reference objects using the KD tree according to an embodiment of the present invention. According to various embodiments of the present invention, the KD tree can be divided into a KD tree for bright feature point matching and a KD tree for dark feature point matching.

If the attribute of the feature point has a property of 'bright' in operation 609, it is not necessary to match the feature point having the attribute of 'dark'. Thus, at operation 611, a comparison can be performed using a light KD tree. On the other hand, if the attribute of the feature point has an attribute of 'darkness' in operation 609, it is not necessary to match with the feature points having the property of 'brightness'. Thus, at operation 613, a comparison can be performed using a dark KD tree. As described above, according to the embodiment of the present invention, if matching is performed according to the attributes of lightness and darkness, the probability of matching normally can be higher than when matching all feature points.

On the other hand, the KD tree structure can be represented as shown in FIG. The feature descriptor described above is a multi-dimensional vector. Referring to FIG. 9, an internal node of the KD tree may include a dimension to be compared with a value to be compared. Thus, for example, if the dimension to be compared is fourth and the value is 0.5, the fourth value of the input minutiae descriptor is compared with 0.5. As a result of the comparison, if the value is less than 0.5, it goes to a left child node, and if it is greater than 0.5, it goes to a right child node. The above process can be repeatedly performed until reaching a leaf node. Each leaf node is mapped with an ID of a reference object to which the feature point descriptor finally matched.

In operation 615, the frequency of IDs corresponding to the histogram may be increased by using the ID of the searched object. At this time, a histogram for each object ID can be generated by matching all the un-masked feature points using the KD tree and increasing the frequency of the corresponding object ID. Object IDs that are above a certain threshold in the histogram may be the result of the query process.

The retrieved objects are highly likely to be present in the current image. Therefore, after each object is localized, it is verified whether the object is actually present on the image, and the initial pose of the object is calculated.

In the above description, a method of increasing the frequency of the histogram has been described. At this time, according to various embodiments of the present invention, various modified embodiments may also exist in the method of increasing the frequency of the histogram.

For example, as shown in FIG. 10, when the inputted minutiae reaches a specific leaf node in the KD tree, n nearest neighborhood nodes can be considered in addition to the corresponding node. As the number of reference objects increases, the success rate of accurate matching may gradually decrease. Accordingly, rather than recognizing the object through one leaf node, the minutiae having similar values of the neighboring nodes are also found according to various embodiments of the present invention, and the result of the query can be obtained more accurately by being reflected in the histogram.

A method of considering the n nearest neighborhoods can be represented as shown in FIG. Referring to FIG. 10, if the input minutiae reaches the leaf node a and takes three closest neighbors for the leaf node a, it goes up to the parent node twice, (B), (c), (d), and so on. Referring to FIG. 10, the nodes of the leaf nodes (a), (b), (c), and (d) may be similar to each other because they have passed through an internal node test that is almost common until arriving at the leaf node.

Thus, if four leaf nodes are determined, the ID of the object corresponding to each leaf node is increased on the histogram. At this time, according to various embodiments of the present invention, it is possible to increase all the IDs with the same weight, or to give a higher weight to the leaf nodes closer to each other (for example, leaf node having a shorter path on the KD tree) Can be. In operation 617, the ID of the most probable object may be determined by the histogram.

As such, when the query phase is completed, an initial posture localization step may be performed through operations 619 through 625. [

Referring to FIG. 8, identification data (for example, a KD tree) for a query step and descriptors for each object may be separately stored in a memory or a storage as shown in FIG. At this time, according to the embodiment of the present invention, only the descriptor corresponding to the ID of the object found first in operation 619 is loaded into the memory and used, thereby improving the efficiency and processing speed of the memory.

At operation 621, the memory loaded descriptor may be used to directly create a KD tree for matching in the initial posture calculation step. At this time, according to the embodiment of the present invention, two kinds of KD trees can be generated considering the brightness and the darkness property for a higher matching probability.

In addition, non-masked input minutiae can be matched with reference minutiae again through the KD tree. At this time, the KD tree used in the query step matches with the minutiae extracted from all reference objects, while the KD tree used in the initial attitude calculation step performs matching with the minutiae in one object. As described above, since the KD tree in the initial posture calculation step is limited to a single object to be matched, accurate feature point matching can be performed with a high probability.

A matching pair between the input minutiae and the reference minutiae through the minutia matching can be generated. There may be pairs that are normally matched, and pairs that are not matched correctly. A method such as RANSAC, PROSAC, MLESAC, and M-estimator can be used to filter the erroneously matched pairs. If the normally matched pair is determined, an operation 623 may calculate the posture of the object from the pair. Various methods such as p3p algorithm can be used for the calculation of the posture, and the embodiments of the present invention are not limited to the specific method. If the matching pair is less than a certain threshold, then the object is likely not actually on the image. Therefore, the object can be ignored.

After the pose estimation process ends and the object found in the query step has an object that does not perform the initial posture calculation step, the initial posture calculation step can be performed again on the object. At this time, before performing the initial posture calculation step, in operation 625, the minutiae points belonging to the object found previously can be masked.

As described above, the electronic device 100 for recognizing an object and calculating an initial posture of an object according to an embodiment of the present invention may be any device including a controller, and may be a camera, a portable device, a mobile terminal, A communication terminal, a portable communication terminal, a portable mobile terminal, or the like. For example, the electronic device may be a digital camera, a smart phone, a mobile phone, a game machine, a television (TV), a display device, a car head unit, a notebook computer, a laptop computer, a tablet computer, Personal Digital Assistants), a navigation device, a bank ATM, a POS device in a store, and the like. Further, the electronic device in the present invention may be a flexible device or a flexible display device.

Hereinafter, with reference to FIG. 11, a detailed structure of an electronic device 100 to which embodiments of the present invention can be applied will be described as an example.

11 is a block diagram showing the detailed structure of an electronic device 100 according to an embodiment of the present invention. 11, an electronic device 100 according to an exemplary embodiment of the present invention includes a controller 110, a mobile communication module 120, a multimedia module 140, a camera module 150, an input / output module 160, A sensor module 170, a status instruction unit 171, a storage unit 175, a power supply unit 180, and a touch screen 190. [

More specifically, the electronic device 100 may be connected to an external electronic device (not shown) using at least one of the mobile communication module 120, the connector 165, and the earphone connection jack 167. In addition, the electronic device 100 may be connected to one of other portable or electronic devices, such as a mobile phone, a smart phone, a tablet PC, a desktop PC, and a server, using wire or wireless.

The communication module 120 may include a mobile communication module 121, a sub communication module 130, and a broadcasting communication module 141. The sub communication module 130 may include at least one of a wireless LAN module 131 and a local communication module 132. The multimedia module 140 may include at least one of an audio reproducing module 142 and a moving picture reproducing module 143. The camera module 150 may include at least one of a first camera 151 and a second camera 152. The camera module 150 may further include a flash 153, a motor unit 154, and a barrel unit 155. The input / output module 160 may include at least one of a button 161, a microphone 162, a speaker 163, a vibration element 164, a connector 165 and a keypad 166.

The control unit 110 stores a ROM (Read Only Memory) 112 storing a control program for controlling the electronic apparatus 100 and a signal or data input from the outside of the electronic apparatus 100 Or a RAM (Random Access Memory) 113 used as a storage area for a job to be performed in the electronic device 100. The CPU 111 may include a single core, a dual core, a triple core, or a quad core. The CPU 111, the ROM 112, and the RAM 113 may be interconnected via an internal bus.

The control unit 110 includes a communication module 120, a multimedia module 140, a camera module 150, an input / output module 160, a sensor module 170, a storage unit 175, a power supply unit 180, The touch screen 190, and the touch screen controller 195, as shown in FIG.

The control unit 110 may also detect a user input event, such as a hovering event, as the input unit 168 approaches or is located proximate the touch screen 190. The control unit 110 may detect various user inputs received through the camera module 150, the input / output module 160, and the sensor module 170 as well as the touch screen 190. The user input may include various types of information input into the apparatus 100 such as a gesture, a voice, a pupil movement, an iris recognition, a biological signal, etc. of the user. The control unit 110 may control the predetermined operation or function corresponding to the detected user input to be performed in the apparatus 100. [ Further, the control unit 110 can output the control signal to the input unit 168 or the vibration element 164. This control signal may include information about the vibration pattern, and the input unit 168 or the vibration element 164 generates vibration according to the vibration pattern.

In addition, the electronic device 100 may include at least one of a mobile communication module 121, a wireless LAN module 131, and a local communication module 132 according to performance.

The mobile communication module 121 may control the electronic device 100 to be connected to an external electronic device through mobile communication using at least one or more than one antenna (not shown) under the control of the controller 110 have. The mobile communication module 121 is connected to a mobile phone (not shown) having a telephone number input to the electronic device 100, a smart phone (not shown), a tablet PC or other electronic device (not shown) And may transmit / receive wireless signals for calls, text messages (SMS) or multimedia messages (MMS).

The sub communication module 130 may include at least one of a wireless LAN module 131 and a local area communication module 132. For example, it may include only the wireless LAN module 131, only the short range communication module 132, or both the wireless LAN module 131 and the short range communication module 132.

The wireless LAN module 131 may be connected to the Internet at a place where a wireless access point (AP) is installed under the control of the controller 110. [ The wireless LAN module 131 may support the IEEE 802.11x standard of the Institute of Electrical and Electronics Engineers (IEEE). The short-range communication module 132 can perform short-range wireless communication between the electronic device 100 and the external electronic device under the control of the controller 110. [ The short-distance communication method may include bluetooth, infrared data association (IrDA), WiFi-Direct communication, and Near Field Communication (NFC).

The broadcast communication module 141 receives a broadcast signal (e.g., a TV broadcast signal, a radio broadcast signal, or a data broadcast signal) transmitted from a broadcast station through a broadcast communication antenna (not shown) under the control of the controller 110, (E. G., An Electric Program Guide (EPG) or an Electric Service Guide (ESG)).

The multimedia module 140 may include an audio reproducing module 142 or a moving picture reproducing module 143. The audio reproduction module 142 may reproduce a digital audio file (for example, a file having a file extension of mp3, wma, ogg, or wav) stored or received in the storage unit 175 under the control of the controller 110. [ The moving picture playback module 143 may play back digital moving picture files (for example, files having file extensions mpeg, mpg, mp4, avi, mov, or mkv) stored or received under the control of the controller 110. [ At this time, the multimedia module 140 may be integrated into the controller 110. [

 The camera module 150 may include at least one of a first camera 151 and a second camera 152 for capturing a still image, a moving image, or a panoramic image under the control of the control unit 110. [ The camera module 150 includes a barrel section 155 for zooming in / zooming out to photograph a subject, a motor section 154 for controlling the movement of the barrel section 155, an auxiliary light source And a flash 153 that provides a flash memory. The first camera 151 may be disposed on the front surface of the electronic device 100 and the second camera 152 may be disposed on the rear surface of the electronic device 100.

The input / output module 160 includes at least one button 161, at least one microphone 162, at least one speaker 163, at least one vibration element 164, a connector 165, a keypad 166, An earphone connection jack 167, and an input unit 168. In addition, This input / output module 160 is not limited to this, and a cursor control such as a mouse, trackball, joystick or cursor direction keys may be provided for controlling the movement of the cursor on the touch screen 190 have.

A button 161 may be formed on the front, side, or rear of the housing (or case) of the electronic device 100 and may include a power / lock button, a volume button, a menu button, a home button, And a search button. The microphone 162 may receive voice or sound under the control of the controller 110 to generate an electrical signal. The speaker 163 outputs sound corresponding to various signals or data (for example, wireless data, broadcast data, digital audio data, digital moving picture data, etc.) to the outside of the electronic device 100 under the control of the controller 110 Can be output. The speaker 163 can output a sound corresponding to the function performed by the electronic device 100 (for example, a button operation sound corresponding to a telephone call, a ring back tone, a voice of the other user). The speaker 163 may be formed at one or a plurality of positions at appropriate positions or positions of the housing of the electronic device 100.

The vibration element 164 can convert the electrical signal into mechanical vibration under the control of the control unit 110. [ For example, when the electronic device 100 in the vibration mode receives a voice or video call from another device (not shown), the vibration device 164 operates. The vibration element 164 may be formed in the housing of the electronic device 100 in one or more. The vibrating element 164 may operate in response to user input through the touch screen 190.

The connector 165 may be used as an interface for connecting the electronic device 100 to an external electronic device or a power source (not shown). The control unit 110 can transmit data stored in the storage unit 175 of the electronic device 100 to an external electronic device or receive data from the external electronic device via a wired cable connected to the connector 165. [ The electronic device 100 may receive power from a power source via a wired cable connected to the connector 165, or may charge a battery (not shown) using a power source.

The keypad 166 may receive key input from a user for control of the electronic device 100. The keypad 166 may include a physical keypad (not shown) formed on the electronic device 100 or a virtual keypad (not shown) displayed on the touch screen 190. The physical keypad formed on the electronic device 100 may be excluded depending on the performance or structure of the electronic device 100. An earphone (not shown) may be inserted into an earphone connecting jack 167 and connected to the electronic device 100.

The input unit 168 may be inserted and stored within the electronic device 100 and may be withdrawn or disconnected from the electronic device 100 during use. A disassembly / recognition switch 169 is provided in one area of the electronic device 100 in which the input unit 168 is inserted. The disassemble / disassemble recognition switch 169 operates in response to the attachment / detachment of the input unit 168, The attachment recognition switch 169 can output a signal corresponding to the attachment and detachment of the input unit 168 to the control unit 110. The removal / attachment recognition switch 169 can be configured to be in direct or indirect contact with the input unit 168 when the input unit 168 is mounted. Thus, the removal / attachment recognition switch 169 detects a signal corresponding to mounting or dismounting of the input unit 168 (that is, mounting of the input unit 168 Or a signal for notifying the separation), and outputs the generated signal to the control unit 110. [

The sensor module 170 includes at least one sensor that detects the condition of the electronic device 100. [ For example, the sensor module 170 may include a proximity sensor that detects whether the user is accessing the electronic device 100, an ambient light sensor (not shown) that detects the amount of light around the electronic device 100, A motion sensor (not shown) that detects the motion of device 100 (e.g., rotation of electronic device 100, acceleration or vibration of electronic device 100) A geomagnetic sensor for detecting the point of the compass, a gravity sensor for detecting the direction of action of gravity, an altimeter for detecting the altitude by measuring the atmospheric pressure, a GPS module (157). ≪ / RTI >

The GPS module 157 receives radio waves from a plurality of GPS satellites (not shown) on the earth orbit and transmits the radio waves to the electronic device 100 using the time of arrival from the GPS satellite to the electronic device 100 100 can be calculated.

The storage unit 175 is connected to the communication module 120, the multimedia module 140, the camera module 150, the input / output module 160, the sensor module 170, or the touch screen 190 The signal or data to be input / output can be stored. In addition, various status information and setting information of the electronic device 100 may be stored according to an embodiment of the present invention.

The storage unit 175 may store control programs and applications for controlling the electronic device 100 or the control unit 110. [ One of the control programs or applications may be a messenger client application installed in accordance with an embodiment of the present invention.

The term 'storage unit' refers to a storage unit 175, a ROM 112 in the control unit 110, a RAM 113 or a memory card (for example, an SD card or a memory stick) mounted on the electronic device 100 May be used to refer to any data storage device in the system. The storage unit 175 may include a nonvolatile memory, a volatile memory, a hard disk drive (HDD), or a solid state drive (SSD).

In addition, the storage unit 175 may store various applications of various functions such as navigation, video calls, games, time-based alarm applications to users, etc., and images for providing a graphical user interface (GUI) (Such as menu screens, idle screens, etc.) or operating programs necessary to drive the electronic device 100, user data, documents, methods associated with the method of processing touch input, Images photographed by the camera, and the like.

In addition, the storage unit 175 is a medium that can be read by a machine (e.g., a computer), and the term " machine readable medium " refers to a medium that provides data to the machine Can be defined. In addition, the storage unit 175 may include non-volatile media and volatile media. All such media must be of a type such that the commands conveyed by the medium can be detected by a physical mechanism that reads the commands into the machine.

The machine-readable medium includes, but is not limited to, a floppy disk, a flexible disk, a hard disk, a magnetic tape, a compact disc read-only memory (CD-ROM) A punch card, a paper tape, a programmable read-only memory (PROM), an erasable PROM (EPROM), a flash-EPROM, and an embedded multimedia Card (eMMC (embedded Multi Media Card)).

The power supply unit 180 may supply power to one or a plurality of batteries disposed in the housing of the electronic device 100 under the control of the controller 110. One or more batteries supply power to the electronic device 100. In addition, the power supply unit 180 can supply power to the electronic device 100 from an external power source through a wired cable connected to the connector 165. In addition, the power supply unit 180 may supply power to the electronic device 100 wirelessly input from an external power source through a wireless charging technique.

The electronic device 100 may then include at least one touch screen 190 that provides a user graphical interface corresponding to various services (e.g., call, data transfer, broadcast, photo taking) to the user. The touch screen 190 may output an analog signal corresponding to at least one user input to the user graphic interface to the touch screen controller 195.

The touch screen 190 may receive at least one user input via a user's body (e.g., a finger including a thumb) or an input unit 168 (e.g., a stylus pen, an electronic pen, etc.) . The touch screen 190 may be implemented by, for example, a resistive method, a capacitive method, an infrared method, an acoustic wave method, or a combination thereof.

The touch screen 190 also includes at least two touch panels capable of sensing touches or accesses of the finger and input unit 168, respectively, to receive inputs from the fingers and the input unit 168 . At least two touch panels provide different output values to the touchscreen controller 195 and the touchscreen controller 195 recognizes the values entered at the at least two touchscreen panels differently, The input from the input unit 168 or the input from the input unit 168 can be distinguished.

Further, the touch is not limited to the contact between the touch screen 190 and the user's body or the touchable input means, and may be a contactless touch (e.g., between the touch screen 190 and the user's body or a touchable input means) 1 mm or less). The detectable interval on the touch screen 190 may vary depending on the performance or structure of the device 100

The touch screen controller 195 converts the analog signal input from the touch screen 190 into a digital signal and transmits the digital signal to the controller 110. [ The control unit 110 may control the touch screen 190 using the digital signal received from the touch screen controller 195. The touch screen controller 195 detects a value (e.g., a current value) output through the touch screen 190 to confirm a hovering interval or a distance as well as a user input position, , Z coordinate), and provides it to the control unit 110. In addition, the touch screen controller 195 may detect a value (e.g., a current value) output through the touch screen 190 to detect a pressure at which the user input means presses the touch screen 190, May be converted into a digital signal and may be provided to the controller 110.

The methods according to the exemplary embodiments of the present invention described above can be implemented in the form of program instructions that can be executed through various computer means and recorded in a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, and the like, alone or in combination. Program instructions to be recorded on the medium may be those specially designed and constructed for the present invention or may be known and available to those of ordinary skill in the computer software arts.

Further, the methods according to the exemplary embodiment of the present invention may be embodied and stored in the form of program instructions in the storage unit of the above-described electronic device 100, (RAM) 113 included in the control unit 110 for execution of the programs. Accordingly, the control unit 110 performs control of the hardware components contained in the device 100, corresponding to the program instructions according to the methods according to the exemplary embodiment of the present invention, The data generated while executing the methods according to the embodiment may be stored temporarily or continuously in the storage unit 150 and the UI necessary for executing the methods according to the exemplary embodiment of the present invention may be provided to the touch screen controller 172 .

12 is a block diagram showing a software structure of an electronic device to which embodiments of the present invention can be applied. 12, the software structure of an electronic device to which the embodiments of the present invention can be applied can be divided into levels of an application, an application framework, a library, a kernel, and the like.

The application level may include a home, dialer, SMS / MMS, IM, browser, camera, alarm, calculator, address book, voice dial, email, calendar, media player, album, The application framework level may include an activity manager, a window manager, a content provider, a view system, a notification manager, a package manager, a phone manager, a resource manager, a location manager, and the like.

The library level may include a surface manager, a media framework, SOLite, OpenGL / ES, FreeType, Webkit, SGL, SSL, Libc, Android runtime (core library, The kernel level may include a display driver, a camera driver, a Bluetooth driver, a shared memory driver, a Bider (IPC) driver, a USB driver, a keypad driver, a WiFi driver, an audio driver, a power manager, and the like.

As described above, the present invention has been described with reference to particular embodiments, such as specific elements, and specific embodiments and drawings. However, it should be understood that the present invention is not limited to the above- And various modifications and changes may be made thereto by those skilled in the art to which the present invention pertains.

Accordingly, the spirit of the present invention should not be construed as being limited to the embodiments described, and all of the equivalents or equivalents of the claims, as well as the following claims, belong to the scope of the present invention .

100: Electronic device 101: AR processor
102: AR content management unit 103: Reference information DB
104: Content information DB 105:
106: CPU 107: GPU
210: control unit 220:
221: minutiae point detector 222:
223: image search unit 230: initial posture calculation unit
231: feature point matching unit 232:
240: tracking unit 241:
242: minutiae point detector 243:
244: feature point matching unit 245:
410: electronic device 420: communication network
430: Content server 440: Reference information DB
450: Content information DB

Claims (39)

A method of operating an electronic device,
The method comprising: recognizing at least one object from a digital image,
Generating at least one descriptor from the digital image;
Determining an object on the digital image based at least in part on identification data corresponding to the at least one descriptor and the at least one reference object; And
Determining an orientation of the object based at least in part on reference descriptors corresponding to the determined object and the at least one descriptor.
The method according to claim 1,
Wherein the identification data is indexed in a tree structure.
3. The method according to claim 2,
A K-Dimensional (KD) tree, a randomized tree, or a spill tree.
2. The method of claim 1, wherein the act of recognizing the object further comprises loading the identification data onto a memory,
Wherein the act of loading the identification data comprises loading identification data indexed into a tree structure.
The method according to claim 1,
Wherein the reference descriptors are indexed in a tree structure.
3. The method according to claim 2,
A K-Dimensional (KD) tree, a randomized tree, or a spill tree.
The method of claim 1, wherein the determining of the posture of the object comprises:
Loading the reference descriptors corresponding to the determined object onto a memory; And
And indexing the reference descriptors into a tree structure.
2. The apparatus of claim 1, wherein the descriptor and the reference descriptor comprise:
Wherein the attribute has an attribute given based on a relationship between a feature point corresponding to each descriptor and at least one adjacent pixel adjacent to the feature point.
9. The method of claim 8,
Wherein the relative brightness of the adjacent pixels is an attribute indicating relative brightness of the corresponding feature point with respect to the neighboring pixels.
2. The method of claim 1, wherein determining an object on the image comprises:
Determining an object corresponding to an identifier matching the set number of target descriptors among identifiers corresponding to leaf nodes reached according to path tracing for each descriptor as the object . ≪ / RTI >
2. The method of claim 1,
The identifier corresponding to the reached leaf node and the identifiers corresponding to the neighboring leaf node branched from the superior node having the node distance set from the leaf node according to the path trace for each target scripter, And determining the object corresponding to the matching identifier as the target object.
12. The method of claim 11, wherein the determining of the at least one target object comprises:
And applying different weights to the identifier corresponding to the leaf node and the identifier corresponding to the neighboring leaf node.
13. The method of claim 12, wherein the determining of the at least one target object comprises:
And applying different weights to the identifiers corresponding to the neighboring leaf nodes according to the node distances to the leaf nodes.
2. The method of claim 1, wherein generating the at least one target descriptor comprises:
And extracting feature points from regions other than the region where the object being tracked is located to generate the at least one target descriptor.
2. The method of claim 1, wherein generating the at least one target descriptor comprises:
And generating a target descriptor for remaining minutiae except minutiae extracted from an area where an object being tracked is located among minutiae extracted from the target image.
The method according to claim 1,
Further comprising generating identification data of a set type using the loaded reference descriptor,
Wherein the determining of the posture of the target object comprises determining the posture of the target object based on the at least one target descriptor and the created identification data.
17. The method according to claim 16, wherein the operation of creating the identification data comprises:
And creating a plurality of identification data based on different attributes.
17. The method according to claim 16, wherein the operation of creating the identification data comprises:
And generating identification data of a tree structure capable of determining an identifier of a reference object corresponding to each of the target descriptors.
17. The method of claim 16,
Is a property indicating the brightness of a corresponding feature point with respect to adjacent pixels.
A method of operating an electronic device,
The method comprising: recognizing an object from a digital image based on object data stored in a database and descriptor data associated with each object data,
Determining an object on the image using the object data; And
Determining an orientation of the object based at least in part on one or more descriptors associated with the determined object.
21. The method of claim 20, wherein the act of determining an object on the image comprises loading the object data onto a memory of the electronic device.
21. The method of claim 20, wherein determining an orientation of the object comprises loading only one or more descriptors associated with the determined object onto a memory of the electronic device.
In an electronic device,
A memory configured to store a digital image; And
And a processor configured to process the digital image,
The processor comprising:
A recognition unit for generating at least one descriptor from the digital image and determining an object on the digital image based at least in part on identification data corresponding to the at least one descriptor and the at least one reference object; And
And an initial posture calculator for determining the posture of the object based at least in part on the reference descriptors corresponding to the determined object and the at least one descriptor.
24. The method of claim 23,
Wherein the target descriptor and the reference descriptor,
And an attribute assigned based on a relationship between a feature point corresponding to each descriptor and at least one adjacent pixel adjacent to the feature point.
The image processing apparatus according to claim 24,
And loads identification data having the same attribute as the target descriptor among a plurality of identification data based on different attributes.
25. The apparatus according to claim 24,
And loading a reference descriptor having the same attributes as the target descriptor.
25. The method of claim 24,
And is an attribute indicating a brightness of a corresponding feature point with respect to the adjacent pixels.
24. The method according to claim 23,
And a tree structure capable of determining an identifier of a reference object corresponding to each of the target descriptors.
29. The apparatus of claim 28,
And determines, as the target object, an object corresponding to an identifier matching the set number of target descriptors among the identifiers corresponding to the reached leaf nodes according to the path tracing for each target descriptor.
29. The apparatus of claim 28,
The identifier corresponding to the reached leaf node and the identifiers corresponding to the neighboring leaf node branched from the superior node having the node distance set from the leaf node according to the path trace for each target scripter, And determines an object corresponding to the matching identifier as the target object.
The apparatus of claim 30,
And applies different weights to the identifier corresponding to the leaf node and the identifier corresponding to the neighboring leaf node.
The information processing apparatus according to claim 31,
And applies different weights to the identifiers corresponding to the neighboring leaf nodes according to the node distances to the leaf nodes.
29. The method according to claim 28,
(K-Dimensional) tree, a Huffman tree, and a (Gagie) tree.
24. The apparatus according to claim 23,
And extracting feature points from regions other than the region where the object being tracked is located to generate the at least one target descriptor.
24. The apparatus according to claim 23,
And generates a target descriptor for remaining minutiae except for minutiae extracted in an area where an object being tracked is located among minutiae extracted from the target image.
24. The apparatus according to claim 23,
Creates identification data of a set type using the loaded reference descriptor, and determines a posture of the target object based on the at least one target descriptor and the created identification data.
37. The apparatus as claimed in claim 36,
And creates a plurality of identification data based on different attributes.
37. The apparatus as claimed in claim 36,
And generates identification data of a tree structure capable of determining an identifier of a reference object corresponding to each of the target descriptors.
37. The method of claim 36,
And is an attribute indicating a brightness of a corresponding feature point with respect to adjacent pixels.

Images