JP5963609B2 - Image processing apparatus and image processing method - Google Patents

Description

  The present invention relates to an image processing technique for image retrieval in units of objects using local features.

  There has been proposed a method of searching for a similar image using a local feature amount (local feature amount) of an image. In this method, first, characteristic points (local feature points) are extracted from an image (Non-Patent Document 1). And based on this local feature point and the surrounding pixel group, the feature-value (local feature-value) corresponding to this local feature point is calculated (nonpatent literature 2). Image retrieval is performed by matching local feature amounts.

  Conventional search result display methods often calculate the similarity between a query image and a search destination image (sample image) by matching local feature amounts, and display thumbnail images in order of similarity.

  On the other hand, an image recognition technique that performs machine learning using local feature amounts has also been proposed (Non-Patent Document 3). This is a technique for detecting a common object by preparing a plurality of learning images and performing machine learning, and has been put into practical use by a face detection technique in a digital still camera or the like. Compared to the above-described image search using local features, there is a characteristic that the rate of erroneous recognition is relatively small due to the effect of machine learning. However, as described above, the image recognition technology needs to perform machine learning in advance, and it is unrealistic to deal with everything.

  A proposal has been made to improve recognition accuracy by using both image recognition technology and image feature value search (Patent Document 1). In this method, a face image is detected from the reproduced image data, and personal identification information is extracted from the detected face image for each of a normal face image, a face image with sunglasses, and a face image with a hat. . Then, they are stored in association with each other, assuming that they belong to the same person. Thereafter, a face is detected from the captured image, and when there is a motion near the feature point of the face image, the personal identification information is switched based on the position where the motion has occurred and the direction of the motion. The person is identified including the hat and sunglasses. Therefore, it is possible to improve the individual identification accuracy and the face tracking accuracy.

C. Harris and M.J. Stephens, “A combined corner and edge detector,” In Alvey Vision Conference, pages 147-152, 1988. David G. Lowe, “Distinctive Image Features from Scale-Invariant Keypoints,” International Journal of Computer Vision, 60, 2 (2004), pp.91-110. Yanai, “Current Status and Future of General Object Recognition,” IPSJ Transactions on Computer Vision and Image Media Nov.2007, Vol.48 No.SIG 16 (CVIM 19)

JP 2011-76439 A

  In Patent Document 1, it is determined that the face detection result and the image features in the vicinity thereof are the same, and it is not a search process, but matching is performed by combining the face detection result and the image feature amount of other parts. doing. However, features of feature points based on organ detection and features such as hats near the face are different in shape and color features, and features used for face detection and image features used for image feature search are shared. There is no concept, and image recognition and image feature amount comparison are independent processes.

  Image search that produces good search results with the property that the object to be searched using local features is not chosen and the property that the target image objects for image recognition are limited but there are relatively few false detections. It has not been realized. The local feature amount extracted from the image is not used for image recognition to improve efficiency.

  Image recognition technology can be expected to perform a search with higher accuracy than image feature search by performing machine learning on a certain recognition target using a large amount of learning data. However, it is usual to give learning images with different appearances, and since the recognition likelihood does not immediately show similarity to the query image, it is not sufficient to search using the image recognition result alone. .

  Furthermore, since the number of image recognition targets increases with time, when applying a new image recognition result to an image registered in the past, it is usually necessary to perform re-registration processing. Must be in an environment where can be referenced. On the other hand, in a search using local feature amounts, a search result with similar feature amounts is obtained mechanically, and therefore the search result may be a completely different object when a human sees the search result.

  The present invention has been made in view of the above problems, and performs an image search that has both the property of not selecting an object to be searched using local features and the property of relatively few errors in image recognition. The purpose is to provide technology to realize.

In order to achieve the object of the present invention, for example, the image processing apparatus of the present invention, for each image, each attribute and recognition likelihood of one or more objects in the image obtained by image recognition processing for the image. An image processing apparatus capable of accessing a database registered in association with the image,
An identification means for acquiring an attribute of an object in the query image by performing an image recognition process on the query image, and specifying an image registered in the database in association with the acquired attribute;
The recognition likelihood registered in the database in association with the image specified by the specifying unit is set as the similarity of the specified image with respect to the query image, and the information about the specified image and the similarity are defined as an image. Output means for outputting as a search result.

  According to the configuration of the present invention, it is possible to realize an image search that has both the property of not selecting an object to be searched using local feature amounts and the property of relatively few image recognition errors.

The block diagram which shows the hardware structural example of an image processing apparatus. FIG. 3 is a block diagram illustrating an example of a functional configuration of the image processing apparatus. The block diagram which shows the function structural example of the image feature extraction and recognition process part 202 (202 '). The flowchart of the process which the feature point / local feature-value calculation part 303 performs. The figure explaining a scale space. The flowchart of an image registration process. The figure which shows the index schema in DB204. The figure which shows a registration image and a query image. The figure which shows each index registered into DB204. The flowchart of an image search process. The flowchart of an image search process. The flowchart of an image search process. The flowchart of the process which updates DB index.

  Preferred embodiments of the present invention will be described below with reference to the accompanying drawings. The embodiment described below shows an example when the present invention is specifically implemented, and is one of the specific examples of the configurations described in the claims.

<Introduction>
Before describing the first embodiment, an outline described in each embodiment will be described. A feature point is detected from the registered image, a local feature amount of the feature point is extracted, further image recognition processing is performed, metadata for recognition in the image is extracted, and these are also stored together. As preparation for image recognition, an image including a recognition target and an image not including the recognition target are prepared for learning and machine learning is performed in advance.

  The metadata to be recognized as a result of performing the image recognition processing includes at least the attributes to be recognized. Furthermore, at least one of recognition target area information and recognition likelihood is stored in the metadata. Note that local features include position information of feature points. At the time of retrieval, as in the case of registering a query image, image recognition processing is performed to extract recognition target metadata in the image.

  By making the local feature amount used in the image recognition processing common with the local feature amount used in the image search processing, the comparison result of the similarity of the local feature amounts and the image recognition result are not greatly different. For example, if the local feature amount is obtained from the luminance information and the image recognition uses color information, the appearance of the hit image will be greatly different.

  When image recognition processing is performed on the query image and the metadata to be recognized is obtained, matching is performed using at least one of the metadata to be recognized and the local feature amount of the region to be recognized. If there is no match, only local features are used for matching. As a result, it is possible to perform image search processing in which the influence of the fact that the object of image recognition is limited is reduced by the image feature amount search processing, while taking advantage of fewer false detections of image recognition.

  In addition, when the recognition target metadata is obtained by performing image recognition processing on the query image, it narrows down to images with the same recognition target attribute, performs local feature comparison in the recognition target region, and There is also a method of performing image feature amount comparison processing using local features when there is no image.

  As a merit of this, by comparing the local features in the recognition area of the registered image and the query image, the recognition result was used for narrowing down while reflecting the similarity with the query that could not be solved only by the recognition likelihood. Efficient search processing can be performed.

  There are also the following methods. In other words, when image recognition processing is performed on the query image and the metadata of the recognition target is obtained, matching is performed using the attributes of the recognition target, recognition likelihood is obtained, and image feature amount comparison is further performed using local features. To obtain the similarity and calculate the total similarity by the weighted sum of the recognition likelihood and the similarity. Then, when there is no recognition target, an image feature amount search process is performed using local features. As a result, although the processing cost is increased, it is possible to reduce omissions even when image recognition fails.

  When a recognition target with a low recognition rate is detected in the query image, a good result is obtained by reducing the weight to the recognition likelihood and setting a high value for a recognition target with a high recognition rate. It is done. Further, when the similarity weight is zero, useless processing can be prevented if image feature comparison processing using local features is not performed.

  By the way, when matching is performed using metadata, it is the simplest method to determine whether comparison of recognition metadata has the same attribute and convert the likelihood to the similarity. In addition, when matching is performed using local features, the similarity obtained by matching the local features included in the image recognition area is used to perform comparison with true similarity to the corresponding area of the query image. Is possible.

  Of course, the similarity obtained by integrating the likelihood and the similarity obtained by matching the local feature amount with the feature point of the entire image may be calculated. This merit is that the likelihood obtained by the image recognition can be gradually reflected in the search result, the influence of the failure of the image recognition can be reduced, and a more robust search can be performed.

  By the way, by using the same local features in the image recognition processing and the image feature search processing, even if the number of targets for image recognition increases, the already extracted local features can be used without reanalyzing the registered image. Image recognition can be performed. It is possible to additionally store the metadata obtained as a result.

  As a result, when the number of image recognition targets increases, it is possible to update the image search DB reflecting the image recognition targets increased at a low processing cost without analyzing the images registered in the past from the beginning. Become.

[First Embodiment]
First, a hardware configuration example of the image processing apparatus according to the present embodiment will be described with reference to the block diagram of FIG. Note that the configuration illustrated in FIG. 1 is merely an example, and any configuration may be employed as long as each configuration described below can be performed by the image processing apparatus. For example, a digital camera for taking an image and inputting it to the apparatus may be connected to the apparatus. Moreover, you may add the structure which can output the information obtained as a search result with this apparatus with respect to an external apparatus.

  The CPU 101 executes processing using computer programs and data stored in the RAM 105 and ROM 106, thereby controlling the operation of the entire image processing apparatus and executing each process described later as what the image processing apparatus performs. .

  The input device 102 is a keyboard having various function keys such as alphabet keys, hiragana keys, katakana keys, character symbol input keys such as punctuation marks, cursor movement keys for instructing cursor movement, etc., a mouse, a stick pointer, etc. It is configured. The user can input various instructions to the CPU 101 by operating the input device 102.

  The output device 103 is a display device configured by a CRT, a liquid crystal screen, or the like, and can display a processing result by the CPU 101 using an image, text, or the like.

  The storage device 104 stores an OS (Operating System), a database, which will be described later, and computer programs and data for causing the CPU 101 to execute each process which will be described later as performed by the image processing apparatus. Each information described later as known information is also stored in the storage device 104. Computer programs and data stored in the storage device 104 are appropriately loaded into the RAM 105 under the control of the CPU 101 and are processed by the CPU 101. The storage device 104 can be a hard disk, CD-ROM, DVD-ROM, flash memory, or the like.

  The RAM 105 has an area for temporarily storing computer programs and data loaded from the storage device 104, a work area used when the CPU 101 executes various processes, and the like. That is, the RAM 105 can provide various areas as appropriate. The ROM 106 stores setting data and a boot program for the image processing apparatus. Each unit described above is connected to a bus (BUS) 107.

  Next, an image registration process to the image processing apparatus and an image search process for searching for a target image from registered images based on a query image will be described. A configuration example of functions of an image processing apparatus used for image registration processing and image search processing will be described with reference to the block diagram of FIG.

  The image feature extraction / recognition processing unit 202 extracts and extracts feature points (local feature points) and feature quantities (local feature quantities) from the registered image 201 loaded from the storage device 104 or input from an external device. An image recognition process is performed on the registered image 201 using the feature amount. Then, the image feature extraction / recognition processing unit 202 outputs the result of the image recognition processing as metadata. In this metadata, the coordinate position of the feature point in the registered image 201 and the feature amount at the feature point, the attribute information indicating the attribute of the object recognized in the registered image 201, the image recognition likelihood, the area of the object Area information to indicate. Details of the image feature extraction / recognition processing unit 202 will be described later with reference to FIG.

  In the present embodiment, the local feature value is made common so that the comparison result of the local feature value and the result of the image recognition are not greatly different. An example in which the results differ greatly is that local feature amounts are obtained from luminance information. When color information is used in image recognition, the appearance of images that hit each other when they are hybridized differs greatly. Naturally results.

  The image feature / recognition result registration processing unit 203 registers the metadata output by the image feature extraction / recognition processing unit 202 in the database (DB) 204 in association with the registered image 201 from which the metadata is generated. As described above, each time the registered image 201 is input, the image processing apparatus 200 generates the metadata described above from the registered image 201, and registers the generated metadata in the DB 204 in association with the registered image 201. A so-called image registration process is performed.

  The image feature extraction / recognition processing unit 202 ′ has the same configuration as the image feature extraction / recognition processing unit 202 and performs the same operation except that the received image is different from the image feature extraction / recognition processing unit 202. is there. The image feature extraction / recognition processing unit 202 ′ extracts feature points and feature amounts from the query image 205 loaded from the storage device 104 or input from an external device, and uses the extracted feature amounts to extract the query image. An image recognition process for 205 is performed. Then, the image feature extraction / recognition processing unit 202 'outputs the result of the image recognition processing as metadata. The metadata includes the coordinate position of the feature point in the query image 205 and the feature amount at the feature point, the attribute information indicating the attribute of the object recognized in the query image 205, the image recognition likelihood, and the area of the object. Area information to indicate. Details of the image feature extraction / recognition processing unit 202 'will be described later with reference to FIG.

  The image comparison processing unit 206 performs an image search from the DB 204 using the metadata generated from the query image 205 by the image feature extraction / recognition processing unit 202 ′, and outputs an image search result (search processing result) 207. The search here is not a search for comparing local feature amounts but a search for comparing whether or not the metadata of the image recognition result is identical or similar. Details of the image comparison processing unit 206 will be described later.

  The DB 204 is not limited to be provided in the image processing apparatus. If the image processing apparatus is directly or indirectly connected to the image processing apparatus so as to be accessible, the connection form is limited to a specific connection form. It is not a thing.

  2 may be configured by hardware. In the present embodiment, each unit (except for the recognition dictionary 305) except the DB 204 is implemented by a computer program, and the DB 204 and the recognition dictionary 305 are storage devices. This is implemented at 104. In the following description, each unit in FIG. 2 will be described as the subject of the operation, but actually, the corresponding operation is executed by the CPU 101 executing the computer program of the corresponding functional unit.

  Next, a functional configuration example of the image feature extraction / recognition processing unit 202 (202 ') will be described with reference to the block diagram of FIG.

  Upon receiving the input image 301 (corresponding to the registered image 201 in the case of the image feature extraction / recognition processing unit 202 and the query image 205 in the case of the image feature extraction / recognition processing unit 202 ′), the image input unit 302 receives this input. The image 301 is transferred to the feature point / local feature amount calculation unit 303.

  A feature point / local feature amount calculation unit 303 calculates a feature point and its feature amount from the input image 301 transferred from the image input unit 302. Details of the operation of the feature point / local feature amount calculation unit 303 will be described later.

  The image recognition processing unit 304 performs image recognition processing on the input image 301 using the feature points obtained by the feature point / local feature amount calculation unit 303 and the feature amounts with reference to the recognition dictionary 305. A plurality of pieces of image recognition target information are registered in the recognition dictionary 305. Then, the image recognition processing unit 304 outputs the above metadata as the local feature amount and the image recognition result 306 as a result of the image recognition process.

  Next, the feature point / local feature quantity calculation unit 303 will be described in more detail. As described in the related art, a method of searching for a similar image by matching using a local feature amount (local feature amount) of an image and an image feature amount search have been proposed.

  In the method using the local feature amount, the local feature amount is defined as information including a plurality of elements that are rotation invariant and enlargement / reduction invariant. Thereby, even when the image is rotated, enlarged or reduced, the search can be performed.

  The local feature amount is generally expressed as a vector. However, it is a theoretical story that local features are invariant to rotation and enlargement / reduction, and in actual digital images, linear interpolation is performed, so local features before image rotation and enlargement / reduction processing are performed. And some corresponding local feature after processing.

  In order to calculate the rotation-invariant local feature amount, for example, in Non-Patent Document 2, the main direction is calculated from the pixel pattern of the local region around the local feature point, and the local region is rotated based on the main direction when calculating the local feature amount. To normalize the direction.

  Further, in order to calculate the local feature quantity that does not change in size, the image of different scales is generated internally, and local feature points are extracted from the images of the respective scales and the local feature quantities are calculated. Here, a series of image sets having different scales generated inside is generally called a scale space, and an outline thereof is shown in FIG. The feature amount of each feature point also stores a rotation angle and a scale space in which the feature amount is calculated when normalizing the direction.

  Next, processing performed by the feature point / local feature amount calculation unit 303 will be described with reference to FIG. 4 showing a flowchart of the processing. In step S401, the input image 301 transferred from the image input unit 302 is received. In step S402, a luminance image of the input image 301 is generated.

  In step S403, a reduced image of the luminance image generated in step S402 is generated. In step S404, feature points are extracted from the reduced image. In step S405, feature amounts at the feature points are extracted.

  Next, details of the image comparison processing unit 206 will be described. There are various image collation methods based on the feature point / local feature quantity comparison. Here, a method using RANSAC described in the following document will be described.

M.M. A. Fischler and R.M. C. Bolles, “Random sample consensus: A paradigm formatfitting with applications to imaging analysis,” Commun. ACM, no. 24, vol. 6, pp. 381-395, June 1981.
For each feature point of the query image, a feature point of each registered image managed by the DB 204 that has a minimum distance between features is described in pairs. Next, three feature points are randomly selected from the query image, and an affine transformation matrix is obtained from the correspondence of the coordinates with the feature point group of the registered image that minimizes the distance between the features. Using this affine transformation matrix, the coordinates of the remaining feature points of the query image are converted to the coordinates of the registered image, and it is confirmed whether there is a pair of feature points that minimize the distance between the features in the vicinity. If you don't exist, you won't vote. Finally, when the number of votes reaches a predetermined value, it is determined that there is a partially matching area between the query image and the registered image, and the larger the number of votes, the larger the matching area.

  On the other hand, if the number of votes does not reach the predetermined value, three feature points are newly selected at random from the query image, and the processing is performed again from the processing for obtaining the affine transformation matrix. This reprocessing is determined. Repeat within the repeat count. If the number of votes does not exceed a predetermined value even when the repetition count is reached, it is determined that there is no partially matching area, and the comparison process is terminated. If there is a partially matching region, a feature point corresponding to the feature point of interest in the query image can be obtained using the pair of the affine transformation matrix obtained above and the one having the smallest feature distance.

  Furthermore, the rotation angle relationship between the query image and the registered image can be obtained from the rotation angle at the time of normalizing the direction when obtaining the feature point. Furthermore, it is possible to know the enlargement / reduction relationship between the query image and the registered image by using the scale space for obtaining the feature point.

  As described above, Non-Patent Document 3 describes the following technique as a general object recognition technique. A feature amount calculated from a group of images belonging to a certain recognition target category is used as learning data. The distribution model is estimated by maximum likelihood estimation using learning data, and the posterior probability is calculated for unknown data using the distribution model. Then, it is determined which category it belongs to by posterior probability maximization (Maximum A Posteriori, MAP) estimation, and this is used as a recognition result. In addition, there are many methods combining various models and machine learning methods, and the image recognition processing used in the present embodiment is not limited to any machine learning method as long as it is performed using local feature amounts.

  Next, image registration processing performed by the image processing apparatus 200 will be described with reference to FIG. 6 showing a flowchart of the processing. In step S601, the image input unit 302 acquires a registered image 201 loaded from the storage device 104 or input from an external device. In step S602, the feature point / local feature amount calculation unit 303 extracts the feature point and its feature amount from the registered image 201 acquired in step S601.

  In step S603, the image recognition processing unit 304 performs image recognition processing on the registered image 201 using the feature amount extracted in step S602. A machine-learned detector is used for the image recognition processing. When there are a plurality of types of image recognition targets, it is necessary to prepare a plurality of types of detectors accordingly.

  For each detector, images of various appearances in a category to be recognized as a correct image are prepared in advance, and images outside the category are prepared as non-correct images, and machine learning is performed. By using rotation / scaling-invariant local feature quantities such as SIFT, it is possible to recognize an image having a recognition target object that has been rotated, enlarged, or reduced.

  Furthermore, with regard to where the object exists in the image, the mask area is set for the image determined to be present, the inside of the image is scanned, the mask size is reduced, and the image is input to the detector. Then, the recognition likelihood is obtained. And it is estimated that the object used as recognition object exists in the mask area | region where the recognition likelihood became higher than the predetermined threshold value.

  In step S604, the image feature / recognition result registration processing unit 203 registers the result of the image recognition process performed in step S603 as metadata in the DB 204 in association with the registered image 201.

  Information registered in the DB 204 by the above image registration process, that is, the index schema in the DB 204 will be described with reference to FIG. FIG. 7A shows a configuration example of an image management index, which is registered in the DB 204 for each registered image.

  In the “image ID” field, an ID unique to the registered image 201 is registered, and this ID is, for example, an integer value unique to the registered image 201. A search result to be described later is represented by a set of an image ID and a similarity.

  In the “image data file name” field, the path name of the registered image 201 in the registration destination (for example, DB 204) where the registered image 201 is registered is registered. Of course, the registration destination of the registered image 201 is not limited to the DB 204, and may be another device.

  In the “recognition result” field, the ID (recognition ID) of each object recognized by the image recognition processing for the registered image 201 is registered. The recognition ID is also link information to the recognition result management index shown in FIG. The “recognition result” field is a variable-length field.

  In the “image feature amount” field, the feature amount ID (image feature ID) extracted from the registered image 201 is registered. The image feature ID is also link information to the image feature index shown in FIG. The “image feature amount” field is a variable-length field.

  Next, a configuration example of the recognition result management index will be described with reference to FIG. The recognition result management index is registered in the DB 204 for each recognized object.

  In the “recognition ID” field, an ID unique to the object recognized from the registered image 201 is registered, and this ID is, for example, an integer value unique to this object.

  In the “recognition attribute” field, an attribute value indicating the attribute of the object recognized from the registered image 201 is registered. In FIG. 7, if the attribute of the object is “human body”, the attribute value is “1”, and if the attribute of the object is “car”, the attribute value is “2”. The type and number of attributes are not limited to this. If there are N types of attributes, the attribute values range from 1 to N.

  In the “recognition likelihood” field, the recognition likelihood of the object recognized from the registered image 201 is registered, and is represented by a numerical value in the range of 0 to 100 (%), for example. Of course, the method of expressing the recognition likelihood is not limited to this.

  In the “recognition area” field, area information indicating a mask area (= recognition area) in which the recognition likelihood obtained by the image recognition processing performed in step S603 is higher than a predetermined threshold is registered. Here, the respective coordinate positions of the upper left corner vertex and the lower right corner vertex of the mask area are registered. Of course, the method of expressing the region information is not limited to this.

  In the “image feature quantity in the recognition area” field, the image feature ID of the feature quantity of each feature point in the area (recognition area) indicated by the area information defined in the “recognition area” field is registered. ing. The information registered in the “image feature amount in the recognition area” field is not used in the first embodiment, but is used in the second embodiment. Therefore, the information registered in this field is the second embodiment. This will be described in detail in the form.

  Next, a configuration example of the image feature index will be described with reference to FIG. The image feature index is registered in the DB 204 for each feature amount of the feature point, and stores the feature amount for one feature point in one record.

  In the “image feature ID” field, an ID unique to the feature amount is registered. In the “image ID” field, the image ID of the registered image from which the feature amount is extracted is registered. In the “feature point coordinates” field, the coordinate position of the feature point on the registered image is registered. In the “local feature value” field, a feature value expressed in vector representation is registered.

  Here, the image feature extraction / recognition processing unit 202 and the image feature / recognition result registration processing unit 203 are operated for registered images (registered images 1 to 3) shown in FIGS. 8A to 8C, respectively. In this case, each index registered in the DB 204 is shown in FIGS.

  The registered image 1 (registered image with image ID = 1) shown in FIG. 8A includes car objects 801 to 803 and a motorcycle object 804. Here, in this embodiment, it is assumed that various types of images are learned in image recognition, and images of these three cars are recognized, and a motorcycle is not recognized as an image recognition target. The registered images 2 and 3 (registered images with image ID = 2 and 3 respectively) shown in FIGS. 8B and 8C include a car object 805 and a human object 806, respectively.

  At this time, as shown in FIG. 9A, in the image management index registered in the DB 204, the image IDs (1 to 3) of the respective registered images are registered in the “image ID” field.

  In the “image data file name” field in the row (record) in which image ID = 1 is registered, the path name of the registered image with image ID = 1 is registered. The IDs (recognition IDs) of the objects 801 to 803 are registered in the “recognition result” field in the same row, and the registered image with the image ID = 1 in the “image feature amount” field in the same row. The ID of the feature amount extracted from is registered. In addition, although the image recognition is not performed for the object 804, the feature amount ID is registered in the “image feature amount” field. In FIG. 9, since 800 local feature amounts for all feature points extracted from the registered image 1 are detected, an image feature ID is assigned between 1 and 800.

  In the “image data file name” field in the row where the image ID = 2 is registered, the path name of the registered image with the image ID = 2 is registered. The ID (recognition ID) of the object 805 is registered in the “recognition result” field in the same row, and the “image feature amount” field in the same row is extracted from the registered image with the image ID = 2. The feature amount ID is registered. In FIG. 9, since 200 local feature amounts for all feature points extracted from the registered image 2 are detected, image feature IDs are assigned between 801 and 1000.

  In the “image data file name” field in the row where the image ID = 3 is registered, the path name of the registered image with the image ID = 3 is registered. The ID (recognition ID) of the object 806 is registered in the “recognition result” field in the same row, and the “image feature amount” field in the same row is extracted from the registered image with the image ID = 3. The feature amount ID is registered. In FIG. 9, since 200 local feature amounts for all feature points extracted from the registered image 3 are detected, an image feature ID is assigned between 1001 and 1200.

  Further, as shown in FIG. 9B, in the recognition result management index registered in the DB 204, the IDs of the objects 801 to 803 and 805 to 806, ie, 1 to 5, are stored in the “recognition ID” field. It is registered. As shown in FIG. 9B, the result recognized from the registered image 1 is registered in the record of recognition ID = 1 to 3, and the result recognized from the registered image 2 is registered in the record of recognition ID = 4. The result recognized from the registered image 3 is registered in the record with recognition ID = 5.

  In the “recognition attribute” field in the row where the recognition ID = 1 is registered, an attribute value indicating the attribute of the object with the recognition ID = 1, that is, the car object 801 is registered. The ID of the registered image 1 that is the recognition source of the object 801 is registered in the “image ID” field of the same row, and the registered image 1 to the object 801 are registered in the “recognition likelihood” field of the same row. The recognition likelihood when the image is recognized is registered. In the “recognition region” field in the same row, region information indicating a mask region in which the recognition likelihood obtained by the image recognition process for the object 801 is higher than a predetermined threshold is registered. In the “image feature value in the recognition area” field of the same line, the feature of each feature point in the mask area in which the recognition likelihood obtained by the image recognition process for the object 801 is higher than a predetermined threshold value. A number of image feature IDs are registered. Each record of recognition ID = 2-5 has the same configuration.

  In this time, when learning the car, the image of the pickup truck was used a lot for learning, so that the recognition likelihood for the object 802 of the pickup truck is the highest and is 90%. The recognition likelihood for the sports car object 801 is 85%, and the recognition likelihood for the bus object 803 is 80%. On the other hand, the recognition likelihood for the sports car object 805 is 87%, and the recognition likelihood for the human object 806 is 90%.

  Regarding the recognition area, the position in the corresponding image among the registered images 1 to 3 is stored for each of the objects 801 to 803 and the objects 805 to 806 corresponding to each recognition ID.

  Further, as shown in FIG. 9C, an image feature index is generated for each of image feature IDs = 1 to 1200. In the “image ID” field in the row in which the image feature ID = 1 is registered, the ID of the registered image from which the feature amount of the image feature ID = 1 is registered. The coordinate position on the registered image 1 of the feature point of image feature ID = 1 is registered in the “feature point coordinate” field in the same row. In the “local feature value” field in the same row, a feature value vector corresponding to the image feature ID = 1 is registered. Each record of image feature ID = 2 to 1200 has the same configuration.

  By the way, as described above, the “image feature value in the recognition area” field is used in the second embodiment, but the feature points included in the recognition area are stored as an image feature ID group.

  Next, image search processing for searching for an image similar to the query image shown in FIG. 8D from the DB 204 will be described with reference to FIG. 10 showing a flowchart of the processing. If this process is expanded, for each object in the input query image, an image including a similar object can be searched using the object as a query.

  In step S <b> 1001, the image input unit 302 acquires the query image of FIG. 8D loaded from the storage device 104 or input from an external device, and transfers the acquired query image to the feature point / local feature amount calculation unit 303. The image input unit 302 displays the acquired query image on the screen of the output device 103, and then uses the image in the object area designated by the user using the input device 102 as a feature image / local feature again as a query image. You may make it transfer to the quantity calculation part 303. FIG.

  In step S1002, the feature point / local feature amount calculation unit 303 extracts the feature point and the feature amount from the query image acquired in step S1001. In step S1003, the image recognition processing unit 304 performs image recognition processing on the query image using the feature amount extracted in step S1002.

  In step S1004, the image recognition processing unit 304 determines whether or not the image recognition processing for the object in the query image has succeeded (for example, whether or not there are one or more objects whose attributes can be specified). As a result of the determination, if there is an object for which image recognition has succeeded, the process proceeds to step S1005. If not, the process proceeds to step S1006.

  In step S1005, the image comparison processing unit 206 first identifies, from the recognition result management index in the DB 204, a record in which the same attribute value as the object attribute value obtained in the image recognition process in step S1003 is registered. Here, an image showing a car shown in FIG. 8D is used as a query image. For this reason, the image of this vehicle is recognized and, as a result, the attribute value of the object is “2”. In this case, from the recognition result management index, the record in which the attribute value “2” is registered, that is, the recognition ID = 1 to 4 records are specified. Here, since the recognition IDs = 1 to 3 are for the registered image 1, the largest recognition likelihood among the recognition likelihoods registered in the records with the recognition ID = 1 to 3 is set as the registered image 1. The recognition likelihood for. Therefore, here, the search result image is obtained assuming that the search result image having a similarity degree of 90% to the query image is the registered image 1 and the search result image having the similarity degree of 87% to the query image is the registered image 2.

  The image comparison processing unit 206 outputs the image ID and the similarity in the order of the recognition likelihood, that is, the registered image 1 and the registered image 2 in this order. In this case (image ID, similarity) = (1,90), (2,87) are output in this order. If the recognition likelihood is not represented by 0 to 100%, the similarity is converted to be 0 to 100%. Thereby, it is possible to search at high speed without image feature comparison processing.

  On the other hand, when the image of FIG. 8E is used as the query image, in step S1004, the image recognition processing unit 304 determines that the image recognition processing for the object in the query image is not successful, and the processing is step S1006. Proceed to

  In step S1006, the image comparison processing unit 206 sequentially refers to the record of each image ID registered in the image management index in FIG. The image comparison processing unit 206 uses the vector registered in the “local feature amount” field in the image feature index of each image feature ID in the referenced record and the feature amount obtained from the query image. To perform matching using the above RANSAC. Then, the image IDs and their similarities are output as search results in the order similar to the feature values obtained from the query images.

  By such processing, it was possible to combine the property of not selecting an object for image search using feature amounts and the property of selecting a target object for image recognition but relatively few erroneous detections. As an effect, when the image recognizable object is a query image, a good search result with extremely high speed and low noise can be obtained.

[Second Embodiment]
In the first embodiment, when the query image can be recognized, the similar image search can be performed at high speed by performing the above-described processing. However, according to the first embodiment, the object having the highest similarity is the object 802 in the registered image 1 having the highest image recognition likelihood. This is a result different from the expectation of many users who think that the appearance of the query image is similar to the object 805 in the registered image 2. In the present embodiment, such a problem is solved.

  In the present embodiment, the image registration process is the same as that of the first embodiment, but the following points are different from the first embodiment in the image search process. Only the differences from the first embodiment will be described below, and unless otherwise noted, the same as the first embodiment. The image search process according to this embodiment will be described with reference to FIG. 11 showing a flowchart of the process.

  In step S1101, the image input unit 302 acquires the query image of FIG. 8D loaded from the storage device 104 or input from an external device, and transfers the query image to the feature point / local feature amount calculation unit 303. The image input unit 302 displays the acquired query image on the screen of the output device 103, and then uses the image in the object area designated by the user using the input device 102 as a feature image / local feature again as a query image. You may make it transfer to the quantity calculation part 303. FIG.

  In step S1102, the feature point / local feature amount calculation unit 303 extracts the feature point and the feature amount from the query image acquired in step S1101. In step S1103, the image recognition processing unit 304 performs image recognition processing on the query image using the feature amount extracted in step S1102.

  In step S1104, the image recognition processing unit 304 determines whether or not the image recognition processing for the object in the query image has succeeded (for example, whether or not there is an object whose attribute can be specified). As a result of the determination, if there is an object for which image recognition has succeeded, the process proceeds to step S1105, and if there is no object, the process proceeds to step S1107.

  In step S1105, as in step S1005, the image comparison processing unit 206 records a record in which the same attribute value as the object attribute value obtained in the image recognition process in step S1103 is registered from the recognition result management index in the DB 204. Identify. Here, an image showing a car shown in FIG. 8D is used as a query image. For this reason, the image of this vehicle is recognized and, as a result, the attribute value of the object is “2”. In this case, from the recognition result management index, the record in which the attribute value “2” is registered, that is, the recognition ID = 1 to 4 records are specified. In addition, it is good also considering only the record whose recognition likelihood is more than a threshold value among the records of recognition ID = 1-4 as a subsequent object.

  In step S1106, the image comparison processing unit 206 performs the following processing for each record specified in step S1105. That is, the feature amount in the image feature index corresponding to the image feature ID in the record, and the feature amount in the region in the query image corresponding to the region defined in the “recognition region” field in the record, The degree of similarity is obtained by performing matching. As a result, it is possible to narrow down the image and its region to be compared with the local feature amount, and the cost of the comparison process can be greatly reduced. As for matching, the above RANSAC method can be used.

  Or, for matching, multi-dimensional, or clustering 128-dimensional local features in SIFT, create a histogram of the number of feature points belonging to each class, and use normalized histogram correlation or intersection. The similarity can be calculated.

  Then, as in the first embodiment, the image comparison processing unit 206 outputs the image IDs in the records and the similarities as search results in the order of the records for which higher similarities are obtained.

  In this way, by comparing the local features in the recognition area of the registered image and the query image, the efficiency used to narrow down the recognition results while reflecting the similarity with the query that could not be solved only by the recognition likelihood. It is possible to perform a good search process.

  On the other hand, when the image of FIG. 8E is used as the query image, in step S1004, the image recognition processing unit 304 determines that the image recognition processing for the object in the query image is not successful, and the processing is step S1107. Proceed to

  In step S1107, the image comparison processing unit 206 sequentially refers to the record of each image ID registered in the image management index in FIG. The image comparison processing unit 206 uses the vector registered in the “local feature amount” field in the image feature index of each image feature ID in the referenced record and the feature amount obtained from the query image. To perform matching using the above RANSAC. Then, the image IDs and their similarities are output as search results in the order similar to the feature values obtained from the query images.

[Third Embodiment]
In the present embodiment, in the first embodiment and the second embodiment, the problem that image recognition does not function well in one of the query image and the registered image, resulting in a search omission as a result is solved.

  In the present embodiment, the image registration process is the same as that of the first embodiment, but the following points are different from the first embodiment in the image search process. Only the differences from the first embodiment will be described below, and unless otherwise noted, the same as the first embodiment.

  The image search process according to the present embodiment will be described with reference to FIG. 12 showing a flowchart of the process. Note that steps S1201 to S1204 and S1208 are the same processing steps as steps S1001 to S1004 and S1006 in FIG. 10, respectively, and steps S1205 to S1207 are performed instead of step S1005, which is different from the first embodiment.

  In step S1205, the image comparison processing unit 206 first identifies, from the recognition result management index in the DB 204, a record in which the same attribute value as the object attribute value obtained in the image recognition process in step S1203 is registered. Then, the image comparison processing unit 206 reads the recognition likelihood from the identified record.

  In step S1206, the image comparison processing unit 206 performs the following process for each record specified in step S1205. That is, the image feature ID in the “image feature amount” field is read from the image management index corresponding to the image ID in the record, the feature amount corresponding to the image feature ID is read from the image feature index, and Matches with the feature value. For this matching, the above RANSAC is used, and the similarity between the feature quantities is obtained.

  In step S1207, the image comparison processing unit 206 performs the following process for each record specified in step S1205. That is, the total similarity is obtained by performing weighted addition on the recognition likelihood LH read in step S1205 for the record and the similarity Sim obtained in step S1206 for the record using the weights w1 and w2. The formula of this weighted addition is shown below.

Total similarity Sim total = w1 × LH + w2 × Sim, where w1 + w2 = 1
Then, by applying the total similarity instead of the similarity and performing the processing in step S1005, even if the image has failed in image recognition, if the image has a high score in the local feature amount comparison, It can be reflected.

  Further, w1 may be changed for each recognition target. For example, when a recognition target with a low recognition rate is detected by a query, a low value w1 is used, and when a recognition target with a high recognition rate is set, w1 has a high value. It is good to use. For an image having an object capable of image recognition, a comprehensive evaluation is obtained by a weighted sum of the image recognition likelihood and the similarity so that the recognition likelihood is moderately effective.

  As described above, according to the present embodiment, a good match between the property of not selecting an object for image search using local features and the property of selecting a target image object for image recognition but relatively few erroneous detections. Image search that yields accurate search results.

  In addition, although the processing cost is higher than that of the first embodiment, the above processing enables a robust search with less search omission and gently reflecting the image recognition result.

[Fourth Embodiment]
Since the number of image recognition targets increases with time, it is preferable to apply a new image recognition result to a registered image. Of course, there is a method of registering an image from the beginning using a detector learned with a new image recognition target, but this is not practical when the number of registered images is enormous. In addition, when the DB and the image are managed separately, it may be difficult to refer to the registered image.

  Therefore, in the present embodiment, a method for efficiently reflecting an increase in the number of image recognition objects in a closed process in the DB without reanalyzing the registered image for updating the DB index reflecting the increase in the number of image recognition objects. Will be described. A process performed for updating the DB index for each registered image will be described with reference to FIG. 13 showing a flowchart of the process.

  In step S <b> 1301, the image feature / recognition result registration processing unit 203 focuses on one of the registered images registered, and uses the image feature ID in the image management index of the registered image of interest to correspond to the image feature index. Then, the coordinate position of the feature point and the feature amount thereof are read out.

  In step S1302, the image feature / recognition result registration processing unit 203 performs image recognition processing on the focused registered image using the information read in step S1301. At this time, the detector used for image recognition can recognize a new object than before.

  In step S1303, the image feature / recognition result registration processing unit 203 determines whether a new object has been recognized from a position different from any of the objects recognized so far from the focused registered image. If the result of this determination is that a new object has not been recognized, this process is terminated. If a new object has been recognized, the process proceeds to step S1304.

  In step S1304, the image feature / recognition result registration processing unit 203 additionally registers a recognition ID obtained by adding 1 to the current maximum recognition ID in the “recognition result” field in the image management index of the registered image of interest. .

  The image feature / recognition result registration processing unit 203 adds one record to the recognition result management index, and registers the information listed below in the added record. That is, the recognition ID additionally registered in step S1304, the new object attribute value, the image ID of the focused registered image, the new object recognition likelihood, the coordinates of the new object recognition area, and the new object recognition The image feature ID of the feature amount is registered. As described above, according to the present embodiment, it is possible to update the DB to reflect the newly added image recognition target at a low processing cost by the process closed in the DB.

[Fifth Embodiment]
In each of the above-described embodiments, SIFT is used as the feature amount. However, the type is not limited as long as it is a local feature amount that can be shared between image recognition and image feature amount search. Further, in the image recognition, the machine learning using MAP estimation has been described, but naturally, as a method based on the generation model, iterative learning using an EM algorithm or a method based on a discrimination model may use SVM. . Also, it is possible to perform a plurality of image recognition processes and update the local feature amount of the registered image.

(Other examples)
The present invention can also be realized by executing the following processing. That is, software (program) that realizes the functions of the above-described embodiments is supplied to a system or apparatus via a network or various storage media, and a computer (or CPU, MPU, or the like) of the system or apparatus reads the program. It is a process to be executed.

Claims (11)

For each image, an image processing apparatus capable of accessing a database in which each attribute and recognition likelihood of one or more objects in the image obtained by image recognition processing for the image is registered in association with the image Because
An identification means for acquiring an attribute of an object in the query image by performing an image recognition process on the query image, and specifying an image registered in the database in association with the acquired attribute;
The recognition likelihood registered in the database in association with the image specified by the specifying unit is set as the similarity of the specified image with respect to the query image, and the information about the specified image and the similarity are defined as an image. An image processing apparatus comprising: output means for outputting as a search result.   The image processing apparatus according to claim 1, wherein the output unit outputs a set of information related to the specified image and the similarity in descending order of similarity. The output means includes
As a result of the image recognition process on the query image, if there is no object that can be image recognized, the query obtained by the image recognition process on the query image among the images registered in the database. The image processing apparatus according to claim 1, wherein information relating to an image having a feature amount similar to the feature amount of the image is output. For each image, it is possible to access a database in which each attribute, feature amount, and recognition likelihood of one or more objects in the image obtained by image recognition processing for the image is registered in association with the image. Image processing apparatus,
Registered in the database using a detector generated for recognizing a new object whose attribute, feature amount, and recognition likelihood are not registered in the database, and a feature amount registered in the database Recognizing means for recognizing the new object from the image and registering the attribute, feature amount, and recognition likelihood of the recognized new object in the database in association with the registered image;
An image recognition process including the new object recognition process is performed on the query image to acquire the attribute of the object in the query image, and the image registered in the database is identified in association with the acquired attribute Specific means to
The recognition likelihood registered in the database in association with the image specified by the specifying unit is set as the similarity of the specified image with respect to the query image, and the information about the specified image and the similarity are defined as an image. Output means for outputting as search results
An image processing apparatus comprising: An image processing apparatus capable of accessing a database in which each attribute and area of one or more objects in the image obtained by image recognition processing for the image is registered in association with the image. And
An identification unit that acquires an attribute of an object in the query image by performing an image recognition process on the query image, and identifies an area registered in the database in association with the acquired attribute;
Means for obtaining a similarity between a feature amount obtained by image recognition processing for the region specified by the specifying means and a feature amount obtained by image recognition processing for a region corresponding to the region in the query image;
An image processing apparatus comprising: output means for outputting information relating to an image registered in the database in association with the area specified by the specifying means, and the similarity as an image search result. For each image, an image processing apparatus capable of accessing a database in which each attribute and recognition likelihood of one or more objects in the image obtained by image recognition processing for the image is registered in association with the image Because
An identification means for acquiring an attribute of an object in the query image by performing an image recognition process on the query image, and specifying an image registered in the database in association with the acquired attribute;
Means for obtaining a similarity between a feature quantity obtained by image recognition processing for the query image and a feature quantity obtained by image recognition processing for the image specified by the specifying means;
The overall similarity obtained by weighted addition of the recognition likelihood registered in the database in association with the image specified by the specifying means and the similarity is defined as the similarity of the specified image with respect to the query image. An image processing apparatus comprising: output means for outputting information relating to the specified image and the similarity as an image search result. For each image, an image processing apparatus capable of accessing a database in which each attribute and recognition likelihood of one or more objects in the image obtained by image recognition processing for the image is registered in association with the image Is an image processing method performed by
The specifying unit of the image processing apparatus acquires an attribute of an object in the query image by performing an image recognition process on the query image, and specifies an image registered in the database in association with the acquired attribute Specific process to
The output means of the image processing apparatus sets the recognition likelihood registered in the database in association with the image specified in the specifying step as the similarity of the specified image to the query image, and information on the specified image And an output step of outputting the similarity as an image search result. For each image, it is possible to access a database in which each attribute, feature amount, and recognition likelihood of one or more objects in the image obtained by image recognition processing for the image is registered in association with the image. An image processing method performed by a simple image processing apparatus,
A recognition unit of the image processing apparatus, a detector generated for recognizing a new object whose attribute, feature amount, and recognition likelihood are not registered in the database; a feature amount registered in the database; Recognizing the new object from the image registered in the database, and registering the attribute, feature amount, and recognition likelihood of the recognized new object in the database in association with the registered image Process,
The specifying unit of the image processing apparatus acquires an attribute of the object in the query image by performing an image recognition process including a recognition process of the new object on the query image, and associates the acquired attribute with the acquired attribute. A specific process for identifying images registered in the database;
The output means of the image processing apparatus sets the recognition likelihood registered in the database in association with the image specified in the specifying step as the similarity of the specified image to the query image, and information on the specified image And outputting the similarity as an image search result,
An image processing method comprising: For each image, an image processing apparatus that can access a database in which attributes and areas of one or more objects in the image obtained by image recognition processing for the image are associated with the image and registered is performed. An image processing method comprising:
The specifying unit of the image processing apparatus acquires an attribute of an object in the query image by performing an image recognition process on the query image, and specifies an area registered in the database in association with the acquired attribute Specific process to
The means for obtaining the similarity of the image processing device is a feature amount obtained by image recognition processing for the region specified in the specifying step, and a feature obtained by image recognition processing for a region corresponding to the region in the query image. A step of determining the similarity between the quantity,
The output unit of the image processing apparatus includes an output step of outputting information related to an image registered in the database in association with the region specified in the specifying step and the similarity as an image search result. A featured image processing method. For each image, an image processing apparatus capable of accessing a database in which each attribute and recognition likelihood of one or more objects in the image obtained by image recognition processing for the image is registered in association with the image Is an image processing method performed by
The specifying unit of the image processing apparatus acquires an attribute of an object in the query image by performing an image recognition process on the query image, and specifies an image registered in the database in association with the acquired attribute Specific process to
A step of obtaining a similarity between a feature quantity obtained by an image recognition process for the query image and a feature quantity obtained by an image recognition process for the image specified in the specifying step; When,
The output means of the image processing apparatus obtains the total similarity obtained by weighted addition of the recognition likelihood registered in the database in association with the image specified in the specifying step and the similarity, and the query image An image processing method comprising: an output step of outputting information relating to the identified image and the similarity as an image search result, the similarity of the identified image with respect to A computer program for causing a computer to function as each unit of the image processing apparatus according to any one of claims 1 to 6 .