JP2006221525A - Object retrieval system and method - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide an object retrieval system and method suitable for retrieving the identical and homogenous object whose feature values change. <P>SOLUTION: This object retrieval system 10 is provided with a first object candidate region extracting part 16 for gradually narrowing down a first object candidate region by using the feature values of each cluster related with colors stored in a feature value storage device 23 for an input image including an object, a second object candidate region extracting part 17 for gradually narrowing down a second object candidate region by using the feature values of each cluster related with shapes stored in the feature value storage device 23 for the first object candidate region and an object region deciding part 18 for deciding an object region from the second object candidate region, and for searching the object included in the input image. <P>COPYRIGHT: (C)2006,JPO&NCIPI

Description

  The present invention relates to an object search system and method, and more particularly to an object search system and method suitable for searching for a blackboard included in a construction photo.

In public works, deliverables must be delivered at the end, and in the case of construction, it is necessary to deliver construction photographs. The blackboard is shown in the construction photos. This blackboard contains important information such as the construction name and construction details, and the blackboard information is extremely important in construction photographs.
For this reason, in construction photo books to be delivered, it is often the case that only the blackboard is purposely cut out and displayed in an enlarged manner. However, there are usually hundreds to thousands of construction photographs, and it takes a lot of labor to manually extract the blackboard.
Therefore, there is a demand for a system that images a construction photo and automatically extracts a blackboard image from the imaged construction photo.

However, like blackboards at construction sites, objects with different shooting environments such as shooting angles and lighting, and similar objects with different features such as color and shape (in other examples, red cars and blue cars) Even if it is the same object for the user, the feature amount changes.
Therefore, by giving a large number of example images containing the same and similar objects to be searched as input, and clustering the extracted feature quantities related to the color and the feature quantities related to the shape, the colors and shapes that can be taken by the object are clustered. It is conceivable to acquire a feature amount and use the feature amount to search for the same and similar objects whose feature amount changes.

Note that methods for searching for an object using a large number of example images include those disclosed in Non-Patent Document 1 and Non-Patent Document 2 below.
In Non-Patent Document 1, a query based on a generalized elliptical distance is realized by estimating a distance function intended by a user and a query position from a feature amount extracted from an example image group.
In Non-Patent Document 2, clustering of images is performed from the feature amounts extracted from the exemplary image group, weighting is performed for each feature amount by the standard deviation method for each image cluster, and similarity is calculated by weighted Euclidean distance at the time of retrieval. Is going.
Yoshiharu Ishikawa, Ravishankar Subramanya, Christos Faloutsos, “MindReader: Querying database through multiple examples”, Proceeding of the 24th VLDB Conference New York, USA, 1998 Roberto Brunelli, Ornella Mich, “Image Retrieval by Examples”, IEEE Trans. On Multimedia, Vol.2, No.3, pp.164-171, Sep. 2000

  However, the methods disclosed in Non-Patent Document 1 and Non-Patent Document 2 estimate the feature quantity or the correlation between feature quantities that the user is paying attention to from the viewpoint of similarity among multi-dimensional feature quantities. However, inquiries are made according to the user's intention, and the purpose of using a plurality of example images in these methods is to automatically estimate the feature quantity that the user is paying attention to. Therefore, it cannot be said that these methods are necessarily suitable for searching for the same and the same kind of objects whose feature values change.

  An object of the present invention is to provide an object search system and method suitable for searching for the same and the same kind of objects whose feature values vary.

An object search system according to the present invention obtains an object area extraction unit (11) for extracting an object area included in an example image, a representative color constituting the object area extracted by the object area extraction unit, A color feature quantity extraction unit (12) for extracting the color feature quantity of the obtained representative color, and after clustering the color feature quantity extracted by the color feature quantity extraction section, the feature quantity possessed by each cluster relating to color A color feature amount clustering unit (13) that extracts a shape feature amount, and a shape feature amount extraction unit (14) that extracts a rough shape feature amount and a detailed shape feature amount of the object region extracted by the object region extraction unit, After performing clustering of the detailed shape feature values extracted by the shape feature value extraction unit, detailed information on each cluster is obtained. A shape feature quantity clustering unit (15) for extracting a shape feature quantity and a feature quantity of each cluster relating to the color extracted by the color feature quantity clustering unit are stored and extracted by the shape feature quantity extraction unit. A feature quantity storage device (23) for storing the feature quantity of each cluster related to the shape including the rough shape feature quantity and the detailed shape feature quantity of each cluster extracted by the shape feature quantity clustering unit; A first object that narrows down first object candidate regions in a stepwise manner using feature amounts of each cluster relating to the color stored in the feature amount storage device for an input image including objects A candidate area extraction unit (16) and a first object candidate area extracted by the first object candidate area extraction unit On the other hand, a second object candidate region extraction unit (17) that narrows down the second object candidate regions step by step using the feature amount of each cluster related to the shape stored in the feature amount storage device. ) And an object region determination unit (18) for determining the object region from the second object candidate region extracted by the second object candidate region extraction unit and obtaining the object included in the input image It is characterized by comprising.
Here, the first object candidate region extraction unit, for the input image, based on the feature amount of each cluster related to the color, the hue, saturation, and color of each component color included in each cluster related to the color By extracting pixels that fall within the respective luminance ranges and labeling connected regions, a first narrowing down is performed based on color feature amounts, and the area of the entire input image is determined for each of the labeled regions. By removing the area where the ratio is equal to or less than the threshold value, the second narrowing down by the color feature amount is performed, and when a plurality of constituent colors of the cluster exist, by extracting only the area including all the constituent colors, After performing the third narrowing by the color feature amount and extracting the spatial relationship and area feature amount from each region extracted by the third narrowing by the color feature amount, the extraction is performed. By calculating the similarity between the spatial relationship and area feature amount and the spatial relationship and area feature amount of each cluster related to the color, and extracting only regions where the two similarities are greater than or equal to a predetermined value, You may perform the 4th narrowing-down by a color feature-value.
The second object candidate region extraction unit extracts a rough shape feature amount from the first object candidate region extracted by the first object candidate region extraction unit, and then each cluster related to the shape has a feature Based on the amount, the similarity between the rough shape feature amount of each cluster related to the shape and the extracted rough shape feature amount is calculated, and only the region where the calculated similarity is within a predetermined range is calculated. By extracting, the first narrowing by the shape feature amount is performed, and after extracting the detailed shape feature amount from the region extracted by the first narrowing by the shape feature amount, the feature amount of each cluster related to the shape is obtained. Based on this, the similarity between the detailed shape feature amount of each cluster related to the shape and the extracted detailed shape feature amount is calculated, and the calculated similarity is calculated. Of by extracting only the above and a region value may be performed to narrow down due to the shape characteristic of the second.

The object search method of the present invention includes a first step (S11) of inputting example image data in which an object area included in an example image is surrounded by a black line to the object search and extraction system (10), and the object search and extraction system. In the second step (2), the object region is extracted from the example image, and based on the extracted image of the object region, a feature amount of each cluster related to color and a feature amount of each cluster related to shape are extracted ( S12 to S17), a third step of inputting an input image including an object to the object search system (S21), and a first object candidate region extraction unit (16) Narrowing down the first object candidate area step by step using the features of each cluster The fourth step (S22 to S25) for performing the above, and the second object candidate area extraction unit (17) for the first object candidate area input from the first object candidate area extraction unit, A fifth step (S27, S28) for narrowing down the second object candidate area step by step using the feature quantity of each cluster related to the shape, and an object area determination unit (18), And a sixth step (S30) of determining the object area from the second object candidate area input from the object candidate area extraction unit and obtaining the object included in the input image. Features.
Here, in the second step, the object region extracting unit (11) extracts the object region by scanning the pixels in the horizontal direction with respect to the exemplary image data to detect black pixels (S12). And a color feature amount extraction unit (12) obtains representative colors constituting the object region extracted by the object region extraction unit, and ranges of hue, saturation, and luminance values of the obtained representative colors And a shape feature quantity extraction unit (14) calculates the ratio of the vertical width to the horizontal width of the minimum rectangular area including the object area extracted by the object area extraction section. Extracting a rough shape feature amount (S14), and the shape feature amount extraction unit applies all the pixels that form the contour of the object region. Calculating a tangent angle, generating an angle histogram for the frequency of the tangent angle for each predetermined angle, extracting the generated angle histogram as the detailed shape feature amount (S15), and color feature amount clustering A step (S16) of performing a clustering on the feature value in the range of the hue value extracted by the color feature value extraction unit, and then extracting a feature value of each cluster relating to the color (S16); A step (S17) of extracting a detailed shape feature amount of each cluster after the shape feature amount clustering unit (15) performs clustering of the detailed shape feature amount extracted by the shape feature amount extraction unit; You may prepare.

  The object search system and method of the present invention can accurately extract, for example, a blackboard image included in an imaged construction photograph.

  For the purpose of searching for the same and the same kind of objects whose feature quantity changes, the feature quantity of each cluster related to color and the feature quantity of each cluster related to shape are extracted using a number of example images, and the input image On the other hand, the first object candidate area is narrowed down step by step using the feature quantity of each cluster relating to the color, and the feature quantity of each cluster relating to the shape is determined for this first object candidate area. The second object candidate area is narrowed down step by step, and then the object area is determined from the second object candidate area and the object included in the input image is obtained.

Embodiments of an object search system and method according to the present invention will be described below with reference to the drawings. In the following, a case where a blackboard image included in a construction photograph image (hereinafter referred to as “database image”) that is an input image is extracted as an object will be described as an example.
FIG. 1 is a schematic block diagram for explaining an object search system according to an embodiment of the present invention. The object search system 10 according to the present embodiment includes an object region extraction unit 11, a color feature quantity extraction unit 12, a color feature quantity clustering unit 13, a shape feature quantity extraction unit 14, a shape feature quantity clustering unit 15, 1 object candidate region extraction unit 16, second object candidate region extraction unit 17, and object region determination unit 18.

  Here, the object region extraction unit 11 and the first object candidate region extraction unit 16 are connected to an external image input device 20. The color feature quantity clustering unit 13, the shape feature quantity clustering unit 15, the first object candidate area extracting unit 16, and the second object candidate area extracting unit 17 are connected to an external feature quantity storage device 23. The object area determination unit 18 is connected to an external object storage device 24.

  The image input device 20 reads out the example image data one by one from the example image storage device 21 in which a large number (a plurality of) of example image data is stored and inputs it to the object search system 10, as well as database images (construction photographs including blackboard The database image data is read from the image storage device 22 in which the database image data representing the construction photograph image obtained by imaging the image is stored and input to the object search system 10.

  FIG. 2 is a diagram illustrating an example of a construction photographic image and an exemplary image extracted from the construction photographic image. The construction photo image shown on the left side of FIG. 2 is an image of the construction photo including the blackboard, and the example image shown on the right side of FIG. 2 is an area extracted from the construction photo image including the blackboard. is there. The object search system 10 extracts a blackboard area included in each of a plurality of exemplary images stored in the exemplary image storage device 21, and individually extracts color feature values and shape feature values of the extracted blackboard areas. After that, the extracted color feature values and shape feature values are clustered, and feature values that represent the data in each cluster (hereinafter referred to as “features that each cluster related to color” and “features that each cluster related to shape have” Called "quantity"). Then, the feature quantity data representing the feature quantity of each cluster relating to the obtained color and the feature quantity possessed by each cluster relating to the shape are stored in the feature quantity storage device 23.

  In order to extract the blackboard area from the exemplary image including the blackboard, the exemplary image data is stored in the exemplary image storage device 21 after a process of surrounding the blackboard area with a black line. As a result, the object region extraction unit 11 extracts a region surrounded by black pixels (R = 0, G = 0, B = 0) from the exemplary image data representing the exemplary image input from the image input device 20. Thus, the object area, that is, the blackboard area can be extracted.

The color feature quantity extraction unit 12 obtains a representative color constituting the blackboard area extracted by the object area extraction unit 11, and extracts a color feature quantity of the obtained representative color.
The color feature amount clustering unit 13 performs clustering of the color feature amounts extracted by the color feature amount extraction unit 12 for a plurality of exemplary images, and then extracts the feature amounts of each cluster relating to colors.

The shape feature amount extraction unit 14 extracts a rough shape feature amount and a detailed shape feature amount of the blackboard region extracted by the object region extraction unit 11.
The shape feature amount clustering unit 15 performs clustering of the detailed shape feature amounts extracted by the shape feature amount extraction unit 14 for a plurality of exemplary images, and then extracts detailed shape feature amounts of each cluster.
Note that the feature amount of each cluster related to the shape includes the rough shape feature amount extracted by the shape feature amount extraction unit 14 and the detailed shape feature amount of each cluster extracted by the shape feature amount clustering unit 15. .

The first object candidate region extraction unit 16 performs stepwise processing on the database image (construction photo image) represented by the database image data input from the image input device 20 by using the feature amount of each cluster relating to color. The object candidate area is narrowed down.
The second object candidate area extraction unit 17 applies object candidates step by step to the first object candidate area extracted by the first object candidate area extraction unit 16 using the feature amount of each cluster related to the shape. Narrow down the area.
The object region determination unit 18 determines an object region from the second object candidate region extracted by the second object candidate region extraction unit 17, and represents the blackboard image with the determined object region as a blackboard image (object). Object area data (blackboard image data) is stored in the object storage device 24.

Next, the operation of the object search system 10 (object search method according to an embodiment of the present invention) will be described.
First, the operation of the object search system 10 when extracting the feature quantity of each cluster relating to color and the feature quantity of each cluster relating to the shape from a plurality of example images will be described with reference to the flowchart shown in FIG.

  The person in charge reads a plurality of exemplary image data stored in the exemplary image storage device 21 one by one using the image input device 20, and then inputs the read exemplary image data to the object search system 10 (step). S11).

In the exemplary image represented by the exemplary image data input from the image input device 20, the blackboard region is processed to be surrounded by a black line, and therefore the object region extraction unit 11 of the object search system 10 performs FIG. As shown in FIG. 4, the pixels are scanned in the horizontal direction for this exemplary image to detect black pixels. When the object area extraction unit 11 detects a black pixel, the pixel area until the next black pixel is detected on the horizontal line of the detected black pixel is set as a pixel in the blackboard area so that it is surrounded by a black line. The blackboard area is extracted (step S12).
The object region extraction unit 11 outputs the blackboard region data representing the extracted blackboard region to the color feature amount extraction unit 12 and the shape feature amount extraction unit 14.

When the blackboard region data is input from the object region extraction unit 11, the color feature amount extraction unit 12 obtains a representative color constituting the blackboard region as described below, and the hue (H) of the obtained representative color. , Saturation (S) and luminance (V) value ranges are extracted as color feature values (step S13).
The color feature quantity extraction unit 12 smoothes the blackboard area data input from the object area extraction unit 11 with, for example, a median filter, and the hue (H), saturation (S), and luminance (V) for all pixels in the blackboard area. An HSV histogram is generated in which the values for each dimension are equally divided into “72”, “32”, and “32”.
Subsequently, the color feature quantity extraction unit 12 calculates all the sums of frequencies of consecutive 5 × 5 × 5 elements (total 72 × 28 × 28), and 5 × 5 × 5 elements that maximize the sum of frequencies. After detecting the range, the element having the highest frequency in the detected range is detected, and the coordinates (s) of each dimension of the hue (H), saturation (S), and luminance (V) of the detected element are detected. , T, u).

Subsequently, the color feature quantity extraction unit 12 generates a 72-segmented hue (H) histogram, a 32-segmented saturation (S) histogram, and a 32-segmented luminance (V) histogram, and then, for each histogram. Based on the coordinate values s, t, u of the obtained coordinates (s, t, u), the adjacent classes of the histogram are searched in order until the condition shown in the expression (1) is satisfied, and the searched range Are the hue, saturation, and luminance ranges of the representative color. Equation (1) indicates that when the frequency f (i) in a certain class i of the histogram becomes smaller than the threshold Th _color , the class is a division point of the histogram. However, more appropriate evaluation is performed by calculating not only the frequency of the class but also the sum of the frequencies of the two classes on the left and right.
f (i-2) + f (i-1) + f (i) + f (i + 1) + f (i + 2) <Th _color (1)
Here, Th _color = (total number of pixels in the object area) / 40

Subsequently, the color feature amount extraction unit 12 detects the range of values on the quantized histogram, not the actual range of hue (H), saturation (S), and luminance (V) values. Therefore, the range of the detected value is converted into the range of the actual hue (H), saturation (S), and luminance (V) values by the equation (2).
minH = minH _hist × 5, MaxH = MaxH _hist × 5 + 4
minS = minS _hist × 8, MaxS = MaxS _hist × 8 + 7 (2)
minV = minV _hist × 8, MaxV = MaxV _hist × 8 + 7
Here, minH _hist to MaxH _hist are the range of hue values (one end to one end) on the hue histogram, and minS _hist to MaxS _hist are the range of saturation values on the saturation histogram (lower limit to upper limit). , MinV _hist to MaxV _hist are ranges of luminance values (from minimum to maximum) on the luminance histogram, minH to MaxH are ranges of actual hue values (from one end to one end), and minS to MaxS are actual colors. It is a range of degree values (lower limit to upper limit), and minV to MaxV is a range of luminance values (minimum to maximum).

  Further, the color feature amount extraction unit 12 determines that the representative color is an achromatic color when 0 ≦ t ≦ 4. When expressing an achromatic color, the value of the hue becomes irrelevant, and thus the hue limitation is eliminated by setting the hue range to 0 ° to 359 °.

  The color feature amount extraction unit 12 repeats this series of processing for unsearched elements until a certain number of representative colors are extracted. FIG. 5 shows an example of the result of extracting representative colors up to two colors. In this example, the hue value range is 0 to 359, the saturation value range is 0 to 39, and the luminance value range is 240 to 255. A second representative color having a value range of 119 to 213, a saturation value range of 0 to 119, and a luminance value range of 240 to 255 is extracted.

  When the number of pixels in the area corresponding to the representative color extracted as described above is 0.05 or less with respect to the total number of pixels in the blackboard area, the color feature amount extraction unit 12 It is determined that it is not the representative color that characterizes the region, and is removed.

  In addition, when there are a plurality of extracted representative colors, the color feature amount extraction unit 12 obtains the ratio of each representative color to the blackboard area to obtain the blackboard area having the number of pixels corresponding to each representative color. Is obtained with respect to the total number of pixels, and the feature amount of the area of each representative color is extracted. In the example shown in FIG. 5, the area of the first representative color = 0.09 and the area of the second representative color = 0.91. Note that the number of dimensions of the feature amount of the area is equal to the number of dimensions of the representative color.

Further, when there are a plurality of extracted representative colors, the color feature quantity extraction unit 12 obtains information about the positional relationship between the representative colors, and the spatial relationship between the representative colors. Extract feature values.
Specifically, the color feature quantity extraction unit 12 first calculates the center of gravity of the representative color area having the largest area and the minimum rectangular area including the blackboard area. Then, the color feature quantity extraction unit 12 has a rectangular area that is ½ in the vertical and horizontal directions of the minimum rectangular area centered on the center of gravity, and the number of pixels of other representative color areas that exist in the vertical and horizontal directions excluding the rectangular area. And calculate the ratio of the number of pixels in the other representative color areas that exist in the vertical and horizontal directions to the total number of pixels in the other representative color areas in the vertical and horizontal directions, and the spatial relationship between the representative colors Feature amount. FIG. 6 shows an example of the feature amount of the spatial relationship between the obtained representative colors. In this example, the spatial relationship between the first and second representative colors is such that the ratios in the middle, upper, left, lower, and right directions are 0.30, 0.66, 0.01, 0.01, and 0, respectively. .02. The number of dimensions of this feature amount is {(number of constituent colors−1) × 5}.
With this feature amount, it is possible to express the spatial relationship of each of the other representative color regions based on the representative color region having the maximum area.

When the blackboard region data is input from the object region extraction unit 11, the shape feature amount extraction unit 14 extracts the shape feature amount of the blackboard region as follows (step S14).
First, the shape feature amount extraction unit 14 sets the aspect ratio of the blackboard region as a rough shape of the blackboard region in order to shorten the processing time required for narrowing down by the shape feature amount when actually extracting the blackboard image from the construction photo image. Calculate Then, the shape feature quantity extraction unit 14 calculates the ratio of the vertical width to the horizontal width of the minimum rectangular area including the blackboard area, and extracts the calculated ratio as a rough shape feature quantity. FIG. 7 shows an example of the extracted rough shape feature amount. In this example, the rough feature amount = 1.2. The number of dimensions of the rough shape feature amount is one dimension.

Subsequently, the shape feature quantity extraction unit 14 extracts a detailed shape feature quantity of the blackboard region as follows (step S15).
As shown in FIG. 8, the shape feature amount extraction unit 14 uses the intersection of the line drawn downward from the center of gravity of the blackboard area and the outline of the object area as a search start point, and the pixel that is the outline of the blackboard area is reflected. Search clockwise. The shape feature quantity extraction unit 14 calculates a tangent angle for all the pixels that form the outline of the blackboard area, and generates a 36-division angle histogram in which the frequency of the tangential angle is obtained every 10 ° as viewed from the center of gravity of the blackboard area. At this time, the tangent angle is obtained, for example, as an angle with respect to the horizontal direction of a straight line drawn four pixels ahead from a pixel on a certain contour. The shape feature quantity extraction unit 14 uses the generated angle histogram as a detailed shape feature quantity. Note that the number of dimensions of the feature amount of the detailed shape is 36 dimensions.

The color feature amount clustering unit 13 is the feature of the hue that most closely matches the subjectively observed color variation among the color feature amounts extracted by the color feature amount extraction unit 12 for a plurality of exemplary images. For the feature amount in the range of 2 × (representative color number) dimensional hue (H), color feature amount clustering is performed according to the following procedure (step S16).
Procedure 1: As shown in FIG. 9, data in a range of all hue (H) values belong to the same cluster.
Procedure 2: As shown in FIG. 10, the number of clusters after division is set to “2”, and clustering is performed by the k-means method with the barycentric coordinates of the clusters as the minimum and maximum values of each dimension of the data belonging to the clusters. .
Procedure 3: As shown in FIG. 11, the variance of each cluster after division is calculated, and a cluster whose value is equal to or greater than a threshold Th _var (= 40) is detected. Note that the threshold value Th _var is, for example, the maximum variance value among all the clusters when subjecting a large number of example images to subjectivity clustering based on colors.
Procedure 4: As shown in FIG. 12, the processing in Procedure 2 and Procedure 3 is performed again for the cluster detected in Procedure 3.
Step 5: As shown in FIG. 13, when the variance of all the clusters is less than the threshold Th _var , the clustering is terminated, and for each cluster, each color feature amount of the data to which it belongs (representative color, spatial relationship and area The average value of the feature amount is set as the color feature amount of the cluster.

However, when the representative colors differ depending on the exemplary images, clustering is performed for each set of exemplary images having the same representative color.
In the k-means method, the number of clusters must be determined first, but the appropriate number of clusters differs depending on the given image group. Therefore, by the above-described processing, it is possible to determine an appropriate number of clusters by further dividing a cluster having a large value in consideration of the dispersion value in the cluster.

The procedure of the k-average method is shown below.
Procedure 1: k points selected at random are set as the center of gravity of the initial cluster.
Procedure 2: The distance from the center of gravity of the cluster is calculated for all data, and it belongs to the nearest cluster.
Procedure 3: The average value of each coordinate of the data belonging to each cluster is set as the barycentric coordinate of the new cluster.
Procedure 4: Repeat the process from Procedure 1 to Procedure 3 until the barycentric coordinates of the cluster converge. For example, it is assumed that convergence has occurred when the result of clustering of each data no longer changes.

The shape feature amount clustering unit 15 performs clustering on the detailed shape feature amounts extracted by the shape feature amount extraction unit 14 for a plurality of exemplary images according to the above-described color feature amount clustering procedures 1 to 5. Thus, a detailed shape feature amount of each cluster is extracted (step S17). The threshold Th _var used at this time is 0, which is the maximum variance value among all the clusters when subjecting a plurality of example images to subjective clustering based on the shape, for example, as in the case of clustering of color feature values. .001.
Here, the reason why the threshold Th _var (= 0.001) used for detailed shape feature quantity clustering is different from the threshold Th _var (= 40) used for color feature quantity clustering is that the feature quantity used for clustering This is because the number of dimensions and the range of values to be taken are different.

  The color feature amount clustering unit 13 stores the feature amount of each cluster relating to the extracted color in the feature amount storage device 23, and the shape feature amount clustering unit 15 stores the feature amount of each cluster relating to the extracted shape. The data is stored in the device 23 (step S18). As a result, the feature quantity data representing the feature quantity of each cluster relating to the color, which is the feature quantity representing the data in each cluster in this example image, and the feature quantity of each cluster relating to the shape are stored in the feature quantity storage device 23. Is done.

Hereinafter, the operation of the object search system 10 when extracting the blackboard images included in the three database images shown at the bottom of FIG. 14 will be described with reference to the flowchart shown in FIG.
The person in charge uses the image input device 20 to read three database image data shown in FIG. 14 out of a large number of database image data (construction photo image data) stored in the image storage device 22 one by one. After that, the read database image data is input to the object search system 10 (step S21).

  The first object candidate area extraction unit 16 of the object search system 10 stores a database image (construction photograph image) represented by database image data from the image input device 20 in the feature amount storage device 23. Based on the feature quantity of each cluster relating to color, pixels that fall within the respective ranges of hue (H), saturation (S) and luminance (V) of each constituent color of each cluster relating to color are extracted, and By labeling the region, the first narrowing down by the color feature amount is performed (step S22). FIG. 16 shows an example of the result of the first narrowing down based on the color feature amount.

Subsequently, the first object candidate region extraction unit 16 performs the ratio of the area with respect to the entire database image with respect to each labeled region with respect to the threshold Th _area (= ( _{area of the} entire database image) / 200) or less. Is determined to be a noise area instead of an object candidate area, and second narrowing is performed based on the color feature amount (step S23). FIG. 17 shows an example of the result of the second narrowing by the color feature amount.

  Subsequently, when there are a plurality of cluster configuration colors, the first object candidate region extraction unit 16 extracts only the region including all of the configuration colors, and removes the other regions to obtain the color. The third narrowing down based on the feature amount is performed (step S24). FIG. 18 shows an example of the result of the third narrowing by the color feature amount.

In addition, the first object candidate region extraction unit 16 extracts the spatial relationship and area feature amount from each extracted region as described above, and then extracts each cluster related to the extracted spatial relationship and area feature amount and color. The similarity between the spatial relationship and the area feature amount is calculated by the expression (3), and only the region where both are 0.6 or more are extracted, and the other regions are removed, so that the fourth feature based on the color feature amount is obtained. Is narrowed down (step S25). FIG. 19 shows an example of the result of the fourth narrowing by the color feature amount.

In Equation (3), m is the number of dimensions of the spatial relationship (or area) feature amount, and color _c (i) is the i-th feature amount of the spatial relationship (or area) of the cluster relating to color. , Color _d (i) is the i-th feature quantity of the spatial relationship (or area) extracted from the region, and Sim _color takes a value from 0 to 1, and the closer to 1, the more similar between them. Get higher.

  The first object candidate area data representing the area (first object candidate area) extracted as described above is output from the first object candidate area extracting unit 16 to the second object candidate area extracting unit 17. (Step S26).

  The second object candidate area extraction unit 17 applies the following to the first object candidate area represented by the first object candidate area data input from the first object candidate area extraction unit 16: Further, the object candidate areas are further narrowed down by using the feature amount of each cluster relating to the shape.

First, the second object candidate region extraction unit 17, after extracting the rough shape feature ratio _d (aspect ratio) from the first object candidate region, each cluster about the shape stored in the feature storage unit 23 Based on the feature quantity possessed by, the similarity Sim _ratio between the rough shape feature quantity ratio _c possessed by each cluster relating to the shape and the extracted rough shape feature quantity ratio _d is calculated by equation (4). Thereafter, the second object candidate region extraction unit 17 extracts only the region where the similarity Sim _ratio is not less than 0.75 and not more than 1.5, and removes the other regions, whereby the first object based on the shape feature amount. Is narrowed down (step S27). FIG. 20 shows an example of the result of the first narrowing down by the shape feature amount. Note that in Equation (4), the similarity Sim _ratio is a value such that the similarity between the two increases as it approaches 1.

Subsequently, the second object candidate region extraction unit 17 extracts the detailed shape feature amount from the region extracted by the first narrowing down based on the shape feature amount, and then relates each shape related to the shape stored in the feature amount storage device 23. Based on the feature quantity possessed by the cluster, the similarity between the detailed shape feature quantity of each cluster related to the shape and the extracted detailed shape feature quantity is calculated by the following equations (5) to (7). Thereafter, the second object candidate area extraction unit 17 extracts only areas where the degree of similarity is 0.6 or more, and removes the other areas, thereby performing the second narrowing by the shape feature amount (step) S28).

In equation (5), n is the number of dimensions of the detailed shape feature value, form _c (i) is the i-th feature value of the detailed shape of the cluster related to the shape, and form _d (k ) Is the k-th feature amount (calculated by equation (7)) of the extracted detailed shape. This similarity calculation is performed by calculating the absolute value of the difference between the i-th feature amount of the detailed shape of the cluster related to the shape and the corresponding feature amount of the region extracted by the first narrowing down by the shape feature amount. Although the basic idea is to calculate, since it is determined that the difference between the i-th feature amounts is not similar because only a slight angle deviation occurs, equation (6) By calculating like this, the similarity is calculated in consideration of a slight angle shift.
Here, in the equation (6), the i-th feature amount of the detailed shape possessed by the cluster related to the shape, the i-th region of the region extracted by the first narrowing down by the shape feature amount, and the k-th up to two before and after that Is calculated after changing the weight according to the difference from i. And the minimum value in it is made into the similarity regarding the i-th feature-value of a detailed shape.

  Subsequently, the second object candidate area extraction unit 17 sets the area extracted by the second narrowing by the shape feature amount as the second object candidate area, and represents the second object candidate area. The data and the detailed shape feature amount data representing the detailed shape feature amount of each region in the second object candidate region are output to the object region determination unit 18 (step S29).

Based on the second object candidate area data and the detailed feature amount data input from the second object candidate area extraction unit 17, the object area determination unit 18 selects one of the second object candidate areas. If there is only an area, that area is determined as an object area. On the other hand, if there are a plurality of areas in the second object candidate area, the feature amount of the detailed shape is the most among the plurality of areas. After determining the similar area as the object area, the image of the determined object area is set as a blackboard image, and object area data representing the blackboard image is stored in the object storage device 24 (step S30). FIG. 21 shows an example of the blackboard image determined by the object area determination unit 18.
As shown in FIG. 21, a blackboard image (object) included in a construction photo image (database image) can be accurately extracted.

In the above description, the example image data that has been processed to enclose the blackboard area with the black line is stored in the example image storage device 21, but the example image data that has not been processed to enclose the blackboard area with the black line is stored in the example image storage device. 21, after the exemplary image data is read from the exemplary image storage device 21 by the image input device 20, the exemplary image represented by the read exemplary image data is displayed on the display device. You may perform the process which surrounds a blackboard area | region with a black line, seeing the displayed example image. In this case, exemplary image data representing an exemplary image including a blackboard region that has been processed to be surrounded by a black line by an operator is input from the image input device 20 to the object region extraction unit 11.
Further, although the clustering is performed by the k-average method, the clustering may be performed by a single pass method, a centroid method, a median method, or the like.

  As described above, the object retrieval system and method of the present invention can be used to retrieve a blackboard image from a construction photo image, for example.

It is a schematic block diagram for demonstrating the object search system by one Example of this invention. Example 1 FIG. 2 is a diagram illustrating an example of a construction photographic image and an exemplary image extracted from the construction photographic image. Example 1 It is a flowchart for demonstrating the operation | movement of the object search system 10 when extracting the feature-value which each cluster regarding a color and each cluster regarding a shape have from the example image. Example 1 FIG. 6 is a diagram for explaining the operation of the object region extraction unit 11. Example 1 It is a figure which shows an example of the result of having extracted the representative color to 2 colors by the color feature-value extraction part 12. FIG. Example 1 It is a figure which shows an example of the feature-value of the spatial relationship between the representative colors calculated | required by the color feature-value extraction part. Example 1 It is a figure which shows an example of the rough shape feature-value extracted by the shape feature-value extraction part. Example 1 It is a figure for demonstrating the extraction method of the detailed shape feature-value by the shape feature-value extraction part 14. FIG. Example 1 FIG. 6 is a diagram for explaining a procedure 1 of color feature amount clustering processing in a color feature amount clustering unit 13; Example 1 FIG. 10 is a diagram for explaining a procedure 2 of color feature clustering processing in a color feature clustering unit 13; Example 1 FIG. 6 is a diagram for explaining a procedure 3 of color feature amount clustering processing in a color feature amount clustering unit 13; Example 1 FIG. 10 is a diagram for explaining a procedure 4 of color feature amount clustering processing in the color feature amount clustering unit 13; Example 1 FIG. 10 is a diagram for explaining a procedure 5 of color feature amount clustering processing in the color feature amount clustering unit 13; Example 1 It is a figure which shows the example which extracted the feature-value which each cluster regarding a color and each cluster regarding a shape have extracted about the three example images 1-3. Example 1 It is a flowchart for demonstrating operation | movement of the object search system 10 when extracting the blackboard image contained in three database images shown at the bottom of FIG. Example 1 It is a figure which shows an example of the result of the 1st narrowing down by the color feature-value in the 1st object candidate area | region extraction part. Example 1 It is a figure which shows an example of the result of the 2nd narrowing-down by the color feature-value in the 1st object candidate area | region extraction part. Example 1 It is a figure which shows an example of the result of the 3rd narrowing-down by the color feature-value in the 1st object candidate area | region extraction part. Example 1 It is a figure which shows an example of the result of the 4th narrowing-down by the color feature-value in the 1st object candidate area | region extraction part. Example 1 It is a figure which shows an example of the result of the 1st narrowing down by the shape feature-value in the 2nd object candidate area | region extraction part. Example 1 It is a figure which shows an example of the blackboard image calculated | required by the object area | region determination part. Example 1

Explanation of symbols

DESCRIPTION OF SYMBOLS 10 Object search system 11 Object area extraction part 12 Color feature-value extraction part 13 Color feature-value clustering part 14 Shape feature-value extraction part 15 Shape feature-value clustering part 16 1st object candidate area extraction part 17 2nd object candidate area extraction Unit 18 Object region determination unit 20 Image input device 21 Exemplary image storage device 22 Image storage device 23 Feature amount storage device 24 Object storage device S11 to S18, S21 to S30 Steps

Claims (5)

An object area extraction unit (11) for extracting an object area included in the exemplary image;
A color feature quantity extraction unit (12) for obtaining a representative color constituting the object area extracted by the object area extraction unit, and extracting a color feature quantity of the obtained representative color;
A color feature quantity clustering unit (13) for extracting the feature quantity of each cluster relating to color after clustering the color feature quantity extracted by the color feature quantity extraction unit;
A shape feature amount extraction unit (14) for extracting a rough shape feature amount and a detailed shape feature amount of the object region extracted by the object region extraction unit;
After performing clustering of the detailed shape feature values extracted by the shape feature value extraction unit, a shape feature value clustering unit (15) for extracting detailed shape feature values of each cluster;
The feature amount of each cluster relating to the color extracted by the color feature amount clustering unit is stored, and the rough shape feature amount extracted by the shape feature amount extraction unit and the shape feature amount clustering unit are extracted. A feature quantity storage device (23) for storing the feature quantity of each cluster relating to the shape including the detailed shape feature quantity of each of the clusters,
A first object candidate that narrows down the first object candidate region step by step using the feature quantity of each cluster relating to the color stored in the feature quantity storage device for an input image including an object A region extraction unit (16);
For the first object candidate area extracted by the first object candidate area extraction unit, the feature quantity of each cluster relating to the shape stored in the feature quantity storage device is used in stages. A second object candidate area extraction unit (17) for narrowing down two object candidate areas;
An object region determining unit (18) for determining the object region from the second object candidate region extracted by the second object candidate region extracting unit and obtaining the object included in the input image;
An object search system comprising: The first object candidate area extraction unit includes:
For the input image, based on the feature value of each cluster related to the color, pixels that fall within the range of the hue, saturation, and luminance of each constituent color that each cluster related to the color has are extracted, and By performing the first narrowing down by the color feature amount,
For each of the labeled regions, by removing a region whose area ratio to the entire input image is equal to or less than a threshold value, a second narrowing by color feature amount is performed,
When there are a plurality of cluster constituent colors, the third narrowing down by the color feature amount is performed by extracting only the region including all the constituent colors.
After extracting the spatial relation and area feature quantity from each region extracted by the third narrowing by the color feature quantity, the extracted spatial relation and area feature quantity and the spatial relation and area of each cluster related to the color Calculating the degree of similarity with the feature amount, and extracting only a region in which the two similarities are equal to or greater than a predetermined value, thereby performing a fourth narrowing by the color feature amount.
The object search system according to claim 1, wherein: The second object candidate area extraction unit
After extracting a rough shape feature amount from the first object candidate region extracted by the first object candidate region extraction unit, each cluster related to the shape is determined based on the feature amount of each cluster related to the shape. The degree of similarity between the approximate shape feature amount possessed and the extracted approximate shape feature amount is calculated, and only the region in which the calculated similarity value falls within a predetermined range is extracted. Narrow down,
After extracting a detailed shape feature value from the region extracted by the first narrowing down by the shape feature value, based on the feature value of each cluster related to the shape, the detailed shape feature value included in each cluster related to the shape Calculating the degree of similarity with the extracted detailed shape feature amount, and extracting only the region where the calculated degree of similarity is equal to or greater than a predetermined value, thereby performing second narrowing by the shape feature amount;
The object search system according to claim 1, wherein the object search system is an object search system. A first step of inputting example image data in which an object region included in the example image is surrounded by a black line to the object search and extraction system (10) (S11);
In the object search and extraction system, the object region is extracted from the example image, and based on the extracted image of the object region, the feature amount of each cluster related to color and the feature amount of each cluster related to shape are extracted. A second step (S12 to S17);
A third step of inputting an input image including an object to the object search system (S21);
A fourth step in which the first object candidate area extraction unit (16) narrows down the first object candidate areas in a stepwise manner using the feature values of each cluster relating to the color with respect to the input image. (S22 to S25),
The second object candidate area extraction unit (17) uses the feature amount of each cluster related to the shape for the first object candidate area input from the first object candidate area extraction unit, A fifth step (S27, S28) for narrowing down the second object candidate area step by step;
An object region determination unit (18) determines the object region from the second object candidate region input from the second object candidate region extraction unit, and obtains the object included in the input image A sixth step (S30);
An object region extraction method comprising: The second step comprises:
A step (S12) of extracting the object region by detecting a black pixel by scanning a pixel in the horizontal direction with respect to the exemplary image data, the object region extracting unit (11);
A color feature amount extraction unit (12) obtains a representative color constituting the object area extracted by the object area extraction unit, and calculates a range of hue, saturation, and luminance values of the obtained representative color. Extracting as a feature quantity (S13);
The shape feature quantity extraction unit (14) calculates a ratio of the vertical width to the horizontal width of the minimum rectangular area including the object area extracted by the object area extraction unit to extract a rough shape feature quantity (S14). When,
The shape feature amount extraction unit calculates a tangent angle for all pixels serving as an outline of the object region, generates an angle histogram that calculates a frequency of the tangent angle for each predetermined angle, and generates the generated angle histogram. Extracting as a detailed shape feature quantity (S15);
A step of clustering the feature quantity in the range of the hue value extracted by the color feature quantity extraction section after the color feature quantity clustering section (13) extracts the feature quantity of each cluster relating to the color ( S16)
A step (S17) of extracting a detailed shape feature amount of each cluster after the shape feature amount clustering unit (15) performs clustering of the detailed shape feature amount extracted by the shape feature amount extraction unit;
The object search method according to claim 4, further comprising:

Images