JP2003203090A - Image space display method - Google Patents

Abstract

(57) [Summary] [PROBLEMS] To provide an image space display method capable of improving the viewing efficiency. In an image space display method for visualizing a feature amount space of a large amount of images based on a plurality of feature amounts extracted from an image, extremely similar images are grouped, and a feature amount space is represented by an image representing an image group. Generate a screen for The grouping is performed by comparing the feature amounts between the images obtained by the feature amount extraction process using a distance function, and if the feature amounts are smaller than a preset very similar image identification threshold, the images are grouped as very similar images. As a result, all images belong to one of the image groups,
Even if there is no very similar image, the image group includes only the representative image.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an image display technique for finding a desired image from a large number of images by a user, and in particular, it relates to not only a feature amount extracted from an image but also a feature amount extracted from a document. The present invention relates to an image space display method for spatially displaying a feature amount.

[0002]

2. Description of the Related Art As this type of technology, for example, (1) "Visualization interface of feature quantity space in similar image retrieval" (Technical Report of IEICE 98)
Vol. 204) (2) "Information acquisition support from document collection by interactive visualization" (Japan Society for Software Science, 13th Congress) is known.

The above-mentioned known technique (1) is a method of reducing the number of dimensions of the multidimensional space of the feature by the principal component analysis and mapping it to the display space of the two-dimensional space. Further, the above-mentioned known technique (2) introduces a dynamic updating of the visualization result by a user's operation into a visual classification technique that arranges a large number of documents and keywords based on their association.

[0004]

However, according to the known technique (1), the principal component analysis can be performed when the image feature cannot be represented by vector data or when the image similarity cannot be represented by linear combination. No, you cannot use this method. Further, the known technique of (2) is not suitable for interactive display because the amount of calculation is huge and the processing time is required.

On the other hand, a plurality of feature quantities are extracted from the image,
When displaying the feature amount space in a two-dimensional or three-dimensional display space, it is preferable for the user to assign the feature amount to the dimension axis of the display so that the dimension axis is displayed with meaning. Easy to grasp. However, when displaying the images so that the feature amount space can be easily grasped, when the images that are very similar are displayed side by side on the screen, the visual efficiency becomes poor because the screen size is limited.

The present invention has been made in view of the above points, and an object thereof is to provide an image space display method capable of improving the visual recognition efficiency.

[0007]

To achieve the above object, a first means is an image space display method for visualizing a feature quantity space of a large number of images based on a plurality of feature quantities extracted from an image, Group very similar images,
It is characterized in that a screen of the feature quantity space is generated from images representing the image group.

A second means is characterized in that, in the first means, only an image representative of the image group is displayed on the screen.

A third means is characterized in that, in the second means, the number of images belonging to the image group is displayed on the screen together with a representative image as a numerical value.

A fourth means is characterized in that, in the second means, a color and shade indicating the number of images belonging to the image group are displayed on a screen together with a representative image.

A fifth means is characterized in that, in the second means, the display size of the representative image is changed according to the number of images belonging to the image group.

A sixth means is characterized in that, in the first means, the images belonging to the image group are overlapped and displayed with a slight shift.

A seventh means is, in the third to sixth means, that when the cursor is moved onto an image of the image group, the images belonging to the image group under the cursor are sequentially replaced and displayed. Characterize.

An eighth means is characterized in that, in the third to sixth means, when the cursor is moved onto an image of the image group, the image under the cursor is enlarged and displayed.

A ninth means uses the number of images belonging to the image group as the feature quantity in the first means.

A tenth means uses the information added to the image as the feature amount in the image space display for visualizing the feature amount space of a large number of images based on the plurality of feature amounts extracted from the image. Characterize. In this case, examples of the information given to the image include a creation date, an image size, an image format, a keyword, a file name, and the like.

[0017]

BEST MODE FOR CARRYING OUT THE INVENTION Embodiments of the present invention will be described below with reference to the drawings.

1. First Embodiment FIG. 1 shows a first embodiment for implementing the image space display method of the present invention.
2 is a block diagram showing a configuration of an image display device according to the embodiment of the present invention, and FIG. 2 is a block diagram showing processing executed in the image display device of FIG.

In FIG. 1, the image display device includes a CPU 1 that controls the image display device, a ROM 2 that stores a program executed by the CPU 1, and a work area when the CPU 1 executes the program stored in the ROM 2. A RAM 3 which stores dynamic data for executing a program while functioning, a keyboard 4 and a mouse 5 as input devices, a monitor 6 as a display device, and an image display unit 8 for executing an image application 7.
And a peripheral I / F 9 that functions as an interface with the peripheral device 11 such as a digital camera or a scanner.
And a network I / F 10 that functions as an interface with the network 12, and these are connected to be controllable from the CPU 1 via a bus 13.
The image display unit 8 has a feature extraction processing unit 8 as shown in FIG.
1, a very similar image grouping processing unit 82, a tree structure generation processing unit 83, and a display screen generation processing unit 84. In each of the units 81 to 84, a feature extraction process, a very similar image grouping process, a tree structure generation process, and a display screen. The generation process is executed according to the image application 7.

Hereinafter, the image display section 8 having the above-mentioned structure
The processing executed by will be described in the order of 1) feature amount extraction processing 2) very similar image grouping processing 3) feature amount spatial tree structure extraction processing 4) image display screen generation processing.

1) Histogram feature, edge feature,
There are various texture features and the like, and the present invention can be applied to any feature. Of course, the feature amount extracted from the data other than the image may be used. For example, it can be applied to the feature amount extracted from the text. Here, a general histogram feature extraction process will be described. As the image, various images such as an image input from the peripheral device 11 such as a digital camera and a scanner, an image downloaded from the Web, and the like can be considered, but the input method is not limited in this device.

First, an appropriate color space (for example, Lab, Lu
(v, HSV, etc.) to divide this space into multiple areas, check which area of the image each pixel corresponds to in the color space, count the number of pixels in each area, and then determine the total number of pixels To normalize. The normalized pixel count data for each region is a histogram feature. The feature amount of this histogram becomes the point of the feature amount space of the histogram. As the distance between the two feature amounts in the feature amount space of the histogram, the sum of the differences in the number of pixels in each region corresponding to the two feature amounts and the Euclidean distance are generally used. In this way, the distance between the feature quantities can be obtained.

Further, the extracted feature amount is not limited to the feature amount extracted from the image, and the information attached to the image can be used as the feature amount. For example, image creation date, image size, image format, title (file name),
Keywords etc. can be considered. It is necessary to define the distance between the feature amounts at the same time so that the process described below can be performed as the feature amount. What is represented by a numerical value such as an image creation date and an image size may be a distance function using a difference between the numerical values. Further, it is possible to define a distance function that returns 0 in the case of the same format in the case of the image format, and returns 1 in the case of a different format. In the case of a character string such as a title or a keyword, if there is only one word, the difference between the character string codes can be used as the distance function. When the word is composed of a plurality of words, the ratio of words not overlapping can be used as the distance function.

When using a plurality of feature quantities, it is necessary to add the values of the distance function of each feature quantity. This is similar to the case of using only the distance function of the image feature amount, but specifically, it is expressed by D = ΣKi * Di (F1i, F2i). F1i and F2i are the feature amount i, and Di is the distance function of the feature amount i. Ki is a weight for the feature amount i.

2) Very similar image grouping process The characteristic amounts between the images obtained by the characteristic amount extraction are compared by a distance function, and if they are smaller than a preset very similar image identification threshold value, they are grouped as very similar images. Details will be described below.

One image feature amount is sequentially selected. The distance between the image feature amount and the feature amount of the representative images of all image groups is calculated, and if the distance is smaller than the very similar image identification threshold value, the distance is added to the image group. When the image group is not very similar, a new image group is generated and the image is set as a representative image of the image group. If there is an image feature amount that has not been subjected to the above processing, the process returns to. As a result, all the images belong to any one of the image groups, that is, the image group including only the representative images even if there are no images that are very similar. The following processing is executed on the image feature amount of the representative image.

When the number of images in the image group is used as the feature quantity, the feature extraction processing section does not perform the feature quantity calculation but the feature extraction processing section.
It is added to the feature amount of the representative image in this grouping process.

3) Feature Space Tree Structure Extraction Process The feature space is a high-dimensional space, which cannot be simply displayed in two dimensions on the screen. Therefore, first, the structure of the space of the characteristic amount is expressed by a tree structure, and then the tree structure is mapped to the space of the screen, whereby the space of the characteristic amount can be virtually expressed on the screen.

In order to express the feature quantity space in a tree structure, the feature quantity space is clustered and divided into a plurality of subspaces to form a tree structure node. Furthermore, each subspace (node) is clustered and divided into subspaces to form nodes. By performing this recursively, it is possible to generate the tree structure of the space of the feature amount. That is, all the feature quantities are arranged one by one at the lowest individual node (leaf node). An example of the tree structure generated in this way is shown in FIG. The number of nodes dividing the space may be two or more, but for simplicity, the number of nodes is two here.
In this tree structure, similar feature amounts are arranged close to each other, and the tree structure represents the positional relationship of images in the image space.

As a clustering method, a general N
The earliest Neighbor method or the K-means algorithm method can be used. In this tree structure, similar images are arranged close to each other, but the accuracy depends on the clustering accuracy. Therefore, as a method of further increasing the accuracy of clustering, clustering according to the following procedure is possible. In this method, the number of divisions is limited to 2.

Obtaining the central point of the cluster a) Select an arbitrary point A in space. b) The point farthest from the selected point A is the center point C1 of the cluster. c) The point farthest from C1 is the center point C2 of the second cluster.

Sorting Points a) Select an arbitrary point P in space. b) The distance between the center points C1 and C2 of the two clusters and the point P is calculated, and the difference between the distances is obtained as the difference distance. Difference distance = | C1-P | − | C2-P | c) a) -b) is repeated to obtain the difference distances of all points. d) Sort all points in ascending order of difference distance.

The difference between the difference distances before and after the points sorted by the divided difference distances of points is obtained, and the point having the largest difference is set as the boundary of the cluster. The side with a smaller difference distance value than the boundary of the cluster belongs to C1, and the larger side belongs to C2.
Further, it is also possible to make the tree structure well balanced, that is, when it is desired to distribute the images at equal intervals on the final display screen, they can be separated in the middle of the number.
Using the two clusters thus obtained as tree nodes, a tree is generated by recursively performing the same processing for each node, and finally each image belongs to a leaf node. .

In the prior art, the screen was generated from the one tree structure generated in this way, but in the present invention, this tree structure is generated for each meaningful feature amount unit, and each tree structure of the bifurcated branch is used. To generate. The meaningful feature amount is, for example, color, shape, or texture, and the axis of the screen generated by the next screen generation corresponds to each feature amount.

3) Image display screen generation processing After the tree structure is generated, each tree structure is one-dimensionally mapped. The display area on the screen is a one-dimensional data area.

Following the tree structure from the root, a) dividing the one-dimensional data area into two, and arranging two child nodes in each area. The area may be divided into equal parts, and in order to improve the display accuracy, the division point is determined in proportion to the number of nodes belonging to the child node, and the distance from the division point to each central point during clustering. It is also possible to determine the division point in proportion to. Furthermore, considering the distance of the gap between the clusters, a region corresponding to the space of the gap is set, and no child node is designated in the space. By doing so, a display screen that more accurately represents the similarity can be obtained.

B) The above-mentioned 1) feature amount extraction processing and 2) feature amount spatial tree structure extraction processing are performed for each child node.

By recursively processing in this way, the entire tree structure is one-dimensionally mapped. FIG. 4 shows an example of mapping the tree structure of the previous figure. This is an example of dividing a region into equal parts while tracing a tree structure.

By performing this for all tree structures, dimensional data for the number of tree structures can be obtained. Since it is difficult to express more than three dimensions as a screen display, it is desirable to use data up to three dimensions. For example, if a tree structure is generated with color, shape, and texture feature amounts, a three-dimensional display space in which each axis corresponds to color, shape, and texture can be generated.

An example of the final spatial screen display is shown in FIG. The points in the figure are the positions of the feature amount with the shape, texture, and color extracted from the image as parameters, and the image may be displayed as they are. In this embodiment, the shapes, textures, and colors extracted from the image are used as the characteristic amounts, but the present invention can be applied to not only these characteristic amounts but also all the characteristic amounts (such as the characteristic amounts extracted from a document).

For example, in the image displayed as shown in FIG. 6, only the representative image is displayed. In addition, as shown in FIG. 7, the number of images indicating how many images belong to the image group is displayed as a numerical value. Alternatively, as shown in FIG. 8, as the number of images increases, the frame of the image may be changed to white, yellow, and red, and the number of images may be expressed by the color of the frame. In this case, it is also possible to express by shading instead of changing the color. FIG. 9 shows a display example in which a representative image having a large number of images has a large image size. FIG. 10 shows an example in which the images belonging to the image group are slightly overlapped and displayed.

Further, although the number of image groups is three in the above example, a large number of image groups are generated in the actual usage pattern. Therefore, the display area of each representative image becomes narrow, and it may be difficult to visually recognize the image. Therefore, when the cursor is moved on the representative image, it is possible to detect that the cursor is on the image and automatically enlarge and display the image. In addition, if you click further (when the cursor is on the image for a certain period of time without clicking)
The images belonging to the image group may be sequentially enlarged and displayed with each click (at fixed time intervals).
Further, it is possible to sequentially display the images belonging to the image group without the enlarged display.

Further, in the display method of the image feature amount space according to the present embodiment, the number of display dimensions can be selectively used, and the display can be expressed on the one-dimensional axis as shown in FIG. It is possible. In the case of one-dimensional, by folding back the axis, it can be applied to the generally used list display mode shown in FIG.

[0044]

As described above, according to the present invention, very similar images are grouped, and a screen of the feature amount space is generated by the images representing the image groups. The display efficiency can be improved, and thus the visual recognition efficiency of the image can be improved.

Further, according to the present invention, the information added to the image as the feature amount is treated at the same time in addition to the image, and the image is displayed in consideration of the added information. Therefore, it is possible to display from various viewpoints. Therefore, the visual recognition efficiency can be improved.

[Brief description of drawings]

FIG. 1 is a block diagram showing a configuration of an image display device according to an embodiment of the present invention.

FIG. 2 is a block diagram showing processing executed in the image display device according to the embodiment of the present invention.

FIG. 3 is a diagram showing an example of a tree structure generated in the embodiment of the present invention.

FIG. 4 is a diagram showing an example in which all of the tree structure of FIG. 3 is one-dimensionally mapped.

FIG. 5 is a diagram showing a relationship between a three-dimensional display space and a position of a feature amount with a feature amount of a shape, a texture, and a color extracted from an image.

FIG. 6 is a diagram showing a display example in which only representative images are displayed.

FIG. 7 is a diagram showing a display example in which the number of images indicating how many images belong to an image group is displayed numerically.

FIG. 8 is a diagram showing a display example in which a frame color of an image is changed and displayed.

FIG. 9 is a diagram showing a display example in which a representative image having a large number of images is displayed with a large image size.

FIG. 10 is a diagram showing a display example in which images belonging to the same image group are gradually shifted and overlapped and displayed.

FIG. 11 is a diagram showing a display example in which the display method according to the present embodiment is applied to a commonly used display form.

[Explanation of symbols]

1 CPU 2 ROM 3 RAM 6 monitors 7 image applications 8 Image display section 81 Feature Extraction Processing Unit 82 Very similar image grouping processing unit 83 Tree Structure Generation Unit 84 Display screen generation unit 11 Scanners, digital cameras (peripheral devices)

Front page continued (51) Int.Cl. ⁷ Identification code FI theme code (reference) H04N 1/393 H04N 1/393 F term (reference) 5B050 BA06 BA11 BA12 BA13 CA07 DA02 DA04 EA08 EA09 EA12 EA18 EA19 FA02 FA05 FA19 5B057 BA26 CC01 CD05 CE10 CH07 CH08 CH11 CH18 DA08 DA12 DA16 DB02 DB06 DC05 DC19 DC34 DC36 5B075 ND06 PQ02 PQ73 QM05 5C076 AA16 AA19 AA21 AA22

Claims (10)

[Claims] 1. An image space display method for visualizing a feature quantity space of a large number of images based on a plurality of feature quantities extracted from an image, wherein very similar images are grouped, and feature quantities are represented by images representing image groups. An image feature quantity space display method characterized by generating a screen of space. 2. The image feature quantity space display method according to claim 1, wherein only the image representative of the image group is displayed on the screen. 3. The image feature quantity space display method according to claim 2, wherein the number of images belonging to the image group is displayed as a numerical value together with a representative image on the screen. 4. The image feature quantity space display method according to claim 2, further comprising displaying a color and shade indicating the number of images belonging to the image group together with a representative image on a screen. 5. The image feature quantity space display method according to claim 2, wherein the display size of the representative image is changed according to the number of images belonging to the image group. 6. Overlapping images belonging to the image group,
The image feature quantity space display method according to claim 1, wherein the display is performed with a slight shift. 7. The image according to claim 3, wherein when the cursor is moved onto an image of the image group, the images belonging to the image group under the cursor are sequentially switched and displayed. The image feature amount space display method described in [3]. 8. The image under the cursor is enlarged and displayed when the cursor is moved over the image of the image group.
The image feature quantity space display method described in the item. 9. The image feature quantity space display method according to claim 1, wherein the number of images belonging to the image group is used as the feature quantity. 10. An image space display method for visualizing a feature quantity space of a large number of images based on a plurality of feature quantities extracted from an image, wherein information added to the image is used as the feature quantity. Image feature space display method.