JP7568595B2 - Foreground extraction device and program - Google Patents

Description

The present invention relates to a foreground extraction device and program for extracting foreground from an image.

Technology for extracting foreground object regions such as people from video (foreground extraction technology) is widely used as a component technology for video processing applications such as person tracking, object recognition, and 3D space reconstruction. Many foreground extraction technologies have been proposed so far, but the latest technology, "Foreground Extraction Technology," in Non-Patent Document 1, proposes a highly accurate foreground extraction technology that utilizes region division (extraction) technology such as semantic segmentation and machine learning based on graph structures.

Figure 1 is an explanatory diagram of this existing foreground extraction technology, and shows a schematic diagram of the relationship between the data D1 to D5 handled by the technology and the processes P1 to P4 applied to these data.

For part (or all) of the input video data D1 from which foreground extraction is to be performed, feature vectors such as histograms and maximum values of pixel values are calculated (process P2) from each image region (segment) D2 resulting from segmentation (process P1). This is then performed for multiple frames, and a graph D3 is constructed (P31, part of process P3 for constructing graph D3) in which each segment corresponds to a node, using the distance between each vector as an index for the group of feature vectors obtained. In addition, a certain number of frames are selected from a different video D4 to which correct label data (binary image containing the correct foreground region) has been added in advance, and nodes obtained from the segments in the same manner as above are added to graph D3 (P32, the remaining part of process P3 for constructing graph D3).

Here, the process P3 for constructing the graph D3 is composed of a process P31 for constructing nodes with undetermined labels and a process P32 for constructing nodes with correct labels, as shown in the explanatory column in FIG. 1. In the process P32 for correct labels, for nodes corresponding to the video D4 to which correct label data has been assigned, the correct foreground region is compared with each segment to determine whether the segment is foreground (or background), and a foreground/background label is assigned as the correct label to construct the node. On the other hand, in the process P31 for undetermined labels, nodes are constructed with an undetermined label for segments included in the frames of the input video D1 that do not have corresponding correct data. (In FIG. 1, graph D3 is diagrammatically shown with nodes with undetermined labels as white nodes and nodes with correct labels assigned as gray.)

Next, some (or all) of the nodes are sampled from the graph D3, and the undetermined labels are determined (process P4) by an interpolation process (semi-supervised learning) based on the graph structure. Note that at this time, only the foreground/background labels for each segment are determined by the interpolation process, and the pixel values of each segment (such as the size of the area) are not changed. By performing these processes on all of the video data D1 to be extracted, a highly accurate foreground extraction result D5 is obtained.

J. H. Giraldo, S. Javed and T. Bouwmans, "Graph Moving Object Segmentation," IEEE Transactions on Pattern Analysis and Machine Intelligence, 2020. R.Achanta, A.Shaji, K.Smith, A.Lucchi, P.Fua, and S.Susstrunk, "SLIC superpixels compared to state-of-the-art superpixel methods", IEEE Transactions on Pattern Analysis and Machine Intelligence, 2011.

In the existing method, various segmentation methods can be used for process P1, but if the entire screen is divided into small parts, the number of segments and therefore the number of nodes will increase explosively, resulting in a huge amount of calculations for the graph structure-based interpolation process P4. For this reason, Non-Patent Document 1 mainly uses instance segmentation (or semantic segmentation) such as Mask-RCNN, which performs rough object recognition and extraction. Therefore, the obtained foreground extraction result D5 will also be rough.

Here, in order to refine the foreground extraction result D5, it is possible to over-segment the segmentation in process P1 rather than roughly, but non-patent document 1 does not consider any measures to reduce the amount of calculations required in the case of over-segmentation.

From the above discussion, there is still room for improvement in the existing method of Non-Patent Document 1 in the following areas (1) to (3).

(1) The final foreground extraction result D5 is highly dependent on the segmentation result D2, so areas that are not extracted in the instance segmentation P1 cannot be extracted in the final output D5. In other words, foreground areas may disappear from each frame without being processed.

(2) Similarly, if a segmentation method that roughly extracts object boundaries, such as instance segmentation P1, is used, the object boundaries will not be accurately extracted even in the final output D5.

(3) If segmentation (over-segmentation) were used to finely divide the entire image in order to solve the above two problems, the number of nodes in graph D3 would increase explosively, requiring an enormous amount of calculations during the semi-supervised learning process P4.

In view of the problems with the conventional technology described above, the first object of the present invention is to provide a foreground extraction device and program capable of efficiently extracting foreground from video based on a graph structure. The second object of the present invention is to provide a foreground extraction device and program capable of simply and efficiently extracting foreground from an input image using a new configuration that includes only a portion of the foreground extraction device that achieves the first object.

In order to achieve the first object, the present invention provides a foreground extraction device that extracts the foreground in each frame of a first video image that is not assigned a distinction between foreground and background and a second group of images that are assigned the distinction, and applies a first method to each image of the first video image and the second group of images to extract a foreground candidate region. The foreground candidate region is compared with a region segmentation result obtained by applying a second method that achieves a finer region segmentation than the first method to the corresponding image to obtain a target region in which the foreground candidate region is corrected. Each of the target regions is a node, and a graph is constructed in which the distinction between foreground and background is assigned as a label when each node originates from the second group of images. The first feature of the present invention is to obtain a foreground extraction result in each frame of the first video image by estimating the distinction between foreground and background as a label for each node of the graph that originates from the first video image so as to minimize the cost of the graph.

To achieve the second object, the present invention is characterized in that it is a foreground extraction device that extracts the foreground in an input image, which applies a first method to the input image to extract a foreground candidate region, compares the foreground candidate region with a region segmentation result obtained by applying a second method to the input image that achieves a finer region segmentation than the first method, and extracts the foreground in the input image as a target region obtained by correcting the foreground candidate region. The present invention is also characterized in that it is a program that causes a computer to function as the foreground extraction device.

According to the first feature, by using the first and second segmentation methods in combination to reduce the number of extracted regions and extract them as appropriate regions, it is possible to efficiently extract foreground from an image based on a graph structure. According to the second feature, by using the first and second segmentation methods in combination to simply and efficiently extract foreground from an image.

FIG. 1 is an explanatory diagram of a foreground extraction technique according to an existing method. 1 is a functional block diagram of a foreground extraction device according to an embodiment. FIG. 13 is a diagram showing an example in which foreground extraction is inaccurate on an image. FIG. 13 is a diagram showing an example of determining a foreground candidate region as a union; FIG. 13 is a diagram showing a schematic example of processing in a multiplexing division unit. 3 is a functional block diagram of a foreground extraction device according to another embodiment, which includes only a part of the configuration of FIG. 2 . FIG. 1 is a diagram illustrating a hardware configuration of a typical computer.

Figure 2 is a functional block diagram of a foreground extraction device 10 according to one embodiment. The foreground extraction device 10 includes a video input unit 1, a background image generation unit 2, a foreground candidate region determination unit 3, a multiple division unit 4, a feature extraction unit 5, a graph construction unit 6, a label estimation unit 7, a video output unit 8, and a correct answer DB (database) 9.

The foreground extraction device 10 follows the existing method of FIG. 1 as its overall operation, but can perform efficient foreground extraction by an improved method of the existing method. That is, the foreground extraction device 10 reads the input video (without correct answer label) to be subjected to foreground extraction as the first input in the video input unit 1, reads the video (with correct answer label) with the foreground/background distinction prepared in advance in the correct answer DB 9 as the second input, and constructs a graph from these first and second inputs to perform semi-supervised learning, thereby outputting the foreground extraction result of the first input at the video output unit 8. (That is, these first input, second input, and output correspond to the data D1, D4, and D5 shown in FIG. 1, respectively.)

The processing of each functional unit of the foreground extraction device 10 is described in detail below.

<<Video input section 1>>
Video input unit 1 receives as input a video to be subjected to foreground extraction prepared by a user or the like, and outputs this input video to background image generation unit 2 and foreground candidate area determination unit 3. That is, successive frames I _t (t=1, 2, ...) of the video to be subjected to extraction are received as input (first input) in foreground extraction device 10, and are output to background image generation unit 2 and foreground candidate area determination unit 3.

In the following, unless otherwise specified, video frames (images) will be described as having only one channel such as brightness. However, in the case of multiple channels such as color images, the label of the corresponding image region is determined by majority vote of the final output of each channel (foreground/background label for each image region) and one channel from multiple channels (such as the Green channel) is used as input, or a representative value such as the average value is used as input. This allows the method to be applied to color images as well by performing processing for each color channel in the same way.

<<The correct answer is DB9>>
The correct answer DB9 holds, as inputs (second inputs) to the foreground extraction device 10, an image different from the image input to the image input unit 1 and data indicating an image area that is a foreground prepared in advance corresponding to the image (i.e., foreground area data as a correct answer label in each frame of the image). The data of the second input is prepared in advance by a user or the like and recorded in the correct answer DB9. Here, the image data (multiple image frames) and the corresponding correct answer label data (binary images of the corresponding frames) are held in pairs. In addition, for the binary image of the correct answer label data, an image is assumed in which a value of 1 is assigned only to image areas that are thought to be foreground, such as a person area, and a value of 0 is assigned to other background areas.

The data recorded in the correct answer DB9 is output to the background image generation unit 2 and the foreground candidate area determination unit 3 regarding the image, and is output to the graph construction unit 6 regarding the correct answer label data corresponding to the image, and is used in each unit to which it is output.

<<Background image generation unit 2>>
The background image generating unit 2 receives as input the video obtained from the video input unit 1 and the video obtained from the answer DB 9, generates a background image for each video, and outputs the generated image to the feature extraction unit 5. Specifically, as in Non-Patent Document 1, for example, a certain video is received as input, and a background image can be generated by taking the median or average value of each pixel in multiple consecutive frames.

In this embodiment, the background image is assumed to be configured as follows. The video as the first input of the video input unit 1 is composed of a single scene and corresponds to a single background image, and the background image is generated in the background image generation unit 2. (If you want to obtain a foreground extraction result for videos with different scenes, you can divide the video into scenes, input the video composed of a common scene to the first input unit 1, and perform the operation of the entire foreground extraction device 10 multiple times for each common scene video.) On the other hand, depending on the video content, the background image generation unit 2 may generate different background images for each section in the video for the first input video composed of a single scene. For example, if the first input video is long, multiple background images may be generated by taking the average value of a certain section of time near the video to be extracted in order to absorb changes in lighting, etc. Alternatively, in the case of a long video, it may be divided into short videos and one background image may be generated for each short section of the video.

In addition, from the viewpoint of constructing an appropriate graph for semi-supervised learning, the second input video prepared in the answer DB9 is composed of a variety of scenes and corresponds to a variety of background images, and background images are generated in the background image generation unit 2. If background images have been prepared in advance manually for all or part of the second input video, the automatic generation of background images in the background image generation unit 2 may be omitted and the prepared background images may be used.

<<Foreground candidate area determination unit 3>>
The foreground candidate region determination unit 3 receives the image obtained from the image input unit 1 and the image obtained from the correct answer DB 9 as input, and determines the region that is a candidate for foreground extraction (foreground candidate region) for each frame of each image. The foreground candidate region is determined and output to the multiple division unit 4. (Note that, as shown in the configuration of FIG. 2, in the subsequent processing up to the graph construction unit 6, the correct label data is not utilized.) Specifically, the foreground candidate regions can be determined as follows.

First, for a frame I _t at time t in a video to be extracted, an image region (or a bounding box (rectangular surrounding frame) containing the region) recognized as a foreground object by instance segmentation (or semantic segmentation) such as Mask-RCNN is defined as IS (instance segmentation (or semantic segmentation)) region I _t ^s , and a set of pixel indices corresponding to this IS region I _t ^s is defined as Θ(I _t ^s ). Note that the IS region I _t ^s includes all regions (multiple independent regions) extracted in the frame at time t.

In the following description, the same function notation as above will be used as follows.
Θ(・) ... A function that returns a set of pixel indices corresponding to the image region of the input variable Θ _x ^-1 (・) ... A function that takes a set of pixel indices as input and returns the image region of image x

It is desirable that all foreground objects are extracted with high accuracy when this pixel index set Θ(I _t ^s ) (same as the information of IS region I _t ^s ) is obtained, but there are cases where candidate regions that should be extracted cannot actually be extracted due to a lack of learning data for instance segmentation, noise in the target image, etc. In other words, there are cases where the original foreground object is extracted as a region, but the object region is inaccurate due to an excess or deficiency in its boundary, or a region that should be extracted as a foreground object is not extracted at all (extracted completely as background), or conversely, a part that should be background is extracted as foreground.

Figure 3 shows an ^example of inaccurate extraction. With regard to the extraction results _I1S , _I2S , and _I3S of frames _I1 , _I2 , and _I3 that are consecutive in time on ^the video, foreground ^objects (for example, cars) are extracted in two frames _I1 and _I3 , but the boundaries are inaccurate. On the other hand, for the middle frame _I2 , extraction results similar to the extraction results in the two adjacent frames _I1 and _I3 that are consecutive on both sides of the video should be obtained, but no foreground area is extracted at all. (Note that in the example of Figure 3, the extraction results are shown with the white area on the image as the foreground and the black area as the background.)

To solve this problem, the foreground candidate region determining unit 3 performs the following process on the IS region I _t ^s of each frame I _t, and determines the result as the foreground candidate region f _t .

That is, the pixel index set to be output as the foreground candidate region at time t is calculated using the extraction results for λ frames I _{t-λ ,} I _t-λ+1 , ..., I _t , ..., I t _+λ-1 , I _t+λ (2λ+1 frames around frame It, which is the sum of λ frames before frame I _t and λ frames after frame I _t , where λ is a predetermined number) before and after frame I t. Specifically, the union of pixel indexes corresponding to the IS regions of the preceding and following λ frames is output and adopted as the foreground candidate region f _t of the target frame, as shown in the following formula (1).

Note that f _t denotes all foreground candidate regions in the frame at time t, and may include multiple independent foreground candidate regions. Hereinafter, these independent image regions (or bounding boxes containing them) are referred to as f _t,i (i ∈ {1, ..., N}). (That is, f _t = {f _t,i |i ∈ {1, ..., N}} (where N is the number of independent image regions (element regions)), and each region f _t,i is a connected region that does not overlap with other regions f _t,j (j ≠ i).)

Fig. 4 is a diagram showing an example of finding a foreground candidate region as a union using formula (1), in which foreground candidate region f2 is obtained as the union ∪( ^I1S , _I2S , _I3S ⁾ of extraction results _I1S , _I2S , and ^I3S of consecutive frames _I1 , ^I2 , and ^I3 obtained under the ^same circumstances as in ^Fig . ₃ , _with λ=1 set for ^the middle extraction result _I2S , ^and the extraction results _I1S and _I3S of one frame before and after _it . Whereas _no foreground was present in extraction result _I2S ^, _the foreground candidate region _f2 obtained by taking the union shows that the foreground is present.

When a foreground object is moving, it is considered that a spatial positional shift occurs between the foreground object in the image I _t at the actual time t and the foreground candidate region f _t obtained as described above. However, it is expected that the accuracy of the foreground extraction will not be significantly affected because it is expected that the result of over-segmenting the image I _t will be used in the multiple division unit 4 at the subsequent stage to enable foreground extraction according to the image content at the actual time t.

As a modified example, instead of using Θ(I _t+k ^s ) as an element for taking the union in formula (1), the spatial movement amount in each image may be estimated using optical flow or the like, and the pixel positions of each of the λ previous and next images I _t+k may be shifted to time t to use Θ(I' _t+k ^s ) simulating the foreground object position in frame I _t at time t. (Here, region I' _t+k ^s is obtained by applying the movement of the optical flow between times t+ _k , t to region I t+k ^s , and moving region I _t+k ^s at time t+k to a position equivalent to time t. Note that optical flow may be applied in pixel units, or in block units obtained by dividing the image range, or the average value over the entire image range may be applied.)

Note that, when the video as the second input in the correct answer DB9 is not composed of consecutive frames (i.e., when it is composed only of an independent group of images (multiple individual still images)), there is no spatial correlation between adjacent time frame images as video, so the foreground candidate region determination unit 3 may simply output the output result of instance segmentation such as Mask-RCNN for each image in the correct answer DB9 as a foreground candidate region. When both the first input and the second input are composed of video, the above optical flow method may be used (Θ(I' _t+k ^s ) is used instead of Θ(I _t+k ^s )) taking into account spatial correlation for both, and when the second input is composed of individual still images rather than a video, the result of instance segmentation for the second input may simply be used as the foreground candidate region.

<<Multiplex division section 4>>
The multiple segmentation unit 4 multiplexes the segments obtained by applying a plurality of segmentation techniques to the foreground candidate region f _t of each frame I _t , which is the output of the foreground candidate region determination unit 3, to determine the image region to be used for image feature extraction in the downstream feature extraction unit 5. (Note that, as shown in the data processing flow of FIG. 2, the processes of the foreground candidate region determination unit 3 and the multiple division unit 4 are applied to both the image without the correct answer label of the first input and the image (and/or still image group) with the correct answer of the second input. This is expected to have the effect of making the distribution of image regions in the graph construction in the graph construction unit 6 at the latter stage closer between the first input and the second input. On the other hand, the foreground region assigned as the correct answer label of the second input may differ from the image region obtained by the multiple division unit 4 for the second input (corresponding to the foreground region candidate in the case of the first input). Of the image regions obtained by the multiple division unit 4 for the second input, the region that is mostly foreground even in the correct answer label of the second input is assigned a foreground label when the graph is constructed, and the region that is mostly background in the correct answer label of the second input is assigned a background label when the graph is constructed.)

Specifically, the multiple division unit 4 performs region division on a rectangular image including the region f _t,i (each independent region as a component of the foreground candidate region f _t ) obtained by the foreground candidate region determination unit 3, using a segmentation technique different from the segmentation used by the foreground candidate region determination unit 3. As an example, the rectangular image is divided into small regions (SS regions) using superpixel segmentation such as SLIC (Simple Linear Iterative Clustering) disclosed in Non-Patent Document 2. By this operation, each divided segment becomes an over-segmentation of the object or background image, and it becomes impossible to represent the foreground object with only one SS region, but on the other hand, it is expected to obtain a group of regions accurately divided at the object boundary. (That is, since the rectangular image to be segmented contains only one individual region f _t,i as a candidate for one foreground object, and thus has a size similar to that of the individual region f _t,i , the multiple small regions (SS regions) obtained by segmenting the rectangular image of similar size by superpixel segmentation are naturally obtained as a state in which the foreground object region is further subdivided. Note that superpixel segmentation is a region segmentation method, and the multiple small regions (SS regions) obtained contain information on the distinction between regions, but do not contain information on the distinction between foreground and background.)

Next, each element region f _t,i of the foreground candidate region f _t, which represents the general shape of the foreground object, is compared with the SS region set s _t,j (j ∈ {1, … , M}) (where M is the number of SS regions), and the region I _v (feature extraction target region) for feature extraction, v ∈ {1, … V}-th, is determined according to the following equations (2) and (3). (Note that the capital letter V is the number of the determined regions _Iv , and the lower case letter v in the description of "region _Iv " etc. is a continuous index of the feature extraction target regions in the (V) foreground extraction target frames, and in particular, note that _Ivt is used to indicate the feature extraction target region of the frame at time _t . Similarly, in the above, the frame at time t of the input image is referred to as ^It , but in order to more clearly distinguish this from regions _Iv and _Ivt (besides the distinction between the subscripts t and _V used), frame It will be written as ^{frame It} using a superscript.) The SS region s _t,j that does not overlap element region f _t,i , which corresponds to the first line of formula (3), is not considered to be a component of region _Iv , and the SS region s _t, j that overlaps element region f _t, i is considered to be a component of region _Iv . By doing so, for each element region f _t,i , the part that partially overlaps with SS region s _t,j at the boundary (s _t,j ∩f For SS region s _{t,j, t} ,i, the remaining part of the SS region s _t,j that does not overlap (s _t,j \f _t,i , where \ is the difference set) is added to obtain region I _v ^t . (That is, in general, region I _v ^t can be obtained by modifying the foreground candidate region f _t by expanding it.)

In formulas (2) and (3), i∈{1,2,...,N}, and the index i represents each of the N element regions f t,i (N=N(t)) in a frame at time t that is fixed, and the region I _i that is an extension and correction of the element regions f _t,i . On the other hand, "I _v ^t =I _i " in formula (2) represents the extension of the index i of the region I _i that is fixed and used in a frame at time t to the index v of the region I _v ^t in the entire video of the first input (any time t). (For example, if the first input is composed of two frames, and there are two regions i∈{1,2} in the frame at time t=1, and there are three regions i∈{1,2,3} in the frame at time t=2, then the indexes of the five regions in the entire two-frame video will be obtained in the form of v∈{1,2,3,4,5}, etc.) On the other hand, as already explained, j∈{1,2,...,M} is the index of each SS region s _t,j obtained by dividing the containing region (such as a bounding box) of a certain element region f _{t,i in} a frame at a certain time t by superpixel segmentation. (Therefore, for the total number M, M=M(t,i).)

5 shows a schematic example of the processing in the multiple division unit 4, divided into upper, middle and lower parts. For the foreground candidate region f _t in the upper part, as shown in the middle part, each of its element regions f _t,i (in the example of FIG. 5, one of three element regions f _t,i in the foreground candidate region f _t is illustrated) is subject to over-division, and the division result {s _t,j } is obtained. The element region f _t,i is compared with the SS element region s _t,j of the division result by the above formulas (2) and (3), and the SS element region s t,j that at least partially overlaps with the element region f _{t ,} i is added and the boundary of the element region f _t,i is expanded, thereby obtaining the region I _v (=I _v ^t ) of the feature extraction target of formula (2) illustrated in the lower part of FIG. 5.

In this way, the region I _v ^t obtained by the multiple division unit 4 is assumed to be processed by the foreground candidate region determination unit 3 and the multiple division unit 4 so as to include as many correct answers as possible, although it may contain errors (and therefore, conversely, it is processed so as not to miss as many correct answers as possible), and to contain unnecessary parts (however, only the boundaries are neatly divided). In the foreground extraction device 10 of this embodiment, further, by processing after the multiple division unit 4, it is possible to obtain a foreground extraction result by removing unnecessary parts (regions I _v ^t determined to be background) from this region I _v ^t by graph cost optimization. (Note that, as will be described later with reference to FIG. 6, it is also possible to extract the foreground from only one still image by the method exemplified in FIG. 5 using a configuration in which only the foreground candidate region determination unit 3 and the multiple division unit 4 are extracted as part of the foreground extraction device 10 configured in FIG. 2.)

As a segmentation method other than superpixel segmentation, a method of dividing into small regions by a rectangular grid, determining whether each small region is foreground/background, and using the determined small regions as a replacement for the above SS region group s _t,j may be used. (In the method of the above formulas (2) and (3), when the SS region s _t,j overlaps with the boundary of the element region f _t,i , only the non-overlapping portion is expanded, and no portion is removed from the element region f _t ,i. However, when determining whether each small region is foreground/background, the small region located at the boundary of the element region f _t,i may be deleted if it is background according to the determination result.) Also, a foreground region obtained by a "foreground object technology that has drawbacks such as being weak against noise but can represent object boundaries relatively accurately" such as background subtraction may be used as the SS region.

As described above, by performing re-division by segmentation in limited regions such as each of the independent regions f _t,i that make up the foreground candidate region f _t and then narrowing down those regions from which features are to be extracted, it is possible to suppress an increase in the regions subject to feature extraction, which is expected to have the effect of suppressing an increase in the number of nodes in graph construction.

<<Feature Extraction Unit 5>>
The feature extraction unit 5 extracts image features (feature vectors) using as input each of the regions _Iv (obtained by the background image generation unit 2 as corresponding to each region _Iv ) that are subject to feature extraction obtained by the multiple division unit ⁴ , and outputs the extracted image features to the graph construction unit 6. Specifically, for the v-th feature extraction subject region _Ivt , a feature vector can be calculated as follows, in accordance with the method of Non-Patent Document 1.

Let _Bv be the background image region (or its rectangular image) corresponding to the region _Ivt to be extracted, output from the background image generator 2, and ^let _vxt ( ^Θ ( _Ivt )) and ^vyt (Θ( ^Ivt )) be ^the _optical flow ^vectors of the current frame _It ( _⊃Ivt ⁾ corresponding to the horizontal (image x-axis direction) and vertical (image y-axis) indexes Θ( _{Ivt )} , _{respectively. For these Ivt, Ivt-1, Bv, |Ivt -Bv|, vxt(Θ(Ivt)), and vyt} ⁽ _{Θ (} ^Ivt _{) )} ^, _{texture patterns ,} ^{intensity histograms ,} _statistics such as maximum values, minimum values, and standard deviations, features in the hidden ^layer of deep learning, etc. are calculated and combined to output the feature _vector ^xv corresponding to _Ivt .

<<Graph Construction Part 6>>
The graph construction unit 6 uses the feature vector _xv calculated by the feature extraction unit 5 to construct a graph having a node corresponding to the v-th feature extraction target region, and outputs the graph to the label estimation unit 7. Specifically, first, the similarity distance d(i,j) between the two feature extraction target regions (between nodes) (contrary to normal distances, this distance increases as the similarity increases, and the greater the distance, the more similar the regions are, and corresponds to the edge weight d(i,j)) is calculated according to the following formulas (4), (5), and (6).

Here, g(I _i ) is a function that returns the centroid coordinates (or the representative value of similar image coordinates) of the feature extraction target region I _i as a vector, and σ _x and σ _g are the standard deviations corresponding to x _i and g(I _i ), respectively. _{d g} (i,j) works to increase the similarity between regions only when the feature extraction target regions I _i and I _j are included in the same frame (a certain frame I ^t ) and are included within a certain distance (when the spatial distance between the two regions I _i and I _j that exist in the common frame I ^t is within a certain distance β). In addition, α is a user-set variable that adjusts the balance between the similarity of the feature vectors of the feature extraction target regions and the spatial distance on the image plane. Using this similarity distance index d(i,j), the nearest node (most similar node) to the Kth node are connected as neighboring nodes (similar nodes) by the K-nearest neighbor method. In this case, a method may be used in which nodes that are included in the Kth neighbors but have a similarity distance equal to or greater than a certain threshold are not connected. In addition, graph construction based on existing similarity (distance) indices can be widely used.

Particularly in this embodiment, by introducing the term d _g (i, j) in equation (4), it becomes possible to connect feature extraction target regions that are close to each other on the image plane as neighboring nodes, and it becomes possible to collectively extract multiple feature extraction target regions that have been over-segmented by the processing of the label estimation unit at a subsequent stage as the foreground.

In the graph construction unit 6, by determining the distance d(i,j) between different nodes i,j as described above, it is determined whether to provide an edge between nodes i,j (whether nodes i,j are adjacent or not), and a graph is determined between nodes i,j between which an edge is provided, with the similarity distance d(i,j) (the higher the similarity, the larger the edge weight d(i,j) is), and for each node in the graph, if the node corresponds to the first input image of the image input unit 1, a label (a label indicating the distinction between foreground and background) is not given, and if the node corresponds to the second input image of the correct answer DB9, a label is given, and the graph is output to the label estimation unit 7. Note that, if a node v corresponds to the second input image of the correct answer DB9, a label indicating that the node v is foreground/background may be given according to the foreground/background distinction situation in the region I _v from which the feature x _v corresponding to the node v is extracted. If the region I _v is composed of only the background, the node v may be given a label indicating that the node v is background.

That is, some of the nodes of the graph constructed by the graph construction unit 6 are calculated from the image of the correct answer DB 9 and have correct answer label data. Regarding label assignment, the correct answer label data (binary image with 1 assigned to the foreground and 0 assigned to the background) is compared with the feature extraction target region I _v (corresponding to node v) of the image at the corresponding time, and the result of judging whether the region I _v is in the foreground is regarded as the foreground judgment result of node v, and is regarded as the correct answer label in the graph. Specifically, the intersection set of the correct answer label data and the pixel index of I _v is calculated, and the degree of overlap with the correct answer label data is judged using a value ξ _v normalized by the number of pixels of I _v or a value μ _v (Intersection over union) obtained by dividing the number of pixels of the intersection set by the number of pixels of the union of the pixel index of the correct answer label data and I _v , and for I _v with an overlap degree equal to or greater than a threshold, a foreground label is assigned to the corresponding node, and for I _v with an overlap degree less than the threshold, a background label is assigned to the corresponding node.

<<Label Estimation Unit 7>>
In the label estimation unit 7, since the nodes in the graph obtained by the graph construction unit 6 resulting from the video input by the video input unit 1 are in an unlabeled state, the label estimation unit 7 estimates the foreground/background labels for each node in the graph in accordance with Non-Patent Document 1 and outputs the result to the video output unit 8. That is, for the graph obtained by the graph construction unit 6 in which labels are assigned to some nodes, some (or all) of the graph are used to estimate the labels of the unlabeled nodes (I _v ) by semi-supervised learning as shown in the following equation (7).

Here, z is a variable with labels corresponding to all nodes, y is a vector indicating nodes with correct label data, and y(x) is a function that extracts some (or all) of the nodes from y. x is a set of indexes for selecting nodes (often randomly) and is determined by a separate operation (as is commonly done when specifying samples to be used during learning), and user input is also assumed. In other words, "function f_y(x) that changes depending on y" extracts values corresponding to the index specified by the argument x from a vector called a constant y that has already been determined and returns them as a vector, and is written simply as "y(x)". Also, M is a matrix that extracts the nodes corresponding to y(x) from all nodes. (In other words, the constraint equation "s.t.(such that～;～satisfies) Mz=y(x)" in equation (7) indicates that the correct label is fixed in the variable z.)

In addition, ||z|| _TV is a total variation (total variation, regularization term, a type of cost term) defined based on the graph structure, and the correct answer label is fixed according to formula (7), and the undetermined label is varied to minimize the cost term, and the foreground/background estimation result for the undetermined label is obtained. Note that other norms such as Sobolev Norm may be used in addition to total variation. M and y(x) may be designed to extract all nodes rather than extracting some nodes on the graph. By solving this optimization problem (in a group of adjacent nodes (two or more groups of nodes that are determined to have a small distance d(i,j) (normal distance, not a similarity distance) between any two regions (nodes) I _i and I _j belonging to the group of nodes) (i.e., a large similarity (or similarity distance) and a large edge weight d(i,j)), the more similar the labeling results are (the higher the rate of identical labeling results for the nodes in the group of nodes), the lower the cost), the label data of all nodes can be estimated. It should be noted that label estimation is performed simultaneously for all frames of the input video (video obtained from the video input unit 1 as the first input) and is not a frame-by-frame process.

As is commonly done in semi-supervised learning, by setting the node selection index x in formula (7), for example, half of the first input may be label-estimated in the first pass, and this may be added to the correct answer, and the remaining half of the first input may be label-estimated in the second pass. By repeating the process of label-classifying some of the data and using it as training data, it is expected that the effect of estimating errors as little as possible by gradually estimating a small amount of data, rather than estimating the labels of all unlabeled data at once, can be expected.

<<Video output section 8>>
The video output unit 8 outputs the foreground region as an image from the label data corresponding to all nodes estimated from the optimization formula in the label estimation unit 7. For example, for the image region I _v corresponding to the node labeled as foreground from the estimated label data, a pixel value of 1, which is determined as foreground, is assigned, and for the image region corresponding to the node labeled as background, a pixel value of 0 is assigned. By performing these operations for all frames of the input video (the video of the first input of the video input unit 1), the foreground extraction result is obtained as the final result in the foreground extraction device 10. Thus, in this embodiment, a plurality of image regions I _v are obtained in the multiple division unit 4 (as foreground candidates) for each frame of the video of the first input, and a label is estimated for each of these image regions I _v as a node on a graph, thereby obtaining a foreground extraction result in the form of whether or not it was actually foreground. In other words, of all the plurality of image regions I _v obtained in the multiple division unit 4 for each frame of the video of the first input, only a portion that is labeled as foreground, not background, as a node on the graph becomes the foreground extraction result in the foreground extraction result 10.

As described above, according to the embodiment of the present invention, it is possible to perform highly accurate foreground extraction by solving the problems of foreground extraction technology that combines instance segmentation and graph structure, namely (A) "disappearance of foreground regions for each frame" and (B) "inaccuracy of boundaries." As shown in FIG. 4, (A) can be addressed by extracting the foreground target region using multiple frames in the foreground candidate region determiner 3. As shown in FIG. 5, (B) can be addressed by improving the accuracy of the extracted region boundary by combining segmentation techniques in the multiple division unit 4 (and foreground candidate region determiner 3).

The foreground extraction device 10 according to the embodiment of the present invention is expected to improve the accuracy of applications such as 3D reconstruction of foreground objects in free viewpoint video and object recognition/tracking by providing highly accurate foreground extraction that solves the above problems.

Below, we explain various supplementary, additional, and alternative examples.

(1) The foreground extraction technology of the present invention can be applied to various applications as a component technology, and can be used, for example, to extract the movements of an avatar that moves in conjunction with the user's movements, or to generate free viewpoint video of the movements of athletes. These can be used, for example, for remote communication using avatars, or for remotely distributing sports videos as free viewpoint video with a sense of realism, so that users do not need to travel to the site for communication or watching sports, etc., and carbon dioxide emissions can be reduced by saving the energy resources required for user movement, which can contribute to Goal 13 of the United Nations-led Sustainable Development Goals (SDGs), "Take urgent action to combat climate change and its impacts."

(2) The foreground extraction device 10 in Fig. 2 conforms to the framework of Non-Patent Document 1, and obtains the labeling result for the first input as the foreground extraction result by using a first input image to which no correct foreground/background labels have been assigned and a second input image to which correct labels have been assigned. As another embodiment, it is possible to obtain the foreground extraction result from only one input still image by configuring the foreground extraction device 10 in Fig. 2 as only a part of the configuration excerpted from the foreground extraction device 10 in Fig. 2, as shown in Fig. 6, in which the foreground extraction device 10 includes only the foreground candidate region determination unit 3 and the multiple division unit 4. In the configuration of Figure 6, as illustrated in Figure 5, it is assumed that the foreground candidate region determination unit 3 correctly extracts foreground candidate regions (that is, there is no error in that the whole or most part of an extracted foreground candidate region is actually the background and not the foreground, as the correct answer), but if the boundaries of these foreground candidate regions are not necessarily accurate, the boundaries can be made more accurate through processing by the multi-division unit 4, and each region _Iv (the foreground extraction result of the input still image) can be obtained.

(3) FIG. 7 is a diagram showing an example of the hardware configuration of a general computer device 70. The foreground extraction device 10 can be realized as one or more computer devices 70 having such a configuration. When the foreground extraction device 10 is realized by two or more computer devices 70, information required for processing may be sent and received via a network. The computer device 70 includes a CPU (Central Processing Unit) 71 that executes predetermined instructions, a GPU (Graphics Processing Unit) 72 as a dedicated processor that executes some or all of the execution instructions of the CPU 71 in place of the CPU 71 or in cooperation with the CPU 71, a RAM 73 as a main storage device that provides a work area for the CPU 71 (and the GPU 72), a ROM 74 as an auxiliary storage device, a communication interface 75, a display 76, an input interface 77 that accepts user input via a mouse, keyboard, touch panel, etc., a camera 78, and a bus BS for transmitting and receiving data between them.

Each functional unit of the foreground extraction device 10 can be realized by a CPU 71 and/or a GPU 72 that reads from a ROM 74 a predetermined program corresponding to the function of each unit and executes it. Both the CPU 71 and the GPU 72 are a type of arithmetic device (processor). Here, when display-related processing is performed, a display 76 also operates in conjunction with the GPU 72, and when communication-related processing related to data transmission and reception is performed, a communication interface 75 also operates in conjunction with the GPU 72. The processing results by the foreground extraction device 10 may be displayed and output on the display 76. All or part of the image used as input to the foreground extraction device 10 may be obtained by capturing the image with a camera 78.

10...Foreground extraction device, 1...Video input unit, 2...Background image generation unit, 3...Foreground candidate region determination unit, 4...Multiple division unit, 5...Feature extraction unit, 6...Graph construction unit, 7...Label estimation unit, 8...Video output unit, 9...Correct answer DB

Claims (7)

A foreground extraction device that receives a first video in which no distinction between foreground and background is given and a second image group in which the distinction is given, and extracts a foreground in each frame of the first video, the foreground extraction device comprising:
extracting a foreground candidate region by applying a first method to the first image and each image of the second image group;
a second method for realizing a finer region division than the first method is applied to the corresponding image by comparing the foreground candidate region with the region division result to obtain a target region obtained by correcting the foreground candidate region;
constructing a graph in which each region of interest is a node and each node is labeled with a distinction between foreground and background when the node originates from the second set of images;
obtaining a foreground extraction result for each frame of the first image by estimating a foreground or background distinction as a label for each node of the graph that originates from the first image so as to minimize a cost of the graph ;
The foreground extraction device is characterized in that the extraction of the foreground candidate area is performed by taking the sum of the foreground area resulting from applying the first method as a foreground extraction method for each frame of the first video, in a group of frames in the vicinity of each frame in time . 2. The foreground extraction device according to claim 1, wherein the sum is calculated by moving a foreground area in a group of frames in the vicinity of a target frame for which the sum is to be calculated to a position corresponding to the time of the target frame. A foreground extraction device that receives a first video in which no distinction between foreground and background is given and a second image group in which the distinction is given, and extracts a foreground in each frame of the first video, the foreground extraction device comprising:
extracting a foreground candidate region by applying a first method to the first image and each image of the second image group;
a second method for realizing a finer region division than the first method is applied to the corresponding image by comparing the foreground candidate region with the region division result to obtain a target region obtained by correcting the foreground candidate region;
constructing a graph in which each region of interest is a node and each node is labeled with a distinction between foreground and background when the node originates from the second set of images;
obtaining a foreground extraction result for each frame of the first image by estimating a foreground or background distinction as a label for each node of the graph that originates from the first image so as to minimize a cost of the graph ;
The graph is constructed by extracting image features for each of the target regions that are to be nodes, evaluating the different nodes as being more similar to each other as the image features between the different nodes are more similar, and determining whether or not to provide edges between the different nodes;
Evaluating the similarity between the nodes further comprises:
Only when it is determined that the pair of object regions to be regarded as nodes belong to frames at the same time in the video and that the spatial positions of the pair of object regions within the frames at the same time are close to each other,
A foreground extraction device characterized in that the more similar the image features of the pair of target regions are, the more similar the pair of nodes is additionally evaluated . 4. The foreground extraction device according to claim 3 , wherein the cost of the graph is evaluated as being smaller for a group of nodes that are determined to be similar to each other, as the labeling results for distinguishing between the foreground and the background are more similar. 5. The foreground extraction device according to claim 1 , wherein the first technique is instance segmentation or semantic segmentation, and the second technique is superpixel segmentation. 6. The foreground extraction device according to claim 1, wherein the determination of the target area resulting from the correction of the foreground candidate area is performed by determining whether or not each area of the subdivided area division result overlaps with the foreground candidate area, and if there is an overlap, the subdivided area is deemed to belong to the corrected target area, and if there is no overlap, the subdivided area is deemed not to belong to the corrected target area. 7. A program for causing a computer to function as the foreground extraction device according to claim 1.

Images