JP7736613B2 - Learning device, data change estimation device, learning method, data change estimation method, learning program, and data change estimation program - Google Patents

Description

The present invention relates to a learning device, a data change estimation device, a learning method, a data change estimation method, a learning program, and a data change estimation program.

Conventionally, a technique has been known that uses neural networks to extract a target region, which is the region of the object that you want to extract from an image.

For example, the technology described in Patent Document 1 uses a convolutional neural network to extract features for each segmented region of an input image. Then, for each segmented region that overlaps with the inspection region, an integrated feature is calculated using a weight based on the degree of overlap and the extracted feature, and a target region is extracted based on the integrated feature.

Japanese Patent Application Laid-Open No. 2021-114223

However, conventional technology only extracts target areas from images, and is not capable of comparing first data, such as a past image, with second data, such as a current image, to estimate changes, such as the appearance or disappearance of target areas, between the first and second data.

The present invention has been made in consideration of the above circumstances, and aims to provide a learning device, learning method, and learning program that can estimate the disappearance and appearance of an object by training a score calculator that calculates both a score indicating the disappearance or appearance of an object from first data to second data, and a score indicating the disappearance or appearance of an object from second data to first data.

Another object of the present invention is to provide a data change estimation device, a data change estimation method, and a data change estimation program that can estimate the disappearance or appearance of an object by calculating, with a simple configuration, both a score indicating the disappearance or appearance of an object from first data to second data and a score indicating the disappearance or appearance of an object from second data to first data.

In order to achieve the above object, the learning device of the present invention includes: a learning unit that trains the score calculator to input features of the first data and features of the second data in a predetermined order based on learning data indicating a specific change from first data to second data, which is the disappearance or appearance of an object represented by the data, and learning data indicating the specific change from the second data to the first data; the learning unit inputs features of the first data and features of the second data to the score calculator in a predetermined order to calculate a score indicating the specific change in the change from the first data to the second data; and the learning unit trains the score calculator to input features of the first data and features of the second data to the score calculator in the reverse order of the predetermined order to calculate a score indicating the specific change in the change from the second data to the first data.

In the learning device according to the present invention, the learning unit inputs the feature amounts of the first data and the feature amounts of the second data into a score calculator in a predetermined order based on learning data indicating a specific change from first data to second data, which is the disappearance or appearance of an object represented by the data, and learning data indicating the specific change from the second data to the first data, and trains the score calculator to calculate a score indicating the specific change in the change from the first data to the second data, and inputs the feature amounts of the first data and the feature amounts of the second data into the score calculator in the reverse order of the predetermined order, and calculates a score indicating the specific change in the change from the second data to the first data.

In this way, the score calculator is trained to input the feature amounts of the first data and the feature amounts of the second data into a predetermined order to calculate a score indicating a specific change from the first data to the second data, which is the disappearance or appearance of an object represented by the data, and to input the feature amounts of the first data and the feature amounts of the second data into the score calculator in the reverse order of the predetermined order to calculate a score indicating the specific change in the change from the second data to the first data. This allows the disappearance or appearance of an object to be estimated by training the score calculator to calculate both a score indicating the disappearance or appearance of an object from the first data to the second data and a score indicating the disappearance or appearance of an object from the second data to the first data.

The data change estimation device according to the present invention includes: a score calculator that is trained to input, based on training data indicating a specific change from third data to fourth data, which is the disappearance or appearance of an object represented by the data, the specific change from the fourth data to the third data, the feature quantities of the third data and the feature quantities of the fourth data in a predetermined order, and calculate a score indicating the specific change in the change from the third data to the fourth data, and to input the feature quantities of the third data and the feature quantities of the fourth data in the reverse order of the predetermined order, and calculate the score indicating the specific change in the change from the fourth data to the third data; and a score calculation unit that inputs the feature quantities of the first data and the feature quantities of the second data to the score calculator in the predetermined order, and calculates the score indicating the specific change in the change from the first data to the second data, and to input the feature quantities of the first data and the feature quantities of the second data to the score calculator in the reverse order of the predetermined order, and calculates the score indicating the specific change in the change from the second data to the first data.

In the data change estimation device according to the present invention, a score calculator is trained in the same manner as the learning device described above. The score calculation unit inputs the feature amounts of the first data and the feature amounts of the second data in a predetermined order to the score calculator trained by the learning unit, and calculates a score indicating the specific change in the change from the first data to the second data. The score calculation unit also inputs the feature amounts of the first data and the feature amounts of the second data to the score calculator in the reverse order of the predetermined order, and calculates a score indicating the specific change in the change from the second data to the first data.

In this way, the feature amounts of the first data and the feature amounts of the second data are input into the score calculator in a predetermined order to calculate a score indicating the specific change in the transition from the first data to the second data, and the feature amounts of the first data and the feature amounts of the second data are input into the score calculator in the reverse order to the predetermined order to calculate a score indicating the specific change in the transition from the second data to the first data. This makes it possible to estimate the disappearance or appearance of an object by calculating, with a simple configuration, both a score indicating the disappearance or appearance of an object from the first data to the second data and a score indicating the disappearance or appearance of an object from the second data to the first data.

The data change estimation device according to the present invention includes a score calculation unit that inputs the feature amounts of first data and the feature amounts of second data into a score calculator in a predetermined order and calculates a score indicating a specific change from the first data to the second data, which is the disappearance or appearance of an object represented by the data, and inputs the feature amounts of the first data and the feature amounts of the second data into the score calculator in the reverse order of the predetermined order and calculates a score indicating the specific change in the change from the second data to the first data.

In the data change estimation device according to the present invention, the score calculation unit inputs the feature amounts of the first data and the feature amounts of the second data to the score calculator in a predetermined order to calculate a score indicating the specific change in the change from the first data to the second data, and inputs the feature amounts of the first data and the feature amounts of the second data to the score calculator in the reverse order of the predetermined order to calculate a score indicating the specific change in the change from the second data to the first data.

In this way, the feature amounts of the first data and the feature amounts of the second data are input into the score calculator in a predetermined order to calculate a score indicating the specific change in the transition from the first data to the second data, and the feature amounts of the first data and the feature amounts of the second data are input into the score calculator in the reverse order to the predetermined order to calculate a score indicating the specific change in the transition from the second data to the first data. This makes it possible to estimate both the disappearance and appearance of an object from the first data to the second data with a simple configuration.

The score calculator may also calculate the score using asymmetric calculations.

Furthermore, the score calculation unit can further calculate a score indicating no change and a score indicating a change of subject using the score indicating the specific change from the first data to the second data and the score indicating the specific change from the second data to the first data.

The above-mentioned data change estimation device may further include a feature map extraction unit that extracts a feature map for each of the first data and the second data, a candidate area estimation unit that estimates a target likelihood map, which is a map of likelihood indicating the likelihood of a target area, from the feature map using a target area estimation model, a target area extraction unit that extracts a target area from the target likelihood map, and a feature extraction unit that extracts features of the extracted target area from the feature map.

The above-mentioned data change estimation device further includes a score integrating unit, wherein the target region extraction unit extracts multiple target regions from the target likelihood map, the feature extraction unit extracts features for each of the multiple target regions from the feature map, the score calculation unit inputs the features of the target region of the first data and the features of the target region of the second data to a score calculator in a predetermined order for each combination of the target region of the first data and the target region of the second data to calculate a score indicating the specific change from the first data to the second data, and inputs the features of the target region of the first data and the features of the target region of the second data to the score calculator in a reverse order to the predetermined order to calculate a score indicating the specific change from the second data to the first data, and the score integrating unit calculates a maximum, minimum, or average score indicating the specific change from the first data to the second data and a maximum, minimum, or average score indicating the specific change from the second data to the first data.

The data change estimation device further includes a score integrating unit, wherein the first data is a plurality of first data measured at different positions, and the second data is a plurality of second data measured at the same position as each of the plurality of first data but at different measurement times, and the score calculation unit inputs the feature amounts of the first data and the feature amounts of the second data to a score calculator in a predetermined order for each combination of the first data and the second data to calculate a score indicating the specific change from the first data to the second data, and inputs the feature amounts of the first data and the feature amounts of the second data to the score calculation model in a reverse order to the predetermined order to calculate a score indicating the specific change from the second data to the first data, and the score integrating unit can calculate a maximum or minimum score indicating the specific change from the first data to the second data and a maximum or minimum score indicating the specific change from the second data to the first data.

The data change estimation device further includes a score integration unit, wherein the feature map extraction unit extracts multiple types of feature maps for each of the first data and the second data using a neural network, the feature amount extraction unit extracts the feature amount from each of the multiple types of feature maps for each of the first data and the second data, and the score calculation unit inputs the feature amount of the first data and the feature amount of the second data to a score calculator in a predetermined order for each of the feature amounts extracted from each of the multiple types of feature maps, and calculates the specific change from the first data to the second data. The score calculation unit calculates a score indicating the specific change from the second data to the first data, inputs the feature amounts of the first data and the feature amounts of the second data into the score calculation model in the reverse order of the predetermined order, and calculates a score indicating the specific change from the second data to the first data. The score integration unit integrates the scores indicating the specific change from the first data to the second data calculated for each of the feature amounts extracted from each of the multiple types of feature maps, and integrates the scores indicating the specific change from the second data to the first data calculated for each of the feature amounts extracted from each of the multiple types of feature maps.

In a learning method according to the present invention, the learning unit inputs features of the first data and features of the second data into a score calculator in a predetermined order based on learning data indicating a specific change from first data to second data, the specific change being the disappearance or appearance of an object represented by the data, and learning data indicating the specific change from the second data to the first data; the learning unit inputs features of the first data and features of the second data into the score calculator in a predetermined order to calculate a score indicating the specific change in the change from the first data to the second data; and the learning unit inputs features of the first data and features of the second data into the score calculator in a reverse order to the predetermined order to train the score calculator to calculate a score indicating the specific change in the change from the second data to the first data.

In the data change estimation method according to the present invention, the score calculation unit inputs the feature amounts of the first data and the feature amounts of the second data into a score calculator in a predetermined order, calculates a score indicating a specific change from the first data to the second data, which is the disappearance or appearance of an object represented by the data, and inputs the feature amounts of the first data and the feature amounts of the second data into the score calculator in the reverse order of the predetermined order, and calculates a score indicating the specific change in the change from the second data to the first data.

The learning program of the present invention causes a computer to execute the following steps: based on learning data indicating a specific change from first data to second data, which is the disappearance or appearance of an object represented by the data, and learning data indicating the specific change from the second data to the first data, input the features of the first data and the features of the second data into a score calculator in a predetermined order to calculate a score indicating the specific change in the change from the first data to the second data, and input the features of the first data and the features of the second data into the score calculator in the reverse order of the predetermined order to calculate a score indicating the specific change in the change from the second data to the first data.

The data change estimation program of the present invention causes a computer to input the feature amounts of the first data and the feature amounts of the second data into a score calculator in a predetermined order, calculate a score indicating a specific change from the first data to the second data, which is the disappearance or appearance of an object represented by the data, and input the feature amounts of the first data and the feature amounts of the second data into the score calculator in the reverse order of the predetermined order, and calculate a score indicating the specific change in the change from the second data to the first data.

As described above, the learning device, learning method, and learning program of the present invention have the advantage of being able to estimate the disappearance and appearance of an object by training a score calculator that calculates both a score indicating the disappearance or appearance of an object from the first data to the second data, and a score indicating the disappearance or appearance of an object from the second data to the first data.

Furthermore, the data change estimation device, data change estimation method, and data change estimation program according to the present invention have a simple configuration and can effectively estimate the disappearance or appearance of an object by calculating both a score indicating the disappearance or appearance of an object from the first data to the second data, and a score indicating the disappearance or appearance of an object from the second data to the first data.

1 is a block diagram showing a configuration of a change detector learning device according to an embodiment of the present invention; FIG. 10 is a diagram showing an example of a change in belongings. 2 is a block diagram showing a configuration of a change detector learning means of the change detector learning device according to the embodiment of the present invention; FIG. 2 is a block diagram showing a configuration of an object region dividing unit of the change detector learning device according to the embodiment of the present invention; FIG. 4 is a flowchart showing the operation of a target region dividing unit of the change detector learning device according to the embodiment of the present invention. 2 is a block diagram showing the configuration of a change detector learning device and a score calculation unit of an image monitoring device according to an embodiment of the present invention; FIG. FIG. 10 is a block diagram showing the configuration of a score calculator. 1 is a block diagram showing a configuration of an image monitoring device according to an embodiment of the present invention; 1 is a block diagram showing the configuration of a change detector of an image monitoring device according to an embodiment of the present invention; 4 is a flowchart illustrating an operation of the change detector learning device according to the embodiment of the present invention. 4 is a flowchart showing the operation of the image monitoring device according to the embodiment of the present invention. FIG. 10 is a block diagram showing the configuration of a score calculator in a modified example.

Embodiments of the present invention will be described in detail below with reference to the drawings. The following describes embodiments of the present invention: a change detector learning device 2 that constructs a change detector that estimates changes in the presence or absence of belongings of people appearing in input images; and an image monitoring device 1 that uses the change detector created by the change detector learning device 2 to detect changes in the belongings of people appearing in images captured of a specified space and notify the detection results. While this embodiment describes an example in which the object of change detection is a person's belongings, the present invention may also detect changes in the presence or absence of furniture. The change detector learning device 2 is an example of a learning device, and the image monitoring device 1 is an example of a data change estimation device.

<Configuration of the Change Detector Learning Device>
The change detector learning device according to the embodiment of the present invention is a device that learns a change detector using learning data prepared in advance.

The change detector training device can be configured as a computer including a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), RAM (Random Access Memory), and a ROM (Read Only Memory) that stores programs and various data for executing the training processing routine described below. Figure 1 shows a block diagram of the change detector training device 2. As shown in Figure 1, the change detector training device 2 functionally comprises a training data storage means 20, a change detector training means 21, and a trained model storage means 22.

(Learning data storage means 20)
The learning data storage means 20 stores learning data indicating specific changes from a past image set to a current image set, such as the disappearance and appearance of possessions represented by the images.

Specifically, the training data storage means 20 stores a large number of past image sets of changes, a current image set, and change information (appearance or disappearance) as training data. The past images in the training data are examples of first and third data, and the current image in the training data is an example of second and fourth data. This training data is created as follows: First, a person is photographed from multiple viewpoints, and the person area is extracted from the video of each viewpoint at two different times t1 and t2 (t1<t2). The image set at time t1 is designated as the past image set, and the image set at time t2, which captures the same person as in the past image set, is designated as the current image set. If the number of viewpoints from which the person is visible at time t1 is N1 and the number of viewpoints from which the person is visible at time t2 is N2, then the past image set will consist of N1 images, and the current image set will consist of N2 images. The person area is extracted by using an existing person detector or similar to cut out a rectangular area encompassing the person's entire body. The cut-out image is then resized to a specified image size. Information on changes in the possessions of the same person between time t1 and time t2 is then linked to this past image set and current image set and stored.

When comparing two images, there are four possible states for changes in possessions (disappearance only, appearance only, simultaneous disappearance and appearance, and no change). These can be expressed as annotations using the following:

-Whether something that existed in the past is no longer there: Presence of lost objects -Whether something that did not exist in the past but has now appeared: Presence of new objects

Examples of this are shown in Figure 2. As in Figure 2(a), if there are no possessions in the past image but there are possessions in the current image, the change in possessions will be (no lost item, new item). Conversely, as in Figure 2(b), if there are possessions in the past image but no possessions in the current image, this will be (lost item, no new item). Also, as in Figure 2(c), if there are different possessions in the past and current images because the person photographed has changed possessions, this will be (lost item, new item), indicating a change in possessions.

The information on whether or not an object has disappeared and whether or not an object has appeared is represented by two binary values (1 if an object has disappeared or appeared, 0 if it has not), and is stored as change information from the past image set to the current image set.

Note that the past image set and the current image set may be taken in the same location, or in different locations, as long as they are of the same person. The above procedure is performed on videos of various people and changes in various possessions, and a large number of past image sets, current image sets, and change information are created and stored as learning data in the learning data storage means 20.

(Change detector learning means 21)
The change detector learning means 21 reads multiple pieces of learning data, which are a past image set, a current image set, and change information, from the learning data storage means 20, learns a change detector, and stores the resulting change detector in the learned model storage means 22. Figure 3 shows a block diagram of the change detector learning means 21. The change detector is an example of a model. The change detector learning means 21 is composed of a change detector 30 and a learning unit 31.

(Change Detector 30)
The change detector 30 is a neural network that takes a past image set and a current image set as input and outputs a probability value of changes in belongings between those image sets, and is composed of a feature map extraction unit 40, a candidate area estimation unit 41, an individual feature extraction unit 42, an individual score calculation unit 43, an individual score loss calculation unit 44, an overall feature extraction unit 45, an overall score calculation unit 46, an overall score loss calculation unit 47, and a target likelihood distillation loss calculation unit 48.

Here, the overall feature extraction unit 45, overall score calculation unit 46, overall score loss calculation unit 47, and target likelihood distillation loss calculation unit 48 are mainly used to stabilize the learning of the candidate region estimation unit 41. Therefore, these, the individual score loss calculation unit 44, and the learning unit 31 are used only during learning, and are not used during inference.

(Feature map extraction unit 40)
The feature map extraction unit 40 extracts feature maps from input images. Specifically, the feature map extraction unit 40 is a convolutional neural network (CNN) configured using convolution processing, a ReLU function (activation function), pooling processing, and the like. In this embodiment, ResNet is used as the network structure of the feature map extraction unit 40. The feature map extraction unit 40 inputs each image from the input past image set and current image set into ResNet. The feature map that is the final output of ResNet is then combined with several feature maps that are intermediate results within ResNet to output Nf feature maps. These feature maps are in the form of a three-dimensional numerical sequence with dimensions of height, width, and channel. The feature maps calculated by ResNet have different heights and widths, but the other feature maps are resized to fit the feature map with the largest height and width and the highest resolution using bilinear interpolation or the like.

(Candidate area estimation unit 41)
The candidate area estimation unit 41 uses an object area estimation model to estimate an initial object likelihood map, which is a map of likelihood indicating the object area representing a possession, from the feature map. Specifically, the object area estimation model used by the candidate area estimation unit 41 is a CNN configured with convolution processing, a ReLU function (activation function), and the like. Using the feature map calculated by the feature map extraction unit 40, the candidate area estimation unit 41 calculates and outputs an initial object likelihood map for each image in each set of past and current images. In this embodiment, among multiple feature maps, the feature map output from ResNet that is large in length and width and has the highest resolution is input, and an object likelihood map is calculated using a network structure configured with convolution processing, a ReLU function, and another convolution processing. The calculated initial object likelihood map represents the object likelihood, and the more object-like it is, the larger the value.

(Individual feature extraction unit 42)
The individual feature extraction unit 42 is composed of a target region extraction unit 49 and an individual feature vector extraction unit 52. The individual feature vector extraction unit 52 outputs, as an individual feature vector, a feature vector extracted from each of a plurality of target regions for each image in the past image set and the current image set.

(Target region extraction unit 49 of individual feature extraction unit 42)
The object region extracting unit 49 extracts a plurality of object regions from the initial object likelihood map. Specifically, the object region extracting unit 49 is composed of an object region object map normalizing unit 50 and an object region dividing unit 51.

(Target region object map normalization unit 50 of target region extraction unit 49)
The target region object map normalization unit 50 calculates a normalized initial target likelihood map. Specifically, the target region object map normalization unit 50 normalizes the initial target likelihood map calculated by the candidate region estimation unit 41 so that the value of each pixel falls within the range from 0 to 1, as shown in the following equation.
w _i,j = sigmoid(αx _i,j +β)

Here, sigmoid() is a sigmoid function, x _i,j is the value of the coordinate (i,j) of the initial object likelihood map, and α and β are a scale coefficient and a bias coefficient. The scale coefficient α and bias coefficient β are determined by learning. However, they may also be set to fixed values determined in advance. The normalized initial object likelihood map represents the likelihood of an object, and the more likely it is to be an object, the closer the value is to 1. The object map normalization unit for object region 50 outputs this normalized initial object likelihood map as a sigmoid-normalized object likelihood map.

(Target area division unit 51 of target area extraction unit 49)
The object region dividing unit 51 uses the feature map calculated by the feature map extracting unit 40 to divide the sigmoid-normalized object likelihood map calculated by the object region object map normalizing unit 50 into individual object regions so as to sequentially cut out the map, and outputs the divided object regions as object likelihood partial maps. The object likelihood partial map is in the form of a weight map having the same width and height as the sigmoid-normalized object likelihood map and values greater than or equal to 0.

Specifically, as shown in FIG. 4A, the object region segmentation unit 51 is composed of a position likelihood map calculation unit 70, an appearance likelihood map calculation unit 71, an object likelihood map calculation unit 72, and a modification unit 73.

(Position likelihood map calculation unit 70 of object region division unit 51)
The position likelihood map calculation unit 70 first sets the sigmoid normalized object likelihood map calculated by the object region object map normalization unit 50 as an initial object likelihood map. The position likelihood map calculation unit 70 also calculates a position likelihood map from the initial object likelihood map.

(Appearance Likelihood Map Calculation Unit 71 of Object Region Segmentation Unit 51)
The appearance likelihood map calculation unit 71 calculates an appearance likelihood map from the feature map and the initial target likelihood map. Specifically, the appearance likelihood map calculation unit 71 calculates an average feature amount from the feature map using the initial target likelihood map as a weight, and calculates an appearance likelihood map from the feature map, which indicates the likelihood of the feature amount corresponding to the average feature amount.

(Object likelihood map calculation unit 72 of object region division unit 51)
The object likelihood map calculation unit 72 calculates an object likelihood map from the initial object likelihood map, the position likelihood map, and the appearance likelihood map.

(Modification unit 73 of target area division unit 51)
The modifying unit 73 records the target region represented by the target likelihood map as the extraction result of the target region, and modifies the initial target likelihood map so as to change the likelihood of the target region to a predetermined value.

By repeating the calculations by the position likelihood map calculation unit 70, the calculations by the appearance likelihood map calculation unit 71, the calculations by the object likelihood map calculation unit 72, and the modifications by the modification unit 73, multiple object regions are extracted.

(Operation of the target area dividing unit 51)
4B is a schematic flow diagram regarding the operation of the object region dividing unit 51. For the sake of explanation, the sigmoid normalized object likelihood map w _i,j input to the object region dividing unit will be referred to as the initial object likelihood map w _i,j|τ=0 .

First, the position likelihood map calculation unit 70 calculates a position likelihood map from the initial target likelihood map (step SA0).

Specifically, first, the center of gravity μ of the pixel position and the covariance matrix C are calculated using the initial target likelihood map as weights as shown in the following equations, and then the position likelihood map is calculated using these.

Here, p _i,j is the position vector of coordinates i and j, p _i,j = [i,j], M _i,j ^loc is the value of coordinates i and j in the position likelihood map, and K and q are parameters for controlling the distance. Here, if q = 2 and K = 1, the position likelihood map becomes a weight map with a gentle gradient like a Gaussian distribution, but in this embodiment, q = 6 and K = π. This makes the gradient of the weight map steeper, making it easier to separate the object from the background.

Next, the appearance likelihood map calculation unit 71 calculates an appearance likelihood map from the feature map and the initial target likelihood map (step SA1).

Specifically, as shown in the following equation, the initial target likelihood map is used as a weight to find the average feature vector h (h with an overbar), and the similarity map with this is calculated to find the appearance likelihood map.

Here, h _i,j is the feature vector at coordinates i,j of the feature map, and M _i,j ^feat is the value at coordinates i,j of the appearance likelihood map.

Next, the object likelihood map calculation unit 72 calculates a new object likelihood map using the initial object likelihood map, the position likelihood map, and the appearance likelihood map according to the following equation (step SA2).

where SpSoftmax _ij ( ) is the softmax function in the spatial direction,

Furthermore, T _SpSoftmax is a preset temperature parameter.

Next, it is determined whether the iteration termination condition is satisfied (step SA3). If the iteration termination condition is not satisfied, the process returns to step SA0, and the newly calculated object likelihood map w _i,j|τ=1 is used to calculate a position likelihood map and an appearance likelihood map, and then the object likelihood map w _i,j|τ=2 is calculated based on these (steps SA0 to SA2).

If the iteration termination condition is met, the final target likelihood map w _i,j|τ=T is stored in an output list or the like as one of the target likelihood partial maps to be output (step SA3, step SA4).

The iteration termination condition may be, for example, whether the amount of change in the target likelihood map w _i,j|τ has become smaller than a predetermined threshold value, or whether a predetermined number of iterations has been reached.

In the above process, regions are extracted using two types of likelihood maps: a position likelihood map and an appearance likelihood map. Therefore, the extracted regions are regions that are close in location and have similar features (similar appearance in the image). Furthermore, through iterative processing, regions with more similar locations and features are aggregated, improving the accuracy of extraction of possession regions.

Next, the modification unit 73 removes the region of the target likelihood map w _{i,j|τ =T from the initial target likelihood map w i,j|τ} =0 and updates the value of that region of the initial target likelihood map to a predetermined value (step SA5). As a method of removing the region of the target likelihood map w _{i,j|τ= T from the initial target likelihood map w i,j|τ} =0 and changing it to a predetermined value, for example, a method of removing a region equal to or greater than a threshold and changing it to a predetermined value, as shown in the following equation, can be used.

where γ is a predetermined scale factor.

Then, it is determined whether the iteration termination condition is satisfied (step SA6). If the iteration termination condition is not satisfied, the process returns to step SA0, and the newly calculated w _i,j|τ=0 ^new is set as a new initial target likelihood map w _i,j|τ=0 , and the above process is repeated to extract multiple target likelihood partial maps (steps SA0 to SA5).

If the iteration termination condition is met, the target region segmentation unit 51 outputs these as a target likelihood partial map (step SA6, step SA7).

The iteration termination condition may be, for example, whether the number of pixels in the newly calculated w _i,j|τ=0 ^new that are equal to or greater than the threshold value has become smaller than a predetermined threshold value, or whether a predetermined number of iterations has been reached.

The above process divides the object likelihood map into multiple regions in the form of an object likelihood partial map. This makes it possible to extract a region for each item, even if there are multiple items in the image. As a result, the individual feature vector extraction unit 52, described below, can extract the features of each item region without mixing them, improving the accuracy of change detection.

(Individual feature vector extraction unit 52 of individual feature extraction unit 42)
The individual feature vector extraction unit 52 calculates a feature vector for each object likelihood partial map calculated by the object region division unit 51 and outputs it as an individual feature vector. The feature vector is calculated by taking a weighted average of the feature maps, with the object partial map used as the weight, and the feature vector is calculated for each of the multiple feature maps calculated by the feature map extraction unit 40. Specifically, it is calculated using the following equation:

Here, w _r,i,j is the value of coordinates i,j of the r-th target partial map, and h _f,i,j is the feature vector of coordinates i,j of the f-th feature map.

(Individual score calculation unit 43)
The individual score calculation unit 43 calculates an individual score for each of the plurality of target regions based on the feature vector of the target region.

Specifically, for each combination of a target area in the past image set and a target area in the current image set, the individual feature vectors of the target area in the past image set calculated by the individual feature extraction unit 42 and the individual feature vectors of the target area in the current image set are input, and the individual scores are integrated via the score calculation unit 53 and the score integration unit 54 to calculate an integrated individual score.

(Score calculation unit 53 of individual score calculation unit 43)
For each combination of a target area in the past image set and a target area in the current image set, the score calculation unit 53 inputs the individual feature vector of the target area in the past image set and the individual feature vector of the target area in the current image set into the score calculator in a predetermined order to calculate an individual score indicating a change in the amount of a lost possession from the past image set to the current image set, which is a loss score indicating the loss of the possession. Specifically, the individual scores are calculated for various combinations of the individual feature vectors of the past image set and the current image set. For example, the individual scores are calculated for combinations of a target area in the past image set and a target area in the current image set, and for combinations of an image in the past image set and an image in the current image set.

5 shows a block diagram of the score calculation unit 53. The score calculation unit 53 uses Nf score calculators 60 (the same number as the number of feature maps). First, the score calculator 60 calculates individual scores for all combinations of feature vectors of past images and feature vectors of the current image, and sets the scores as loss scores (scores that represent the possibility that something that was previously present may no longer exist, when comparing the past and current images). Specifically, this is expressed as follows:

Here, calcScore _f (, ) is the score calculator 60 for the f-th feature map, h _f,i1,r1 ^t1 is the feature vector calculated from the r1-th subregion of the f-th feature map of the i1-th past image, h _f,i2,r2 ^t2 is the feature vector calculated from the r2-th subregion of the f-th feature map of the i2-th current image, and s _{f,i1,i2,r1,r2} ^disappear is the disappearance score calculated from them.

Next, the score calculation unit 53 uses the same score calculator 60 to input the feature vectors to the score calculator 60 in the reverse order (as in the calculation of the disappearance score described above) to calculate an individual score indicating a change in the disappearance of an item from the current image set to the past image set, i.e., a change in the appearance of an item from the past image set to the current image set, and uses this as an appearance score (a score that indicates the possibility that something that was not present in the past has now appeared, by comparing the past image with the current image). For example, if the feature vector of the past image and the feature vector of the current image are input to the score calculator 60 in this order when calculating the disappearance score, then the feature vector of the current image and the feature vector of the past image are input to the score calculator 60 in this order when calculating the appearance score. Specifically, this is expressed as follows:

The score calculator 60 is composed of a neural network. Figure 6 shows a block diagram of the score calculator 60 of this embodiment. The first feature vector and the second feature vector are input to the score calculator 60, which then applies a common common conversion unit 61 to each of them, performs asymmetric calculations on them, and finally calculates an individual score (disappearance score or appearance score) through a scoring process. Specifically, these are expressed as follows:

Common conversion unit 61:

Asymmetric calculation unit 62:

Scoring processing unit 63: s = sigmoid(FC(v))

Here, h1 is the first feature vector, h2 is the second feature vector, L() is the common conversion unit 61 configured with full connection processing and the ReLU function, h1 ^' and h2 ^' are D-dimensional vectors, FC() is the full connection processing, and sigmoid() is the sigmoid function. Note that in this embodiment, since v is a scalar value, FC() is essentially a linear transformation αv+β with a scale coefficient α and a bias coefficient β.

The asymmetric calculation unit 62 subtracts the second feature vector from the first feature vector, performs threshold processing using the ReLU function, and outputs the average value of those elements.

Here, we will explain the advantages of the score calculator 60 of this embodiment. A conventional method for comparing images is the Siamese Network. This is a neural network that takes two images as input and calculates the distance between them.

On the other hand, in the task of change detection, the value to be output differs depending on the order of images input to the change detector 30. For example, in Figure 2(a), only the current image contains an object, so it is desirable for the change detector 30 to output (no disappeared object, appeared object). On the other hand, if the past image and current image in Figure 2(a) are swapped, it becomes Figure 2(b), in which case it is desirable for the change detector 30 to output (disappeared object, no appeared object), which is different from Figure 2(a). To achieve this asymmetry, the score calculator 60 described above needs to be able to output different values for calcScore(h1, h2) and calcScore(h2, h1).

However, in the aforementioned Siamese Network, the output distance does not change even if the order of images is changed when they are input, so this type of asymmetry cannot be achieved.

In this embodiment, the score calculator 60 uses subtraction, an asymmetric operation, to output different values for calcScore(h1, h2) and calcScore(h2, h1), thereby achieving asymmetry and improving the accuracy of change detection.

(Score integration unit 54 of individual score calculation unit 43)
The score integration unit 54 calculates the maximum, minimum, or average value of the individual scores indicating a change, which is the disappearance of an item from the past image set to the current image set, as an integrated individual score, and calculates the maximum, minimum, or average value of the individual scores indicating a change, which is the disappearance of an item from the current image set to the past image set, i.e., the appearance of an item from the past image set to the current image set, as an integrated individual score.

Specifically, the score integration unit 54 integrates the disappearance scores and appearance scores calculated by the score calculation unit 53 for various combinations, and outputs the integrated individual score. The score integration unit 54 integrates scores for each target region and then for each image. For feature maps, the change detector learning device 2 does not integrate scores, and the individual score loss calculation unit 44 calculates the loss for each.

We will now explain how to integrate the loss scores for each target area. First, for each target area in the past image, the score of the target area with the least change (smallest score) is selected from the target areas in the current image. This is equivalent to matching the target area that represents the most similar possession. Next, the score of the target area with the most change (largest score) is selected from the target areas in the past image. This selects the score of the target area that is determined to have changed the most. This can be written as follows:

The integration of the disappearance scores for each image is performed as in the case of the integration of the scores for each target region, as shown in the following equation.

The integration of the appearance scores is performed in the same manner as the integration of the disappearance scores, as shown in the following equation.
Score integration of target area axis:

Image axis score integration:

The score integration unit 54 outputs the disappearance score s _f ^disappear and the appearance score s _f ^appear for each feature map obtained by the above procedure as an integrated individual score.

(Individual score loss calculation unit 44)
The individual score loss calculation unit 44 calculates a loss using the integrated individual score calculated by the individual score calculation unit 43 and change information (whether a disappeared object is present or absent, and whether an appeared object is present or absent) that is learning data. Specifically, it calculates an error for each disappearance score and each appearance score for each feature map as shown in the following equation, and outputs the sum of these as the individual score loss.

Here, s _f ^disappear and s _f ^appear are the integrated individual scores calculated by the individual score calculation unit 43, t _disappear is a binary value indicating the presence or absence of a disappeared object, t _appear is a binary value indicating the presence or absence of an appeared object, and CE( ) is the cross entropy error function.

The individual score loss calculation unit 44 outputs L _partial as the individual score loss.

(Global feature extraction unit 45)
The global feature extraction unit 45 is composed of a global region object map normalization unit 55 and a global feature vector extraction unit 56. The global feature vector extraction unit 56 outputs, as a global feature vector, a feature vector extracted using the object likelihood map for each image in the past image set and the current image set.

(Global region object map normalization unit 55 of global feature extraction unit 45)
The entire region object map normalization unit 55 normalizes the initial object likelihood map calculated by the candidate region estimation unit 41 using SpSoftmax _ij ( ) so that the sum of the values of each pixel becomes 1, and outputs it as a softmax normalized object map.

(Global feature vector extraction unit 56 of global feature extraction unit 45)
Global feature vector extraction unit 56 calculates a feature vector by taking a weighted average of the feature maps, using the softmax normalized object map calculated by global region object map normalization unit 55 as a weight, and outputs the result as a global feature vector. As with individual feature vector extraction unit 52, a feature vector is calculated for each of the multiple feature maps calculated by feature map extraction unit 40. Specifically, the calculation is performed using the following equation:

Here, w _i,j is the value of coordinates i,j of the softmax normalization target map, and h _f,i,j is the feature vector of coordinates i,j of the f-th feature map.

(Overall score calculation unit 46)
When the global feature vectors of the past image set and the current image set calculated by the global feature extraction unit 45 are input, the global score calculation unit 46 calculates a change detection score through the score calculation unit 57 and the score integration unit 58.

(Score calculation unit 57 of overall score calculation unit 46)
The score calculation unit 57 of the overall score calculation unit 46 calculates an overall score using various combinations of the overall feature vectors of the past image set and the current image set, similar to the score calculation unit 53 of the individual score calculation unit 43. The score calculation unit 57 uses Nf score calculators 60 (the same number as the number of feature maps) to calculate disappearance scores and appearance scores as follows:

Here, calcScore _f (, ) is the score calculator 60 for the f-th feature map, h _i1 ^t1 is the global feature vector calculated from the f-th feature map of the i1-th past image, h _i2 ^t2 is the global feature vector calculated from the f-th feature map of the i2-th current image, and s _f,i1,i2 ^disappear and s _f,i2,i1 ^appear are the disappearance score and appearance score calculated therefrom. The score calculator 60 used in the score calculation unit 57 of the overall score calculation unit 46 is configured by a neural network and has the same structure as the score calculation unit 57 of the overall score calculation unit 46.

(Score integration unit 58 of overall score calculation unit 46)
The score integration unit 58 of the overall score calculation unit 46 calculates the maximum, minimum, or average value of the overall score indicating a change that is the disappearance of a possession from the past image set to the current image set as the integrated overall score, and calculates the maximum, minimum, or average value of the overall score indicating a change that is the disappearance of a possession from the current image set to the past image set, i.e., the appearance of a possession from the past image set to the current image set, as the integrated overall score.

Specifically, the score integration unit 58 integrates the disappearance scores and appearance scores calculated for various combinations by the score calculation unit 57, and outputs the integrated overall score. This score integration unit 58 integrates scores only for each image. For feature maps, the change detector learning device 2 does not integrate scores, and the overall score loss calculation unit 47 calculates the loss for each.

The integration method is the same as that of the score integration unit 54 of the individual score calculation unit 43, and specifically, is as follows:
Score integration of disappearance scores:

Score integration of occurrence scores:

The score integration unit 58 outputs the disappearance score s _f ^disappear and the appearance score s _f ^appear for each feature map obtained by the above procedure as an integrated overall score.

(Total score loss calculation unit 47)
Similar to the individual score loss calculation unit 44, the overall score loss calculation unit 47 calculates the overall score loss based on the integrated overall score calculated by the overall score calculation unit 46 and the change information (with or without disappearance, with or without appearance) that is the learning data. Specifically, the overall score loss calculation unit 47 calculates the error for each disappearance score and appearance score for each feature map as shown in the following equation, and outputs the sum of these as the overall score loss.

where s _f ^disappear and s _f ^appear are the integrated overall scores calculated by the overall score calculation unit 46, t _disappear is a binary value indicating whether an object has disappeared, t _appear is a binary value indicating whether an object has appeared, and CE( ) is a cross-entropy error function. The overall score loss calculation unit 47 outputs L _whole as the overall score loss.

Here, we will discuss the advantages of using the global feature extraction unit 45 and other units during learning. First, in this embodiment, parameters for the candidate area estimation unit 41 and other units are initialized with random numbers, so in the early stages of learning, the target likelihood map calculated by the candidate area estimation unit 41 is often extracted from all areas and does not necessarily appear only in the possession area. Furthermore, because no learning data related to candidate areas is provided and the candidate area estimation unit 41 is trained automatically through training of the change detector, the learning of the candidate area estimation unit 41 is prone to instability.

Furthermore, the processing of the individual feature vector extraction unit 52 uses iterative processing, etc., which makes the neural network deep, making it difficult for losses to propagate to the candidate region estimation unit 41, which further reduces the stability of learning.

In this embodiment, the change detection problem is solved through a simple process of weighted averaging using a softmax normalized target map in the global feature vector extraction unit 56, and the resulting loss is propagated to the candidate area estimation unit 41, etc., thereby facilitating the learning of the candidate area estimation unit 41, etc. This makes it possible to stabilize the learning of the candidate area estimation unit 41, etc.

(Target likelihood distillation loss calculation unit 48)
Object likelihood distillation loss calculation unit 48 measures the similarity between the sigmoid-normalized object likelihood map calculated by object region object map normalization unit 50 and the softmax-normalized object map calculated by entire region object map normalization unit 55, and outputs the object likelihood distillation loss. This prevents the scale coefficient and bias coefficient used in object region object map normalization unit 50 from becoming excessively large or small during learning, resulting in the sigmoid-normalized object likelihood map becoming an inappropriate map for object likelihood.

Specifically, the target likelihood distillation loss is calculated as follows:

Here, wi _,j ^sig is the value of coordinate i,j in the sigmoid normalized target likelihood map, and wi _,j ^sm is the value of coordinate i,j in the softmax normalized target map.

In addition, in this embodiment, the error backpropagation method is used for learning, but in this case, error information is not propagated to the softmax normalized object map, but is propagated only to the sigmoid normalized object likelihood map. This makes it possible to more stabilize the learning of the scale coefficients and bias coefficients used in the object region object map normalization unit 50.

(Learning unit 31)
The learning unit 31 learns the change detector 30, which includes a feature map extraction unit 40, a candidate area estimation unit 41, an overall feature extraction unit 45, an overall score calculation unit 46, an individual feature extraction unit 42, and an individual score calculation unit 43, based on the loss calculated from the overall score and the learning data, and the loss calculated from the individual scores and the learning data.

The learning unit 31 also uses the loss calculated from the normalized object likelihood map obtained by the global feature extraction unit 45 and the normalized object likelihood map obtained by the individual feature extraction unit 42 to train the global feature extraction unit 45 and the individual feature extraction unit 42 of the change detector 30.

More specifically, the learning unit 31 updates the parameters of the change detector 30 so as to minimize the sum L = L _partial + L _whole + L _sim of three losses: the individual score loss L _partial calculated by the individual score loss calculation unit 44, the overall score loss L _whole calculated by the overall score loss calculation unit 47, and the target likelihood distillation loss L _sim calculated by the target likelihood distillation loss calculation unit 48.

A stochastic gradient method is used to minimize the loss. In learning using the stochastic gradient method, the parameters of the feature map extraction unit 40, candidate region estimation unit 41, target region object map normalization unit 50, score calculation unit 53 of the individual score calculation unit 43, and score calculation unit 57 of the overall score calculation unit 46 are first initialized with random values, etc. Then, the gradient of the parameters is calculated using the backpropagation method based on the loss, and the parameters are updated based on this gradient. The change detector learning device 2 learns the change detector 30 by repeating the above processes of reading training data, calculating the loss, and updating the parameters. The iteration termination condition can be, for example, whether the amount of change in loss has become smaller than a predetermined threshold, or whether a predetermined number of iterations has been reached.

In training the change detector 30 described above, the learning unit 31 trains the score calculator 60 to input the individual feature vectors of the past image set and the current image set in a predetermined order to calculate an individual score indicating a change in the form of the disappearance of an item from the past image set to the current image set, and to input the individual feature vectors of the past image set and the current image set in the reverse order to calculate an individual score indicating a change in the form of the appearance of an item from the past image set to the current image set.

Furthermore, in the training of the change detector 30 described above, the learning unit 31 trains the score calculator 60 to input the global feature vector of the past image set and the global feature vector of the current image set in a predetermined order to calculate an overall score indicating a change in the form of the disappearance of an item from the past image set to the current image set, and to input the global feature vector of the past image set and the global feature vector of the current image set in the reverse order to calculate an overall score indicating a change in the form of the appearance of an item from the past image set to the current image set.

The change detector learning device 2 stores the finally determined parameters of the change detector 30, along with the network structure, etc., as a learned model in the learned model storage means 22.

[Configuration of image monitoring device]
An image monitoring device is a device that detects changes in belongings of people captured in an image of a predetermined space and notifies the user of the detection results.

The image monitoring device can be configured as a computer including a CPU, GPU, RAM, and ROM that stores programs and various data for executing the image monitoring processing routine described below. Figure 7 shows a block diagram of the image monitoring device 1. As shown in Figure 7, the image monitoring device 1 functionally comprises an image acquisition means 10, an image storage means 11, a detection means 12, a display means 13, and a trained model storage means 14.

The image acquisition means 10 acquires color images from multiple surveillance cameras capturing images of a specified space from different viewpoints, and outputs the images to the image storage means 11.

The image storage means 11 uses an existing person detector or similar device to cut out rectangular person areas from input images and stores them sequentially along with information on the time of capture. As it stores these images, it uses existing face recognition technology and person identification technology to determine whether they are the same person as images already stored. If they are the same person, the images are linked and stored along with the person ID. If there are more than a certain number of images of the same person stored at two different times that are more than a certain distance apart, the image storage means 11 separates the images by time, and outputs the older images as the past image set and the newer images as the current image set, along with the person ID. Images that are more than a certain distance older than the current time are discarded.

The trained model storage means 14 stores trained change detectors obtained in advance by the change detector training device 2.

The detection means 12 first reads the trained change detector from the trained model storage means 14. Next, when the past image set, current image set, and person ID are input from the image storage means 11, the change detector is used to estimate the probability value of change. Figure 8 shows a block diagram of the change detector 130 in the image monitoring device 1. Note that parts with the same configuration as the change detector 30 in the change detector training device 2 are assigned the same reference numerals and detailed explanations will be omitted. The past image is an example of the first data, and the current image is an example of the second data.

First, the past image set and current image set input to the detection means 12 are input to the feature map extraction unit 40, which outputs a feature map for each image.

The candidate region estimation unit 41 uses the feature map to output an initial target likelihood map. When the feature map and target likelihood map for each image are input, the individual feature extraction unit 42 calculates and outputs the individual feature vectors for each image via the target region extraction unit 49 and the individual feature vector extraction unit 52.

The target region extraction unit 49 extracts multiple target regions from each image through the target region target map normalization unit 50 and the target region segmentation unit 51, and outputs the target regions as target likelihood partial maps.

The individual score calculation unit 43 calculates the individual integrated change score via the score calculation unit 53 and the score integration unit 54.

The score calculation unit 53 inputs the individual feature vectors of the past image set and the current image set into the score calculator 60 in a predetermined order, and calculates an individual score indicating the change in the loss of an item from the past image set to the current image set, which is the loss score.

The score calculation unit 53 uses the score calculator 60 to input the feature vectors to the score calculator 60 in reverse order to calculate individual scores that indicate changes in the appearance of possessions from the past image set to the current image set, and uses these as appearance scores.

The score calculation unit 53 may further use the disappearance score and the appearance score to calculate a score indicating no change and a score indicating a change in belongings. For example, the score calculation may be such that the smaller both the disappearance score and the appearance score are, the larger the score indicating no change, and the larger both the disappearance score and the appearance score are, the larger the score indicating a change in belongings.

Furthermore, the score integration unit 54 in the image monitoring device 1 integrates scores for each feature amount in addition to the score integration process of the score integration unit 54 in the change detector learning device 2. For example, it integrates the individual scores for each combination of a target area in the past image set and a target area in the current image set, integrates the individual scores for each combination of an image in the past image set and an image in the current image set, and integrates the individual scores for each feature map. Therefore, the score integration process is as follows:

Score integration on the target area axis:


Image axis score integration:

Feature axis score integration:

The disappearance score s ^disappear and the appearance score s ^appear are probability values of disappearance likelihood and appearance likelihood, respectively, and the individual score calculation unit 43 outputs these as individual scores.

The detection means 12 outputs the calculated individual scores as detection results, along with the past image set, current image set, person ID, etc.

The display means 13 displays the change detection results, past image set, current image set, and person ID on a display.

[Operation of change detector learning device 2]
FIG. 9 is a flow diagram of a learning processing routine related to the operation of the change detector learning device 2.

When the learning operation begins, the change detector learning device 2 sets the change detector 30 to a predetermined network structure and initializes the network parameters with random values, etc. (step S10).

The change detector learning device 2 reads the past image set, the current image set, and change information as learning data from the learning data storage means 20 (step S11).

The change detector learning device 2 inputs the past image set and the current image set into the change detector 30 using the change detector learning means 21, calculates an individual score, which is a probability value, and calculates the loss using this and the change information, which is the learning data (step S12).

The change detector learning device 2 uses the change detector learning means 21 to calculate the gradient of each parameter using the backpropagation algorithm based on the calculated loss, and then uses this gradient to update each parameter using the stochastic gradient algorithm (step S13).

The change detector learning device 2 determines whether the iteration termination condition is met (step S14). If the iteration termination condition is met, the change detector 30 is stored in the trained model storage means 22 (step S15) and the process ends. If the iteration termination condition is not met, the operations of steps S11 to S14 are repeated until the iteration termination condition is met. The iteration termination condition can be, for example, whether the amount of error variation has become smaller than a predetermined threshold value, or whether a predetermined number of iterations has been reached.

[Operation of image monitoring device 1]
FIG. 10 is a flow chart of an image monitoring processing routine relating to the operation of the image monitoring device 1.

When operation begins, the image monitoring device 1 reads the trained model of the change detector 130 from the trained model storage means 14 (step S20).

Next, the image monitoring device 1 acquires a color image using the image acquisition means 10 (step S21). The acquired captured image is sent to the image storage means 11.

The image monitoring device 1 uses the image storage means 11 to cut out the person area from the received captured image, determine whether it is the same person as the image stored, and if so, link the images together and store them together with the person's ID (step S22).

Next, it is determined whether there are any images to output (step S23). If a certain number or more images of the same person are stored at two different times that are at least a certain distance apart, it is determined that there are images to output, and the image storage means 11 separates these images by time, outputting older images as a past image set and newer images as a current image set together with the person's ID. If there are no images to output, the process returns to image acquisition by the image acquisition means 10 (step S21).

The detection means 12 inputs the received past image set and current image set into the change detector 130, and as a result of the detection process, individual scores, namely, disappearance scores and appearance scores, are calculated, and these scores are output to the display means 13 together with the past image set, current image set, and person ID (step S24).

The image monitoring device 1 displays the acquired image and detection results on the display using the display means 13 (step S25). The monitor then visually checks the displayed detection results to assess the situation, and if an abnormality is detected, dispatches a response officer as necessary.

After displaying the recognition result, the image acquisition means 10 returns to acquiring the image (step S21), and operation continues until the image monitoring device 1 is stopped.

As described above, the change detector learning device and image monitoring device according to this embodiment calculate a target likelihood map from the initial target likelihood map, record the target region represented by the target likelihood map as the target region extraction result, and repeatedly modify the initial target likelihood map to change the likelihood of the target region to a predetermined value. This makes it possible to accurately extract multiple target regions, even if the number of target regions is arbitrary.

Furthermore, with the learning device according to this embodiment, a change detector is trained based on the loss calculated from the overall score and training data, and the loss calculated from the individual scores and training data. In this way, even if the number of target regions is arbitrary, it is possible to train a change detector that can accurately extract multiple target regions.

Furthermore, according to the change detector learning device of this embodiment, the score calculator is trained to input the feature vectors of the past image set and the feature vectors of the current image set into a predetermined order to calculate a score indicating a change in the disappearance of an item from the past image set to the current image set, and to input the feature vectors of the past image set and the feature vectors of the current image set into the score calculator in the reverse order of the predetermined order to calculate a score indicating a change in the disappearance of an item from the current image set to the past image set, i.e., a change in the appearance of an item from the past image set to the current image set. This makes it possible to train the score calculator to calculate both a score indicating the disappearance of an item from the past image set to the current image set and a score indicating the appearance of an item.

Furthermore, according to the data change estimation device of this embodiment, the feature vectors of the past image set and the feature vectors of the current image set are input into a score calculator in a predetermined order to calculate a score indicating a change in the disappearance of an item from the past image set to the current image set, and the feature vectors of the past image set and the feature vectors of the current image set are input into the score calculator in the reverse order to the predetermined order to calculate a score indicating a change in the disappearance of an item from the current image set to the past image set, i.e., a change in the appearance of an item from the past image set to the current image set. This makes it possible to estimate both the disappearance and appearance of an item from the past image set to the current image set with a simple configuration.

<Modification>
The present invention is not limited to the device configuration and operation of the above-described embodiment, and various modifications and applications are possible within the scope of the gist of the present invention.

(Variation 1)
In the above embodiment, an example has been described in which images captured by a camera are used as data to detect changes in belongings as a target, but the present invention can also be applied to detecting changes in a scene. That is, it is possible to detect whether an object in a space has disappeared or a new object has appeared from past and present images of the space. The present invention can also be applied to change detection using not only visible images but also thermal images, distance images, etc. Change detection can also be performed using data other than images. For example, the present invention can be applied to comparing sounds obtained from a microphone as data to detect the appearance or disappearance of abnormal sounds as a target.

(Variation 2)
In the above embodiment, the feature map extraction unit is configured with a neural network, but it may also be configured to extract features such as HOG (Histograms of Oriented Gradients) features, LBP (Local Binary Patterns) features, color histograms, etc. Also, HOG features and CNN features may be used in combination.

(Variation 3)
In the above embodiment, a past image set and a current image set taken from multiple viewpoints are used, but a single viewpoint may also be used.

In addition, instead of images taken from multiple viewpoints, an image set may be a series of images of the same person obtained by tracking a person in images using existing person tracking technology. Furthermore, an image set may be constructed using both multiple viewpoints and person tracking technology.

(Variation 4)
In the above embodiment, the candidate area estimation unit is trained by propagating the change detection loss more directly to the candidate area estimation unit through the overall score loss rather than the individual score loss. However, this is not limited to this. For example, a mask image of the belongings area in the image may be prepared in advance, and the error between the mask image and the initial target likelihood map calculated by the candidate area estimation unit may be added to the loss during training to train the change detector. In this case, the overall feature extraction unit, overall score calculation unit, overall score loss calculation unit, and target likelihood distillation loss calculation unit may not be used.

(Variation 5)
In the above embodiment, the object region dividing unit calculates a plurality of object likelihood partial maps by dividing the object into a plurality of object regions through iterative processing, but the present invention is not limited to this.

One example is a method of grouping similar pixels. Specifically, first, the sigmoid-normalized target likelihood map is searched for local maxima whose centers are largest in local regions, such as 3x3 pixels, and these are stored in a list or similar. Next, pixels that are similar to these in terms of position and features are grouped together. If two local maxima are similar, the one with the smaller value in the sigmoid-normalized target likelihood map is deleted from the list of local maxima. Then, a mask image is created for each local maxima, in which pixels in the same group as the local maxima are assigned a value of 1, and other pixels are assigned a value of 0, and the target region segmentation unit outputs these as target likelihood partial maps.

(Variation 6)
In the above embodiment, a score calculator using asymmetric calculation is used as the score calculator, but this is not limiting. For example, a score calculator using a different feature transformation can be used. Figure 11 shows a block diagram of a score calculator 60A using a different feature transformation.

The score calculator 60A receives the first and second feature vectors as input, and the first and second conversion units 61A and 61B perform different conversion processes on each of them. The symmetric calculation unit 62A then performs a symmetric calculation on them, and finally the scoring processing unit 63 calculates the change score (disappearance score or appearance score). Specifically, these are expressed as follows:

First conversion unit 61A, second conversion unit 61B:

Symmetrical arithmetic unit 62A:

Scoring processing unit 63: s = sigmoid(FC(v))

Here, h1 is the first feature vector, h2 is the second feature vector, _L1 () and _L2 () are the first and second transformations configured using a fully connected process, a ReLU function, etc. Also, h1 ^' and h2 ^' are D-dimensional vectors, FC() is the fully connected process, and sigmoid() is the sigmoid function. In this embodiment, since v is a scalar value, FC() is essentially a linear transformation αv+β with a scale coefficient α and a bias coefficient β.

This score calculator 60A achieves asymmetry by applying different transformations to the first feature vector and the second feature vector.

(Variation 7)
In the above embodiment, only one score calculator is used, but this is not limiting. For example, there is a method in which multiple score calculators are prepared, each calculates a change score, and the average of these is used as a new change score. In this case, by initializing the parameters of each score calculator with different values during learning, each score calculator can be made diverse, thereby improving the accuracy of change detection.

(Variation 8)
In the above embodiment, the score integration unit of the individual score calculation unit of the image monitoring device 1 integrates the scores for each image using the same max-min calculation as the change detector learning device 2, but this is not limited to this. For example, the change detector learning device 2 may learn using the max-min calculation, and the image monitoring device 1 may use an integration method different from that used by the change detector learning device 2, such as the mean-min calculation shown in the following equation:

With max-min calculations, if the past or current image sets contain outlier images that differ significantly from the image expected as input, such as when a person's area is not properly cropped, the change score may fluctuate significantly due to the influence of those images. The mean-min calculation has the advantage of being less susceptible to the influence of outlier images, resulting in a more robust change score.

(Variation 9)
In the above embodiment, the change detection results are displayed on the display means 13 of the image monitoring device 1, but it is also possible to display which locations have changed in more detail. In the change detector, the candidate area estimation unit calculates candidate areas likely to have changed, and calculates a score based on those areas. The final score is then calculated from pairs of areas in multiple images included in the past image set and areas in multiple images included in the current image set. Therefore, by storing the scores of these pairs and tracing which areas contributed to the final score, it is possible to display which areas of which images are different using rectangles or the like on the images.

(Variation 10)
In the above embodiment, the object region dividing unit 51 is described as including the position likelihood map calculating unit 70, the appearance likelihood map calculating unit 71, the object likelihood map calculating unit 72, and the modifying unit 73. However, the present invention is not limited to this. Either the position likelihood map calculating unit 70 or the appearance likelihood map calculating unit may be omitted.

(Modification 11)
In the above embodiment, the image monitoring device 1 and the change detector learning device 2 are separate devices, but the present invention is not limited to this. The image monitoring device 1 may also be configured to further include each of the means of the change detector learning device 2.

As described above, those skilled in the art will be able to make various modifications to suit the implementation within the scope of the present invention.

1 Image monitoring device 2 Change detector learning device 12 Detection means 14 Trained model storage means 20 Training data storage means 21 Change detector learning means 22 Trained model storage means 30, 130 Change detector 31 Learning unit 40 Feature map extraction unit 41 Candidate area estimation unit 42 Individual feature extraction unit 43 Individual score calculation unit 44 Individual score loss calculation unit 45 Overall feature extraction unit 46 Overall score calculation unit 47 Overall score loss calculation unit 48 Object likelihood distillation loss calculation unit 49 Object area extraction unit 50 Object map normalization unit for object area 51 Object area division unit 52 Individual feature vector extraction unit 53, 57 Score calculation unit 54, 58 Score integration unit 55 Object map normalization unit for entire area 56 Overall feature vector extraction unit 60, 60A Score calculator 70 Position likelihood map calculation unit 71 Appearance likelihood map calculation unit 72 Object likelihood map calculation unit 73 Modification unit

Claims (17)

1. A learning device including a learning unit that causes a score calculator to learn based on learning data indicating a change from first data to second data, the change being the disappearance of an object represented by the data, and learning data indicating a change from the second data to the first data, the change being the appearance of an object represented by the data,
The learning unit
the score calculator calculates a score indicating the change that is the disappearance when receiving the feature amount of the first data and the feature amount of the second data as input in a predetermined order; and
causing the score calculator to learn so as to calculate a score indicating the change that is the occurrence when the feature amount of the first data and the feature amount of the second data are received as input in a reverse order to the predetermined order ;
Learning device. 1. A learning device including a learning unit that trains a score calculator based on learning data indicating a change from first data to second data, the change being an appearance of an object represented by the data, and learning data indicating a change from the second data to the first data, the change being a disappearance of the object represented by the data,
The learning unit
the score calculator calculates a score indicating the change that is the occurrence when the feature amount of the first data and the feature amount of the second data are received as input in a predetermined order; and
causing the score calculator to learn so as to calculate a score indicating the change that is the disappearance when the feature amount of the first data and the feature amount of the second data are received as input in a reverse order to the predetermined order ;
Learning device. A data change estimation device including a score calculation unit that calculates a score using a score calculator trained based on learning data indicating a change from third data to fourth data , the change being a disappearance of an object represented by the data, and learning data indicating a change from the fourth data to the third data, the change being an appearance of an object represented by the data,
The score calculator
calculating a score indicating the change that is the disappearance when the feature amount of the third data and the feature amount of the fourth data are received as input in a predetermined order; and
the feature amount of the third data and the feature amount of the fourth data are received as input in a reverse order to the predetermined order, and the feature amount of the fourth data is learned to calculate a score indicating the change that is the occurrence;
The score calculation unit
inputting the feature amount of the first data and the feature amount of the second data into the score calculator in a predetermined order to calculate a score indicating the change that is the disappearance ; and
inputting the feature amount of the first data and the feature amount of the second data into the score calculator in a reverse order to the predetermined order, and calculating a score indicating the change that is the occurrence ;
Data change estimation device. A data change estimation device including a score calculation unit that calculates a score using a score calculator trained based on learning data indicating a change from third data to fourth data , the change being an appearance of an object represented by the data , and learning data indicating a change from the fourth data to the third data, the change being a disappearance of the object represented by the data,
The score calculator
calculating a score indicating the change that is the occurrence when the feature amount of the third data and the feature amount of the fourth data are received as input in a predetermined order; and
the feature amount of the third data and the feature amount of the fourth data are received as input in a reverse order to the predetermined order, and the feature amount of the fourth data is learned to calculate a score indicating the change that is the disappearance;
The score calculation unit
inputting the feature amount of the first data and the feature amount of the second data into the score calculator in a predetermined order, and calculating a score indicating the change that is the occurrence ; and
inputting the feature amount of the first data and the feature amount of the second data into the score calculator in a reverse order to the predetermined order, and calculating a score indicating the change that is the disappearance ;
Data change estimation device. inputting the feature amount of the first data and the feature amount of the second data into a score calculator in a predetermined order, and calculating a score indicating a change from the first data to the second data, the change being the disappearance of an object represented by the data ; and
a score calculation unit that inputs the feature amounts of the first data and the feature amounts of the second data into the score calculator in a reverse order to the predetermined order, and calculates a score that indicates a change from the second data to the first data, the change being an appearance of an object represented by the data;
A data change estimation device comprising: inputting the feature amount of the first data and the feature amount of the second data into a score calculator in a predetermined order, and calculating a score indicating a change from the first data to the second data, the change being the appearance of an object represented by the data ; and
a score calculation unit that inputs the feature amounts of the first data and the feature amounts of the second data to the score calculator in a reverse order to the predetermined order, and calculates a score that indicates a change from the second data to the first data, which is a disappearance of an object represented by the data;
A data change estimation device comprising: 7. The data change estimation device according to claim 3, wherein the score calculator calculates the score indicating the change that is a disappearance or the score indicating the change that is an appearance using an asymmetric operation. The data change estimation device according to any one of claims 5 to 7 , wherein the score calculation unit further uses the score indicating the change that is the disappearance and the score indicating the change that is the appearance to calculate a score indicating no change and a score indicating a replacement of the target. a feature map extraction unit that extracts a feature map for each of the first data and the second data;
a candidate region estimation unit that estimates a target likelihood map, which is a map of likelihood indicating a target region likelihood, from the feature map using a target region estimation model;
a target region extraction unit that extracts a target region from the target likelihood map;
The data variation estimation device according to any one of claims 5 to 8 , further comprising: a feature extraction unit that extracts feature amounts of the extracted target region from the feature map. A learning method in which a learning unit trains a score calculator based on learning data indicating a change from first data to second data, the change being a disappearance of an object represented by the data , and learning data indicating a change from the second data to the first data, the change being an appearance of an object represented by the data ,
the score calculator calculates a score indicating the change that is the disappearance when receiving the feature amount of the first data and the feature amount of the second data as input in a predetermined order; and
A learning method for training the score calculator so that, when the score calculator receives as input the feature amounts of the first data and the feature amounts of the second data in an order reverse to the predetermined order, the score calculator calculates a score indicating the change that is the occurrence . A learning method in which a learning unit trains a score calculator based on learning data indicating a change from first data to second data, the change being an appearance of an object represented by the data , and learning data indicating a change from the second data to the first data, the change being a disappearance of the object represented by the data ,
the score calculator calculates a score indicating the change that is the occurrence when the feature amount of the first data and the feature amount of the second data are received as input in a predetermined order; and
a learning method for training the score calculator so that, when the score calculator receives as input the feature amounts of the first data and the feature amounts of the second data in an order reverse to the predetermined order, the score calculator calculates a score indicating the change that is the disappearance . a score calculation unit inputs the feature amount of the first data and the feature amount of the second data to a score calculator in a predetermined order, and calculates a score indicating a change from the first data to the second data, the change being the disappearance of an object represented by the data ; and
A data change estimation method comprising: inputting the feature amounts of the first data and the feature amounts of the second data into the score calculator in a reverse order to the predetermined order; and calculating a score indicating a change from the second data to the first data, the change being the appearance of an object represented by the data . a score calculation unit inputs the feature amount of the first data and the feature amount of the second data to a score calculator in a predetermined order, and calculates a score indicating a change from the first data to the second data, the change being an appearance of an object represented by the data ; and
A data change estimation method comprising: inputting the feature amounts of the first data and the feature amounts of the second data into the score calculator in a reverse order to the predetermined order; and calculating a score indicating a change from the second data to the first data, which is a disappearance of an object represented by the data . A learning program for training a score calculator based on learning data indicating a change from first data to second data, the change being the disappearance of an object represented by the data , and learning data indicating a change from the second data to the first data, the change being the appearance of an object represented by the data,
the score calculator calculates a score indicating the change that is the disappearance when receiving the feature amount of the first data and the feature amount of the second data as input in a predetermined order; and
A learning program for causing a computer to train the score calculator so that, when the score calculator receives as input the feature amounts of the first data and the feature amounts of the second data in the reverse order of the predetermined order, the score calculator calculates a score indicating the change that is the occurrence . A learning program for training a score calculator based on learning data indicating a change from first data to second data, the change being an appearance of an object represented by the data , and learning data indicating a change from the second data to the first data, the change being a disappearance of the object represented by the data,
the score calculator calculates a score indicating the change that is the occurrence when the feature amount of the first data and the feature amount of the second data are received as input in a predetermined order; and
A learning program for causing a computer to train the score calculator so that, when the score calculator receives as input the feature amounts of the first data and the feature amounts of the second data in an order reverse to the predetermined order, the score calculator calculates a score indicating the change that is the disappearance . inputting the feature amount of the first data and the feature amount of the second data into a score calculator in a predetermined order, and calculating a score indicating a change from the first data to the second data, the change being the disappearance of an object represented by the data ; and
A data change estimation program for causing a computer to execute the following: inputting the feature amounts of the first data and the feature amounts of the second data into the score calculator in a reverse order to the predetermined order ; and calculating a score indicating a change from the second data to the first data, which is an appearance of an object represented by the data . inputting the feature amount of the first data and the feature amount of the second data into a score calculator in a predetermined order, and calculating a score indicating a change from the first data to the second data, the change being the appearance of an object represented by the data ; and
A data change estimation program for causing a computer to execute the following: inputting the feature amounts of the first data and the feature amounts of the second data into the score calculator in a reverse order to the predetermined order; and calculating a score indicating a change from the second data to the first data, which is a disappearance of an object represented by the data .