JP7604858B2 - Monitoring device, monitoring method, and program - Google Patents

Description

This disclosure relates to server monitoring.

Various services provided via the Internet are made possible by running servers that have installed programs on computers. However, computers do not always operate stably, and unexpected problems can cause servers to go down or slow down. Constantly monitoring servers and responding quickly when a problem occurs is important for providing a stable service.

Electronic devices are monitored through a network to detect the occurrence of failures in the electronic devices. Patent Document 1 discloses that a maintenance-managed device such as a personal computer or server, or a monitoring device, notifies the occurrence of a failure in a maintenance-managed device. Patent Document 2 discloses that a managed device such as a printer is monitored at a predetermined monitoring interval according to the usage status.

As a technology related to the present disclosure, Patent Document 3 discloses an operation and maintenance support system that determines whether or not an equipment malfunctions by taking into account information on the cumulative operating status of the equipment and parameters specific to each equipment.

JP 2016-081374 A JP 2014-053027 A JP 2010-271905 A

The failure rate of electronic devices varies depending on the operating environment, such as temperature and humidity. However, in Patent Document 2, the monitoring of electronic devices is not controlled with the operating environment of the devices taken into consideration.

The present disclosure aims to provide a monitoring device etc. that controls monitoring while taking into account the operating environment of the server.

The monitoring device according to the present disclosure includes an acquisition means for acquiring the individual failure rate caused by the hardware, operating time, and operating environment of each server, and a determination means for determining the monitoring interval for the server based on the individual failure rate.

The monitoring method disclosed herein obtains the individual failure rate caused by the hardware, operating time, and operating environment of each server, and determines the monitoring interval for the server based on the individual failure rate.

The program disclosed herein acquires the individual failure rate caused by the hardware, operating time, and operating environment of each server, and causes a computer to execute a process of determining a monitoring interval for the server based on the individual failure rate.

According to this disclosure, monitoring can be controlled taking into account the operating environment of the server.

1 is a schematic diagram showing a configuration of a maintenance management system according to a first embodiment. FIG. 3 is a schematic diagram of the hardware of the server 300. FIG. 2 is a sequence diagram showing an example of the operation of the maintenance management system. 1 is a graph illustrating an example of a relational model. 13 is a graph showing another example of a relational model. FIG. 11 is a block diagram showing a configuration of a monitoring device 100 according to a second embodiment. FIG. 2 is a schematic diagram showing an example of the arrangement of a calculation device 120. 13 is a graph showing another example of the relationship between the individual failure rate and the determined monitoring interval. 10 is a flowchart showing an example of the operation of the monitoring device 100 according to the second embodiment. FIG. 5 is a block diagram showing an example of the hardware configuration of a computer 500.

In this disclosure, a monitoring device that monitors multiple electronic devices that make up a large-scale system for maintenance management purposes is described. As an example of an electronic device to be monitored, a monitoring device that monitors a server is taken as an example and described in the following embodiment with reference to the drawings. However, the monitored device may also include electronic devices other than servers.

[First embodiment]
(composition)
1 is a schematic diagram showing the configuration of a maintenance management system according to a first embodiment. The maintenance management system includes a monitoring device 100 and one or more server racks 200 (200_1, ..., 200_y) to be monitored. Each server rack 200 includes one or more servers 300 (300_1, 300_2, ..., 300_x) and a temperature sensor 400. The monitoring device 100 is communicatively connected to each server 300 and temperature sensor 400 via a network 010. The number of servers 300 in each server rack 200 can be changed arbitrarily.

The monitoring device 100 monitors whether or not a failure has occurred in the server 300. In the first embodiment, the monitoring device 100 includes an acquisition unit 101, a calculation unit 102, and a determination unit 103.

The acquisition unit 101 acquires hardware monitoring information, operation information, and information on the operating environment for each server 300. The hardware monitoring information indicates hardware monitoring data for determining abnormal trends in each component inside the server 300. The operation information indicates the operating status of the server 300, such as the operating time from initial placement or the number of years of use. The information on the operating environment is, for example, information indicating the temperature or humidity inside the server rack 200, or the temperature of the server 300. Furthermore, the acquisition unit 101 acquires the individual failure rate based on the hardware, operation information, and operating environment of each server 300 calculated by the calculation unit 102.

The calculation unit 102 calculates the individual failure rate P for each server 300 based on the hardware monitoring information, operation information, and information on the operating environment acquired by the acquisition unit 101. The calculation of the individual failure rate P will be described later.

The determination unit 103 determines the monitoring interval for each server 300 based on a relational model between the individual failure rate of each server 300 and the monitoring interval. The monitoring interval is the time interval from when the monitoring device 100 executes a fault monitoring process for the server 300 until it executes the monitoring process again. As the monitoring process, the monitoring device 100 may, for example, send an input/output request to the server 300 and check whether the response time to the input/output request is within a normal range. Alternatively, as the monitoring process, the monitoring device 100 may check whether the monitoring data for each component of the server 300 is within a normal range.

The relational model between the individual failure rate and the monitoring interval may show a decreasing trend of the monitoring interval corresponding to an increase in the individual failure rate. The relational model may be represented by a relational equation between the individual failure rate and the monitoring interval, or may be represented by a table. The relational model may be obtained by machine learning using, for example, the failure rate, the monitoring interval, the number of servers, and the network bandwidth as explanatory variables, with the network load as the objective variable.

In the first embodiment, the temperature sensor 400 measures the temperature inside the server rack 200. The temperature sensor 400 transmits the measured temperature to the monitoring device 100 as information about the operating environment of the servers 300_1, ..., 300_x in the server rack 200.

When the temperature of each server 300 is used as information about the operating environment, one temperature sensor 400 may be installed for each server 300. The temperature sensor 400 transmits the measured temperature of each server to the monitoring device 100.

Other sensors may be installed in the server rack 200 depending on the type of information about the operating environment acquired by the acquisition unit 101. If humidity is used as the information about the operating environment, the description of the temperature sensor 400 in the following explanation can be replaced with a humidity sensor.

Figure 2 is a hardware schematic diagram of server 300. It comprises a CPU (Central Processing Unit) 301, memory 302, HDD (hard disk drive) 303, EtherPort 304, and BMC (Baseboard Management Controller) 309. BMC 309 acquires monitoring data from the hardware of server 300 (each component and element configured within server 300) and transmits it to monitoring device 100 as hardware monitoring information.

(Operation)
Fig. 3 is a sequence diagram showing an example of the operation of the maintenance management system. For the sake of simplicity, only one server 300 is shown in Fig. 3, but each server 300 operates in the same manner.

<<Calculating the failure rate due to hardware>>
The acquisition unit 101 of the monitoring device 100 acquires hardware monitoring information from the server 300 via the network 010 and the BMC 309. The calculation unit 102 calculates a failure rate P _h caused by hardware based on the monitoring information acquired by the acquisition unit 101 (step S101).

_The hardware failure rate P _h is the probability that a failure of an individual device may occur, and is calculated, for example, between 0 and 1 (or 0% and 100%).

The hardware monitoring information includes monitoring data corresponding to multiple monitored components such as the CPU 301, memory 302, HDD 303, ethernet port 304, and power supply. The monitoring data related to the CPU 301 is, for example, the CPU usage rate and temperature. The monitoring data related to the memory 302 is, for example, the number of errors in ECC (Error checking and correcting). The monitoring data related to the HDD 303 is, for example, an inspection value obtained by SMART (Self-Monitoring, Analysis and Reporting Technology). The monitoring data related to the power supply indicates, for example, the redundant state of the power supply.

The calculation unit 102 determines whether or not there is an abnormal trend based on, for example, the value of the monitoring data acquired for each hardware component that is subject to failure. The calculation unit 102 determines that a component has an abnormal trend when the monitoring data exceeds a predetermined threshold. Alternatively, the calculation unit 102 may determine that a component has an abnormal trend when the failure rate of the component calculated based on the value of the monitoring data of the component is equal to or greater than a predetermined threshold. When the monitoring information includes information on 100 components, the calculation unit 102 increases the failure rate P _h by, for example, +0.01 (or +1%) by the number of components that have an abnormal trend. Note that the increased failure rate P _h may be adjusted for each component based on the importance of the component included in the monitoring information and the value of the measured data.

<< Calculation of failure rate due to operating environment >>
The acquisition unit 101 of the monitoring device 100 acquires temperatures measured by the temperature sensors 400 installed in the server rack 200 as information about the operating environment of the servers 300. The calculation unit 102 calculates the failure rate _Pe caused by the operating environment based on the temperatures acquired by the acquisition unit 101 (step S102).

The failure rate P _e due to the operating environment is a failure probability calculated based on the difference between the environment in which the server 300 is installed and the stable operating conditions. The failure rate P _e is calculated, for example, between 0 and 1 (0% to 100%).

For example, the calculation unit 102 calculates the failure rate P e as 0 (or 0%) if the temperature is the median of the assumed operating temperatures, as 1 (or 100%) if the temperature is equal to or higher than the upper temperature limit, and as 1 (or 100%) if the temperature is equal to or lower than the lower temperature limit. The calculation unit 102 may calculate the failure rate P _e by taking into consideration the rate of change of the operating environment temperature, the duration of the temperature, or the number of times a predetermined upper or lower limit has been exceeded.

The acquisition unit 101 may acquire the temperature distribution of the server room in which the servers 300 are arranged from an external device. The acquisition unit 101 may acquire the operating environment temperature of each server 300 based on the temperature distribution of the server room and the arrangement of the servers 300.

<<Calculating the failure rate due to operating time>>
The server 300 records and accumulates the total operation time (total power-on time) from the initial placement. The acquisition unit 101 acquires the operation time of the server 300 from the server 300 as operation information. The calculation unit 102 calculates the failure rate _Pt based on the operation time (step S103).

The failure rate _Pt due to operating time is the probability of failure calculated based on a bathtub curve of the device that indicates the lifespan characteristics of the server 300 due to the total operating time. The failure rate _Pt is set to 0 (0%) in an unused state, and the upper limit of operating time is set to 1 (100%).

<<Calculation and acquisition of individual failure rate P>>
The calculation unit 102 calculates the individual failure rate P by comprehensively considering the failure rate P _h due to hardware, the failure rate P _e due to the operating environment, and the failure rate P _t due to the operating time calculated for each server 300 (step S104). The acquisition unit 101 acquires the individual failure rate P calculated by the calculation unit 102 (step S105).

The order of steps S101 to S103 can be changed.

Calculation example 1 of individual failure rate P: The calculation unit 102 may prioritize or weight the three failure rates P _h , P _e , and P _t . The individual failure rate P is expressed by, for example, the following formula.
P = W _h × P _h + W _e × P _e + W _t × P _t (weighting constants W _h , W _e , W _t > 0)
Calculation example 2 of individual failure rate P: The calculation unit 102 may set the weight W _e of the failure rate P _e to be small when the failure rates P _h and P _t increase to a predetermined value. In other words, for a server with a high hardware-related failure rate P _h and a high operating time-related failure rate P _t , the individual failure rate P may be calculated to be high even if the operating environment-related failure rate P _e is low. This allows the server to be monitored at a short monitoring interval.

<<Determining monitoring interval>>
The determining unit 103 determines the monitoring interval for the server 300 based on the relational model between the individual failure rate of the server 300 and the monitoring interval (step S106). Fig. 4 is a graph showing an example of the relational model. In Fig. 4, the relational model between the individual failure rate P and the monitoring interval I is represented by a straight line, but it may be a curved line. Also, in Fig. 4, the monitoring interval I is set to a continuous value as the individual failure rate increases, but the monitoring interval I may be discontinuous (discrete). The monitoring interval I may be represented by a different relational equation depending on the range of the individual failure rate P.

The closer the individual failure rate P is to 100, the shorter the monitoring interval I may be set than the normal monitoring interval of a typical monitoring device. Also, the closer the individual failure rate P is to 0, the longer the monitoring interval I may be set than the normal monitoring interval of a typical monitoring device.

The monitoring device 100 monitors the server 300 at a monitoring interval determined for each server 300.

(effect)
According to the first embodiment, monitoring can be controlled in consideration of the operating environment of the server 300. This is because the acquisition unit 101 of the monitoring device 100 acquires the individual failure rate caused by the hardware, operating time, and operating environment of each server, and the determination unit 103 determines the monitoring interval for the server 300 based on the individual failure rate.

In system maintenance management, many electronic devices are monitored using one monitoring device. The larger the system, the more electronic devices must be monitored. Furthermore, each time a monitoring item is added, the number of accesses increases by the number of electronic devices, which increases the network load and the load on the monitoring device. For this reason, in large systems, the monitoring interval is uniformly lengthened to reduce the network load and the load on the monitoring device.

However, if the monitoring interval is long, it will take a long time for the monitoring device to detect a failure after it occurs. If failures cannot be detected immediately, this will reduce the overall system availability and even lead to a deterioration in service quality.

According to the first embodiment, the determination unit 103 determines the monitoring interval for the server 300 whose individual failure rate P is lower than the predetermined failure rate standard to be longer than the predetermined monitoring interval standard, thereby reducing the load on the network 010 when monitoring many servers and reducing the monitoring load.

Furthermore, according to the first embodiment, the determination unit 103 sets the monitoring interval of the server 300 whose individual failure rate P is higher than the predetermined failure rate standard to be shorter than the predetermined monitoring interval standard, thereby shortening the time from the actual occurrence of a failure to the monitoring device 100 detecting the failure, and preventing delays in detecting the failure.

(Modification)
5 is a graph showing another example of the relational model, in which the relational model is expressed by different relational expressions depending on whether the individual failure rate P is smaller than the reference failure rate _P0 or equal to or greater than the reference failure rate _P0 .

When P≦ _P0 , I=I _A
When 0<P< _P0 , I= _IB
5, for example, the monitoring interval I _A may be set to 2 to 3 minutes, and the monitoring interval I _B may be set to 1 hour. The determination unit 103 may set the reference failure rate P ₀ to, for example, the average or median of the individual failure rates of the servers 300_1, 300_2, ..., 300_x of each server rack 200 in a predetermined period. The determination unit 103 may set the reference failure rate P ₀ using the above-mentioned machine learning.

[Second embodiment]
In the first embodiment, a case has been described in which the monitoring device 100 includes a calculation unit that calculates the individual failure rate P. In the second embodiment, a case will be described in which another device has the function of the calculation unit 102 according to the first embodiment, and the monitoring device 100 acquires the individual failure rate P calculated by the calculation unit. Descriptions similar to those in the first embodiment will be omitted in the description of the second embodiment.

(composition)
6 is a block diagram showing the configuration of a monitoring device 100 according to the second embodiment. The monitoring device 100 according to the second embodiment includes an acquisition unit 121 and a determination unit 122. The monitoring device 100 according to the second embodiment can be substituted for the monitoring device 100 in FIG.

Figure 7 is a schematic diagram of a maintenance management system according to the second embodiment, showing an example of the arrangement of the calculation device 120. As shown in Figure 7, one probability calculation device 120 may be communicatively connected to the monitoring device 100, each server 300 in each server rack 200, and a temperature sensor 400. The calculation device 120 acquires hardware monitoring information and operation information from each server 300, and acquires information about the operating environment from the temperature sensor 400.

The acquisition unit 121 of the monitoring device 100 acquires the individual failure rate based on the hardware, operation information, and operating environment of each server 300 from the external calculation device 120.

The determination unit 122 determines the monitoring interval for each server 300 based on the individual failure rate of the server.

The determination unit 122 may determine the monitoring interval of the server 300 to be a first monitoring interval if the individual failure rate is higher than a predetermined reference failure rate, and may determine the monitoring interval of the server 300 to be a second monitoring interval longer than the first monitoring interval if the individual failure rate is lower than the reference failure rate. If the individual failure rate is equal to the reference failure rate, the determination unit 122 determines the monitoring interval to be an arbitrary time equal to or greater than the first monitoring interval and equal to or less than the second monitoring interval.

The relationship between the individual failure rate and the monitoring interval determined by the determination unit 122 may be shown by the graph of Fig. 4 or Fig. 5. Fig. 8 is a graph showing another example of the relationship between the individual failure rate and the determined monitoring interval. In Fig. 8, monitoring intervals _I1 to _I5 are determined corresponding to reference failure rates _P1 to _P4 . Each of the reference failure rates _P1 to _P4 is an embodiment of the predetermined reference failure rate. Each of the monitoring intervals _I1 to _I5 is an embodiment of the first or second monitoring interval.

The monitoring device 100 monitors the server 300 at the determined monitoring interval. Alternatively, the monitoring device 100 transmits the determined monitoring interval to another monitoring process execution device (not shown) and causes the monitoring process execution device to monitor the server 300.

In FIG. 7, one calculation device 120 is arranged, but one calculation device 120 may be arranged for each server rack 200. In this case, the calculation device 120 calculates the individual failure rate of each server 300 in the server rack 200. Alternatively, the functions of the calculation device 120 may be provided in each server 300. In this case, each server 300 obtains information about the operating environment from the temperature sensor 400 and calculates the individual failure rate of its own device. The monitoring device 100 obtains the individual failure rate of each server 300 from each of the multiple calculation devices 120.

(Operation)
9 is a flowchart showing an example of the operation of the monitoring device 100 according to the second embodiment. The acquisition unit 121 acquires the individual failure rate caused by the hardware, the operating time, and the operating environment of each server 300 (step S121). The determination unit 122 determines the monitoring interval for each server 300 based on the individual failure rate of each server 300 (step S122).

(effect)
According to the second embodiment, monitoring can be controlled in consideration of the operating environment of the server 300. This is because the acquisition unit 121 of the monitoring device 100 acquires the individual failure rate caused by the hardware, operating time, and operating environment of each server, and the determination unit 122 determines the monitoring interval for the server 300 based on the individual failure rate.

(Modification)
The determination unit 122 may determine the monitoring interval for the servers based on the ratio of the number of servers 300 for each monitoring interval or the upper limit number in addition to the individual failure rate. That is, the determination unit 122 determines the reference failure rate so that, for example, when the number of servers 300 monitored at a first monitoring interval reaches a predetermined upper limit number, the remaining servers 300 are monitored at a longer second monitoring interval.

This modified example can reduce the network load when monitoring many servers, thereby meeting both the demand to reduce the monitoring load and the demand to shorten the time from when an actual failure occurs until the monitoring device 100 detects the failure, thereby preventing delays in failure detection.

[Hardware configuration]
In each of the above-described embodiments, each component of the monitoring device 100 is represented by a functional block. A part or all of each component of each device may be realized by any combination of the computer 500 and a program.

FIG. 10 is a block diagram showing an example of the hardware configuration of a computer 500. Referring to FIG. 10, the computer 500 includes, for example, a CPU (Central Processing Unit) 501, a ROM (Read Only Memory) 502, a RAM (Random Access Memory) 503, a program 504, a storage device 505, a drive device 507, a communication interface 508, an input device 509, an input/output interface 511, and a bus 512.

The program 504 includes instructions for implementing each function of each device. The program 504 is stored in advance in the ROM 502, the RAM 503, or the storage device 505. The CPU 501 implements each function of each device by executing the instructions included in the program 504. For example, the CPU 501 of the monitoring device 100 implements the functions of the monitoring device 100 by executing the instructions included in the program 504. The RAM 503 may also store data to be processed in each function of each device. For example, the RAM 503 of the computer 500 may store hardware monitoring information and operating environment temperature used by the calculation unit 102 of the monitoring device 100.

The drive device 507 reads and writes data from the recording medium 506. The communication interface 508 provides an interface with the communication network. The input device 509 is, for example, a mouse or keyboard, and accepts information input from the administrator of the maintenance management system. The output device 510 is, for example, a display, and outputs (displays) information to the administrator. The input/output interface 511 provides an interface with peripheral devices. The bus 512 connects these hardware components. The program 504 may be supplied to the CPU 501 via the communication network, or may be stored in advance on the recording medium 506, read by the drive device 507, and supplied to the CPU 501.

Note that the hardware configuration shown in FIG. 10 is an example, and other components may be added, or some components may not be included.

There are various variations in the way each device can be realized. For example, each device may be realized by any combination of a different computer and program for each component. Also, multiple components of each device may be realized by any combination of a single computer and program.

In addition, some or all of the components of each device may be realized by general-purpose or dedicated circuits including a processor, etc., or a combination of these. These circuits may be configured by a single chip, or may be configured by multiple chips connected via a bus. Some or all of the components of each device may be realized by a combination of the above-mentioned circuits, etc., and a program.

In addition, when some or all of the components of each device are realized by multiple computers, circuits, etc., the multiple computers, circuits, etc. may be centralized or distributed.

In addition, at least a part of the monitoring device 100 may be provided in a SaaS (Software as a Service) format. That is, at least a part of the functions for realizing the monitoring device 100 may be executed by software executed via a network.

Although the present disclosure has been described above with reference to the embodiments, the present disclosure is not limited to the above-mentioned embodiments. Various modifications that can be understood by a person skilled in the art can be made to the configuration and details of the present disclosure within the scope of the present disclosure. Furthermore, the configurations in each embodiment can be combined with each other as long as they do not deviate from the scope of the present disclosure.

REFERENCE SIGNS LIST 100 Monitoring device 101, 121 Acquisition unit 102 Calculation unit 103, 122 Determination unit 120 Calculation device 200 Server rack 300 Server 400 Temperature sensor 500 Computer

Claims (7)

An acquisition means for acquiring an individual failure rate caused by hardware, operating time, and operating environment of each server;
a determination means for determining a monitoring interval for each of the servers based on a relational model showing a decreasing trend of the monitoring interval corresponding to an increase in the individual failure rate;
A monitoring device comprising: the acquiring means acquires the individual failure rate based on hardware monitoring information, operating time, and operating environment temperature of the server in the server rack for each of the servers;
The monitoring device according to claim 1. a calculation unit for calculating the individual failure rate based on the hardware monitoring information for each server, an operating time, and an operating environment temperature of the server;
3. A monitoring device according to claim 1 or 2. The determining means is
If the individual failure rate is higher than a predetermined reference failure rate, a monitoring interval of the server is determined to be a first monitoring interval;
If the individual failure rate is lower than the reference failure rate, determining a monitoring interval of the server to be a second monitoring interval that is longer than the first monitoring interval.
A monitoring device according to any one of claims 1 to 3 . The monitoring device according to claim 1 , wherein the determining unit determines the monitoring interval based on an upper limit number of the servers to be monitored during the monitoring interval. The monitoring device,
Obtain the individual failure rate due to the hardware, operating time, and operating environment of each server,
determining a monitoring interval for each of the servers based on a relational model showing a decreasing trend of the monitoring interval corresponding to an increase in the individual failure rate;
Monitoring methods. Obtain the individual failure rate due to the hardware, operating time, and operating environment of each server,
determining a monitoring interval for each of the servers based on a relational model showing a decreasing trend of the monitoring interval corresponding to an increase in the individual failure rate;
A program that causes a computer to carry out processing.

Images