JP4599435B2 - Computer and program constituting cluster system - Google Patents

Description

  The present invention relates to a computer constituting a cluster system, and more particularly to a computer and a program constituting a cluster system suitable for the computer itself to determine the occurrence of split brain due to slowdown of the computer.

  Conventionally, a cluster system in which a plurality of computers are connected to each other is known. In this type of cluster system, when a failure occurs in any of a plurality of computers (nodes), any other node takes over the service for the client that was executed on the failed node (cluster node). Failover occurs. For this reason, in a cluster system, it is important to detect a failure of a node in the system.

  In a conventional cluster system, detection of a failure of a node in the system is performed by monitoring the state between nodes using a connection system path as described in Non-Patent Document 1, for example. For this state monitoring, each node in the cluster system periodically transmits information (state notification packet) for notifying its own state to other nodes via the connection path. This periodic information transmission is called heartbeat (or heartbeat transmission). Each node monitors heartbeats from other nodes.

Here, it is assumed that a certain node (hereinafter referred to as a first node) detects that a heartbeat from another node (hereinafter referred to as a second node) has been interrupted. In this case, the first node determines that a failure has occurred in the second node. Then, the first node performs failover to take over the service that was being executed by the second node.
Tetsuo Kaneko, 1 other, “Cluster Software”, Toshiba Review, 1999, Vol.54, No.12, p.18-21

  In the above prior art, even when the heartbeat disruption of the second node is caused by a transient failure such as slowdown, the first node is independent of the state of the second node. It is determined that a failure has occurred on the node. In such a case, although the service is executed on the second node, the service is started on the first node due to the failover, and so-called split brain occurs.

  Therefore, it is conceivable to determine the operating state of the service including the split brain by exclusively using the shared resources accessible from all the nodes in the cluster system based on the ownership. However, this requires a shared resource such as a shared disk device.

  The present invention has been made in view of the above circumstances, and an object of the present invention is to provide a computer and a program constituting a cluster system that can determine by itself the occurrence of split brain caused by its own slowdown.

  According to one aspect of the present invention, a computer constituting a cluster system is provided. The computer performs heartbeat transmission at a predetermined time interval for notifying other computers constituting the cluster system of the state of the computer, and monitors reception of heartbeats from the other computers. When the heartbeat reception from the other computer is interrupted beyond a predetermined heartbeat timeout period, it is possible to determine the failure of the other computer and continue the service being executed on the computer of the other computer A cluster control mechanism for performing failover, a time measuring means for measuring time or date and time, monitoring the heartbeat transmission to the other computer by the cluster control mechanism, and measuring the time interval of the heartbeat transmission. The time interval of the measured heartbeat transmission is the heartbeat timeout time. And it determines that the split brain occurs conditions due to the slow down of the computer when it exceeds comprises a heartbeat monitoring mechanism for forcibly stopping the computer.

  According to the present invention, the heartbeat monitoring mechanism of the computers constituting the cluster system is configured to transmit the heartbeat transmission time interval to other computers (other nodes) in the cluster system by the cluster control mechanism of the computer (own node). When the measured transmission time interval exceeds the heartbeat timeout time, it is determined that a split brain has occurred due to the slowdown of the computer itself, and the computer is forced Therefore, the split brain occurrence state can be quickly resolved.

Embodiments of the present invention will be described below with reference to the drawings.
FIG. 1 is a block diagram showing a hardware configuration of a cluster system according to an embodiment of the present invention. This cluster system is composed of, for example, two computers (hereinafter referred to as nodes) 10-1 and 10-2. However, the cluster system may be composed of three or more nodes (cluster nodes).

  Nodes 10-1 and 10-2 include CPUs 11-1 and 11-2, main memories 12-1 and 12-2, external storage devices 13-1 and 13-2, communication mechanisms 14-1 and 14-2. The communication mechanisms 14-1 and 14-2 are connected to the communication path 20. That is, the nodes 10-1 and 10-2 are interconnected by the communication path 20. Although omitted in FIG. 1, the node 10-1 has a real-time clock 230 (see FIG. 3) described later. Similarly, the node 10-2 has a real time clock.

  The external storage devices 13-1 and 13-2 respectively have disks 130-1 and 130-2 as storage media, and the disks 130-1 and 130-2 each have an operating system (OS). 131-1 and 131-2 and programs 132-1 and 132-2 are stored. In this embodiment, the program 132-i (i = 1, 2) is stored in the cluster control mechanism 210-i (see FIG. 2) and the heartbeat monitoring mechanism 220-i (see FIG. 2) described later on the node 10-i. Used to give functionality. This function is realized by the CPU 11-i reading the program 132-i into the main memory 13-i and executing it.

  The program 132-i may be stored in a storage medium other than the disk 130-i, for example, a storage medium such as a nonvolatile memory. The program 132-i may be downloaded to the external storage device 13-i via a network.

FIG. 2 is a block diagram mainly showing a functional configuration of the cluster system having the hardware configuration of FIG.
The nodes 10-1 and 10-2 have OSs 131-1 and 131-2, communication mechanisms 14-1 and 14-2, and cluster control mechanisms 210-1 and 210-2, respectively.

  The cluster control mechanisms 210-1 and 210-2 are generally called cluster kernels and perform well-known cluster control including failover. The cluster control mechanisms 210-1 and 210-2 respectively include main control units 211-1 and 211-2, heartbeat transmission units (HB transmission units) 212-1 and 212-2, and heartbeat reception units (HB). Receiving section) 213-1 and 213-2.

  When the node 10-i including the HB transmission unit 212-i is in a normal state, the HB transmission unit 212-i (i = 1, 2) sends a heartbeat (HB) as a state notification packet to other nodes. Then, transmission is performed at a predetermined time interval Thb (that is, periodically) via the communication mechanism 14-i. The HB receiving unit 213-i receives a heartbeat transmitted from another node via the communication mechanism 14-i.

  The main control unit 211-i forms the center of the cluster control mechanism 210-i. The main control unit 211-i detects a failure of another node based on the heartbeat reception status by the HB reception unit 213-i and performs a failover. Here, in the case of a so-called heartbeat timeout in which a heartbeat is not received over a certain period (heartbeat timeout time Tout), the failure of the other node is detected (determined).

The OSs 131-1 and 131-2 include heartbeat monitoring mechanisms 220-1 and 220-2, respectively.
The heartbeat monitoring mechanism 220-i (i = 1, 2) is located between the HB transmission unit 212-i and the communication mechanism 14-i, and the heartbeat transmission (heartbeat monitoring mechanism) by the HB transmission unit 212-i. Heartbeat transmission by node 10-i including 220-i) is monitored. The heartbeat monitoring mechanism 220-i measures (calculates) a time interval (elapsed time from the preceding heartbeat transmission) with respect to the preceding heartbeat transmission for each heartbeat transmission by the HB transmission unit 212-i. If the heartbeat monitoring mechanism 220-i or OS131-i temporarily stops during the period from the preceding heartbeat transmission to the current heartbeat transmission, the measured time interval exceeds the heartbeat timeout time Tout. there is a possibility.

  The heartbeat monitoring mechanism 220-i determines the occurrence of split brain due to the slowdown of the node 10-i when the measured time interval exceeds the heartbeat timeout time Tout. In this case, the heartbeat monitoring mechanism 220-i forcibly stops the node 10-i (the node 10-i including the heartbeat monitoring mechanism 220-i) in order to maintain consistency as a cluster system.

  Here, the principle of determination of split brain occurrence by the heartbeat monitoring mechanism 220-i will be described. Here, it is assumed that the heartbeat monitoring mechanism 220-i is the heartbeat monitoring mechanism 220-1 (i = 1). The heartbeat monitoring mechanism 220-1 acquires time information for each heartbeat transmission from the HB transmission unit 212-1, and holds the time information until at least the next heartbeat transmission is performed. The heartbeat monitoring mechanism 220-1 measures (calculates) the elapsed time from the previous heartbeat transmission to the current heartbeat transmission (that is, the time interval of heartbeat transmission) based on the time information held by itself. It is checked whether the elapsed time exceeds a predetermined threshold value. Here, the heartbeat timeout time Tout used in the cluster control mechanisms 210-1 and 10-2 is used as the threshold.

  Heartbeat transmission by the HB transmission unit 212-1 is always performed at a constant time interval Thb shorter than the heartbeat timeout time Tout as long as the node 10-1 is operating normally. Therefore, when the heartbeat transmission time interval measured (calculated) by the heartbeat monitoring mechanism 220-1 exceeds the predetermined time interval Thb, the node 10-1 (OS 131-1 or cluster control mechanism) for some reason. It is predicted that a slowdown of 210-1) has occurred. In particular, when the heartbeat transmission time interval exceeds the heartbeat timeout time (threshold value) Tout, the cluster of the node 10-2 serving as a heartbeat transmission partner by the node 10-1 (the HB transmission unit 212-1) The control mechanism 210-2 also detects that heartbeat transmission from the node 10-1 (internal HB transmission unit 212-1) has been interrupted beyond the heartbeat timeout time Tout. In this case, the cluster control mechanism 210-2 determines a failure (heartbeat timeout) of the node 10-1 and starts failover.

  Therefore, when the heartbeat transmission time interval from the HB transmission unit 212-1 exceeds the heartbeat timeout time Tout, the heartbeat monitoring mechanism 220-1 has failed over the node 10-2 that is already the heartbeat transmission partner. Has been started, and it is determined that a split-brain occurrence state has been reached. That is, the heartbeat monitoring mechanism 220-1 in the node 10-1 monitors the time interval of heartbeat transmission from the HB transmission unit 212-1 in the node 10-1, thereby The occurrence of split brain due to slowdown can be determined (detected) independently from the counterpart node 10-2. In this case, the heartbeat monitoring mechanism 220-1 forcibly stops the node 10-1 as described above.

  FIG. 3 is a block diagram showing the configuration of mainly the heartbeat monitoring mechanism 220-1 of the node 10-1. The node 10-1 has a real time clock 230. The real time clock 230 is a hardware time measuring means for measuring the date and time. The date and time (time) indicated by the real-time clock 230 is synchronized with the system clock 232 when the date and time of the system clock 232 described later is operated (changed) by the OS 131-1 based on an instruction from the user, for example. Operated (changed) by -1.

  The OS 131-1 of the node 10-1 includes a time counter (hereinafter referred to as a “tick counter”) 231 and a system clock (system clock) 232 in addition to the heartbeat monitoring mechanism 220-1. The tick counter 231 is a software counter (timer) that counts the elapsed time from the startup of the OS 131-1. The tick counter 231 is updated at regular time intervals by the OS 131-1. The system clock 232 is a software timer (timer) that measures the date and time (time) managed by the OS 131-1.

The heartbeat monitoring mechanism 220-1 includes a time information storage unit 221, a heartbeat detection unit 222, a failure determination processing unit 223, and a forced stop processing unit 224.
The time information storage unit 221 is realized by using a part of the storage area of the main memory 12-1 shown in FIG. The time information storage unit 221 stores time (time) information indicating the time (time) at the time when the heartbeat was transmitted from the previous HB transmission unit 212-1 (at least the heartbeat from the HB transmission unit 212-1 next. Used to store (until it is sent). Here, as the time information stored in the time information storage unit 221, the count value T of the Tick counter 231 (elapsed time since the startup of the OS 131-1), the value (date and time) S indicated by the system clock 232, and the real time clock A value (date and time) R indicated by 230 is used. The time information storage unit 221 is initialized when the OS 131-1 is activated.

  The heartbeat detection unit 222 detects heartbeat transmission from the HB transmission unit 212-1. When a new heartbeat transmission from the HB transmission unit 212-1 is detected by the heartbeat detection unit 222, the failure determination processing unit 223 receives a time counter (time) information from the OS 131-1 as a time counter 231. Count value (Tick count) Tc and the value Sc of the system clock 232 are acquired. In addition, the failure determination processing unit 223 acquires the value Rc of the real time clock 230. At this time, the time information storage unit 221 stores the count value Tp of the Tick counter 231 acquired at that time and the system clock 232 as time (time) information at the time of the previous heartbeat transmission from the HB transmission unit 212-1. Value Sp and real-time clock 230 value Rp are stored. The failure determination processing unit 223 uses the newly acquired time information (Tc, Sc, Rc) and the time information (Tp, Sp, Rp) acquired at the previous heartbeat transmission to slow down the node 10-1. Determine the state of occurrence of split brain. The determination mechanism will be described later.

The forced stop processing unit 224 performs a forced stop processing for forcibly stopping the node 10-1 when the failure determination processing unit 223 determines that a split brain has occurred. In this forced stop process, for example, an operation to stop the OS 131-1 (OS stop operation) 223a or an operation to shut off the power supply of the node 10-1 (power OFF operation) 223b is performed.
It is assumed that the heartbeat monitoring mechanism 220-2 has the same configuration as the heartbeat monitoring mechanism 220-1 shown in FIG. In addition, it is assumed that the real time clock 230 is provided in the node 10-2, and the tick counter 231 and the system clock 232 are provided in the OS 131-2 of the node 10-2.

  Next, details of the operation of the heartbeat monitoring mechanism 220-1 will be described with reference to the flowchart of FIG.

  When the OS 131-1 is activated, the heartbeat monitoring mechanism 220-1 is also activated. Then, the heartbeat detector 222 of the heartbeat monitoring mechanism 220-1 turns off (resets) the flag F (not shown) (step S1). This flag F indicates whether valid time information (Tp, Sp, Rp) is stored in the time information storage unit 221.

  Next, the heartbeat detection unit 222 monitors heartbeat transmission from the HB transmission unit 212-1 (step S2). It is assumed that the heartbeat detection unit 222 has detected heartbeat transmission from the HB transmission unit 212-1 through this monitoring (step S3). Then, the failure determination processing unit 223 of the heartbeat monitoring mechanism 220-1 acquires the count value Tc of the Tick counter 231, the value Sc of the system clock 232, and the value Rc of the real-time clock 230 as time (time) information at that time. (Step S4).

  Next, the failure determination processing unit 223 determines whether the flag F is ON (set) (step S5). If the flag F is OFF (reset) (NO in step S5), the failure determination processing unit 223 indicates that the heartbeat transmission detected by the current heartbeat detection unit 222 is activated by the OS 131-1. It is the first heartbeat transmission later, and it is determined that the time information storage unit 221 does not store valid time information (Tp, Sp, Rp).

  In this case, the failure determination processing unit 223 stores the time information (Tc, Sc, Rc) acquired in step S4 in the time information storage unit 221 as time information (Tp, Sp, Rp) (step S6). As a result, as will be described later, the time information (Tc, Sc, Rc) acquired at the time of detecting the current heartbeat transmission is detected when the next heartbeat transmission is detected (when the previous heartbeat transmission is detected). Used as time information (Tp, Sp, Rp).

  When step S6 is executed by the failure determination processing unit 223, the heartbeat detection unit 222 turns on (sets) the flag F (step S7) and monitors heartbeat transmission again (step S2). Thereafter, it is assumed that the heartbeat detection unit 222 detects a new heartbeat transmission from the HB transmission unit 212-1 (step S3).

  In this case, the failure determination processing unit 223 acquires time information (Tc, Sc, Rc) at the time of detection of a new heartbeat transmission (step S4), and determines whether the flag F is turned on thereafter (step S4). S5). At this time, the time information acquired at the time of detecting the heartbeat transmission (previous heartbeat transmission) preceding the current heartbeat transmission is stored as valid time information (Tp, Sp, Rp) in the time information storage unit 221. The flag F is ON.

When the flag F is ON (YES in step S5), the failure determination processing unit 223 uses the time information storage unit 221 to obtain valid time information (Tp, Sp, Rp) acquired when the previous heartbeat transmission is detected. It is determined that it is stored in. Then, the failure determination processing unit 223 detects Tc in the latest time information (Tc, Sc, Rc) acquired in step S4 this time and the previous heartbeat transmission stored in the time information storage unit 221. Based on Tp in the acquired time information (Tp, Sp, Rp), an elapsed period (latest heartbeat transmission time interval) ΔT from the previous heartbeat transmission time to the current heartbeat transmission time is calculated ( Step S8). That is, the failure determination processing unit 223 calculates the latest heartbeat transmission time interval ΔT from the value Tc of the Tick counter 231 at the time of the current heartbeat transmission and the value Tp of the Tick counter 231 at the time of the previous heartbeat transmission by the following equation ΔT = Tc-Tp (1)
Calculate according to This heartbeat transmission time interval (first time interval) ΔT is equivalent to that measured using the Tick counter 231.

  Next, the failure determination processing unit 223 determines whether the heartbeat transmission time interval ΔT calculated using the value of the Tick counter 231 (that is, measured using the Tick counter 231) exceeds the heartbeat timeout time Tout ( (ΔT> Tout) is determined (step S9).

  When the heartbeat transmission time interval ΔT exceeds the heartbeat timeout time Tout, the time exceeding the heartbeat timeout time Tout has elapsed since the last heartbeat transmission on the OS 131-1 (time space). At least during that period, there is no heartbeat transmission from the HB transmission unit 212-1 in the cluster control mechanism 210-1. Therefore, when the heartbeat transmission time interval ΔT exceeds the heartbeat timeout time Tout (YES in step S9), the failure determination processing unit 223 determines that the cluster system is the cluster control mechanism (cluster kernel) 210-1 of the node 10-1. It is determined that there is a split brain occurrence state (hereinafter referred to as a type 2 failure) due to the failure (slowdown) (step S17).

  As described above, in this embodiment, the split brain occurrence state due to the heartbeat transmission delay in the node 10-1 can be determined not only by the node 10-1 (the heartbeat monitoring mechanism 220-1) but also the cause thereof. It can also be determined that there is a failure (slowdown) in the cluster control mechanism (cluster kernel) 210-1 of the node 10-1.

  The forcible stop processing unit 224 forcibly stops the node 10-1 (local node) including the cluster control mechanism 210-1 when the type 2 failure is determined by the failure determination processing unit 223 (step S17). The forced stop process (local node forced stop process) is performed (step S19). Here, the OS stop operation 223a or the power OFF operation 223b as described above is performed in order to maintain consistency as a cluster system. Thereby, the split brain occurrence state can be quickly resolved.

  Incidentally, the Tick counter 231 is held by the OS 131-1 and updated by the OS 131-1. For this reason, if a failure (slowdown) of the OS 131-1 occurs, the Tick counter 231 is not accurately updated during that time. That is, when the heartbeat transmission time interval ΔT measured using the Tick counter 231 does not exceed the heartbeat timeout time Tout, it actually exceeds the heartbeat timeout time Tout due to the slowdown of the OS 131-1. Time may have passed.

  For this reason, when the heartbeat transmission time interval ΔT does not exceed the heartbeat timeout time Tout, it is necessary to determine whether the heartbeat transmission time interval ΔT itself is a correct value. That is, it is necessary to determine whether the OS 131-1 has been slowed down and the heartbeat transmission time interval ΔT is affected by the slow-down.

Therefore, when the heartbeat transmission time interval ΔT does not exceed the heartbeat timeout time Tout (NO in step S9), the failure determination processing unit 223 obtains the latest time information (Tc, Sc, Rc) acquired in step S4 this time. Elapsed period from the last heartbeat transmission time to the current heartbeat transmission time (latest time) based on the Rc in the time and the Rp in the time information (Tp, Sp, Rp) acquired when the previous heartbeat transmission was detected The heartbeat transmission time interval) [Delta] R is calculated (step S10). That is, the failure determination processing unit 223 calculates the latest heartbeat transmission time interval ΔR from the value Rc of the real-time clock 230 at the time of the current heartbeat transmission and the value Rp of the real-time clock 230 at the time of the previous heartbeat transmission by the following equation: ΔR = Rc-Rp (2)
Calculate according to This heartbeat transmission time interval (second time interval) ΔR is equivalent to that measured using the real-time clock 230.

  Next, the failure determination processing unit 223 determines whether the heartbeat transmission time interval ΔR calculated using the value of the real-time clock 230 (that is, measured using the real-time clock 230) exceeds the heartbeat timeout time Tout ( (ΔR> Tout) is determined (step S11). Here, the value of the real-time clock 230 is updated at a fixed time interval at the hardware level independently of the operation of the OS 131-1. For this reason, the heartbeat transmission time interval ΔT measured using the Tick counter 231 does not exceed the heartbeat timeout time Tout, and the heartbeat transmission time interval ΔR measured using the real-time clock 230 is the heartbeat timeout. If the time Tout is exceeded, it is basically determined that the cluster system is in a split brain occurrence state (due to a heartbeat transmission delay) due to a failure (slowdown) of the OS 131-1 of the node 10-1. be able to.

  However, in order to synchronize with the system clock 232 managed by the OS 131-1, the real time clock 230 may be operated by the OS 131-1 according to an instruction from the user, for example. For this reason, an accurate heartbeat transmission time interval may not be measured with the real-time clock 230 alone. Therefore, since the real-time clock 230 is operated using the system clock 232, it is necessary to determine whether the heartbeat transmission time interval ΔR measured using the real-time clock 230 exceeds the heartbeat timeout time Tout. is there.

  The real-time clock 230 and the system clock 232 are synchronized when the OS 131-1 is activated. However, when a failure (slowdown) of the OS 131-1 occurs, the system clock 232 managed by the OS 131-1 is not updated correctly. As a result, the real-time clock 230 and the system clock 232 are in a state where the values do not match (out of synchronization).

Therefore, when the heartbeat transmission time interval ΔR measured using the real-time clock 230 exceeds the heartbeat timeout time Tout (YES in step S11), the failure determination processing unit 223 is acquired in step S4 this time. Based on the Sc in the latest time information (Tc, Sc, Rc) and the Sp in the time information (Tp, Sp, Rp) acquired when detecting the previous heartbeat transmission, An elapsed period (latest heartbeat transmission time interval) ΔS until the heartbeat transmission is calculated (step S12). That is, the failure determination processing unit 223 obtains the latest heartbeat transmission time interval ΔS from the value Sc of the system clock 232 at the time of the current heartbeat transmission and the value Sp of the system clock 232 at the time of the previous heartbeat transmission by the following equation ΔS = Sc-Sp (3)
Calculate according to This heartbeat transmission time interval (third time interval) ΔS is equivalent to that measured using the system clock 232.

  Next, the failure determination processing unit 223 determines whether the heartbeat transmission time interval ΔS measured using the system clock 232 matches the heartbeat transmission time interval ΔR measured using the real-time clock 230 (ΔS = ΔR). Is determined (step S13). The determination in step S13 is equivalent to determining whether the value of the system clock 232 matches the value of the real-time clock 230, that is, determining whether the real-time clock 230 is synchronized with the system clock 232. In step S13, when ΔS is within a certain error δ range with respect to ΔR (ΔR−δ ≦ ΔS ≦ ΔR + δ), it may be determined that ΔS matches ΔR. The process of the above steps S8 to S13 is referred to as a determination process (step S20).

  The failure determination processing unit 223 uses the real-time clock 230 when the determination in step S13 is NO, that is, the heartbeat transmission time interval ΔT measured using the tick counter 231 does not exceed the heartbeat timeout time Tout. If the measured heartbeat transmission time interval ΔR exceeds the heartbeat timeout time Tout and the heartbeat transmission time interval ΔS measured using the system clock 232 does not match the ΔR, the failure of the OS 131-1 It is determined that the ΔR (the heartbeat transmission time interval measured using the real time clock 230) exceeds the heartbeat timeout time Tout by (slow down). That is, when the determination in step S13 is NO, the failure determination processing unit 223 causes a split brain occurrence state (hereinafter referred to as a heartbeat transmission delay) caused by a failure (slowdown) of the OS 131-1 of the node 10-1. (Referred to as type 1 failure) (step S16). As described above, in this embodiment, the cause of the split brain due to the heartbeat transmission delay in the node 10-1 is that the failure of the OS 131-1 in the node 10-1 indicates that the node 10-1 ( This can be determined by the heartbeat monitoring mechanism 220-1).

  The forcible stop processing unit 224 determines whether the type 1 failure is determined by the failure determination processing unit 223 (step S16) or the node 10-1 as in the case where the type 2 failure is determined (step S17). A forced stop process for forcibly stopping is performed (step S19). As a result, the split brain occurrence state can be quickly resolved.

  On the other hand, if the determination in step S13 is YES, that is, if the system clock 232 and the real-time clock 230 are synchronized, the failure determination processing unit 223 determines that the system clock 232 has been updated normally, and thus the OS 131 -1 failure (slowdown) has not occurred, and the heartbeat transmission time interval ΔR measured using the real-time clock 230 exceeds the heartbeat timeout time Tout. This is because the date / time is operated by the OS 131-1 using the H.232, and it is determined that the node 10-1 (including the cluster system) is in a normal state (hereinafter referred to as a type 2 normal state) (step S15). Thereby, erroneous detection of the split brain occurrence state can be prevented.

  On the other hand, if the determination in step S11 is NO, that is, the heartbeat transmission time interval ΔT measured using the Tick counter 231 and the heartbeat transmission time interval ΔR measured using the real-time clock 230 are both heartbeat timeouts. If the time Tout has not been exceeded, the failure determination processing unit 223 indicates that neither the cluster control mechanism 210-1 nor the OS 131-1 has failed (slow down), and the node 10-1 (including the cluster system). Is in a normal state (hereinafter referred to as a type 1 normal state) (step S14).

If the failure determination processing unit 223 determines that the node 10-1 (including the cluster system) is in the normal state of type 1 or type 2 (step S14 or S15), the time information (Tc) acquired at step S4 this time , Sc, Rc) as new time information (Tp, Sp, Rp), and the new time information (Tp, Sp, Rp) and the time information (Tp, Sp stored in the current time information storage unit 221) , Rp) is updated (step S18). Then, the heartbeat detector 222 monitors heartbeat transmission again (step S2).
The heartbeat monitoring mechanism 220-2 in the node 10-2 also performs the same operation as that of the heartbeat monitoring mechanism 220-1.

[First Modification]
Next, a first modification of the above embodiment will be described.
The first feature of the first modification is that when the normal state of type 1 or type 2 is determined in the determination process (step S20) in the above embodiment, the same determination process as the determination process (step S20) is performed. The threshold value is executed based on a threshold value (hereinafter referred to as a slowdown detection time) Tsd instead of the heartbeat timeout time Tout. The slowdown detection time Tsd is longer than the heartbeat transmission time interval Thb from the HB transmission unit 212-1 when the node 10-1 is normal and shorter than the heartbeat timeout time Tout (Thb <Tsd <Tout). ). By such a determination process based on the slowdown detection time Tsd, a failure (slowdown) of the cluster control mechanism 210-1 or the OS 131-1 that does not reach the split brain occurrence state, that is, a slight slowdown of the node 10-1. Down is determined (detected). In the first modified example, the slowdown detection time Tsd is obtained from the client terminal connected to the nodes 10-1 and 10-2 via the communication path 20 or a communication path (network) different from the communication path 20. It shall be set according to the operation.

  A second feature of the first modification is that when the determination process based on the slow-down detection time Tsd determines that the slow-down of the node 10-1 does not reach the split brain occurrence state, the slow-down state Is notified to the user for warning.

  Hereinafter, the operation of the heartbeat monitoring mechanism 220-1 in the first modification of the above embodiment will be described with reference to the flowchart of FIG. 5 focusing on differences from the above embodiment. In FIG. 5, the same processing steps as those in FIG. 4 are denoted by the same reference numerals.

  First, it is assumed that the normal state of type 1 or type 2 is determined in the determination process (step S20) including steps S8 to S13 (step S14 or S15). In this case, in the above embodiment, step S18 is executed. On the other hand, in the first modified example, the following determination process corresponding to the determination process in step S20 is performed. The feature of the determination process executed here is that the slowdown detection time Tsd (Thb <Tsd <Tout) is used as a threshold corresponding to the heartbeat timeout time Tout.

  First, the failure determination processing unit 223 determines whether the heartbeat transmission time interval ΔT measured using the tick counter 231 exceeds the slowdown detection time Tsd (ΔT> Tsd) (step S21). At this time, the heartbeat transmission time interval ΔT does not exceed the heartbeat timeout time Tout. Therefore, when the heartbeat transmission time interval ΔT does not exceed the heartbeat timeout time Tout but exceeds the slowdown detection time Tsd (YES in step S21), the failure determination processing unit 223 uses the cluster control mechanism (cluster It is determined that this is a minor failure (slow down) of (kernel) 210-1 (step S27). This failure is referred to as type 4 failure.

  On the other hand, when the heartbeat transmission time interval ΔT does not exceed the slowdown detection time Tsd (NO in step S21), the failure determination processing unit 223 may be a minor failure of the OS 131-1. Then, it is determined whether the heartbeat transmission time interval ΔR measured using the real-time clock 230 exceeds the slowdown detection time Tsd (ΔR> Tsd) (step S22). If the heartbeat transmission time interval ΔR exceeds the slowdown detection time Tsd (YES in step S22), it is possible that the real-time clock 230 has been operated on the date and time using the system clock 232. The processing unit 223 determines whether the heartbeat transmission time interval ΔS measured using the system clock 232 matches the heartbeat transmission time interval ΔR measured using the real-time clock 230 (ΔS = ΔR) ( Step S23). In step S23, when ΔS is within a certain error δ range with respect to ΔR (ΔR−δ ≦ ΔS ≦ ΔR + δ), it may be determined that ΔS matches ΔR.

  If the heartbeat transmission time interval ΔS does not coincide with the heartbeat transmission time interval ΔR (step S23 is NO), the failure determination processing unit 223 determines that the OS 131-1 has a slight failure (slow down) ( Step S26). This failure is referred to as type 3 failure.

  When the failure determination processing unit 223 determines a type 3 or type 4 failure (slowdown) (step S26 or S27), the forced stop processing unit 224 determines the determined slowdown state (in the node 10-1). The cluster control mechanism 210-1 or OS 131-1 slowdown) is notified to the user via the client terminal before the cluster system including the node 10-1 reaches the split brain occurrence state (step S28). Here, the forced stop processing unit 224 notifies the client terminal used by the user of the slowdown via the communication mechanism 14-1, thereby performing the slowdown notification to the user.

As described above, in the first modification, when it is determined that the node 10-1 (the cluster control mechanism 210-1 and the OS 131-1 included therein) is in a normal state (normal state of type 1 or type 2). However, the heartbeat monitoring mechanism 220-1 (the failure determination processing unit 223) has a user-specified threshold (slow down) of a determination condition looser than the threshold (heartbeat timeout time Tout) used for the normal state determination condition. By performing the same determination process as in step S20 using the detection time Tsd), the cluster control mechanism 210-1 or OS 131-1 slowdown (type 3 or type 4 failure) that does not reach the split brain occurrence state. ) Can be determined (detected). That is, according to the first modification, the heartbeat monitoring mechanism 220-1 may cause the cluster system to enter a split brain state due to the slowdown of the node 10-1 including the heartbeat monitoring mechanism 220-1. Can be predicted.
The heartbeat monitoring mechanism 220-2 in the node 10-2 also performs the same operation as that of the heartbeat monitoring mechanism 220-1.

[Second Modification]
Next, a second modification of the above embodiment will be described with reference to FIG.
FIG. 6 is a block diagram mainly showing a functional configuration of the cluster system in the second modification of the embodiment. The nodes 10-1 and 10-2 in the cluster system shown in FIG. 6 are different from the above embodiment in that heartbeat monitoring mechanisms are used instead of the heartbeat monitoring mechanisms 220-1 and 220-2 shown in FIG. 240-1 and 240-2 are used.

  The heartbeat monitoring mechanisms 240-1 and 240-2 of the nodes 10-1 and 10-2 have the same configuration as the heartbeat monitoring mechanism 220-1 shown in FIG. That is, the heartbeat monitoring mechanisms 240-1 and 240-2 of the nodes 10-1 and 10-2 are similar to the heartbeat monitoring mechanisms 220-1 and 220-2 in the above-described embodiment. It has a function of judging its own fault. The heartbeat monitoring mechanisms 240-1 and 240-2 further have first and second functions not provided in the heartbeat monitoring mechanisms 220-1 and 220-2, respectively.

Hereinafter, the first function and the second function will be described by taking the first function of the heartbeat monitoring mechanism 240-1 and the second function of the heartbeat monitoring mechanism 240-2 as examples.
When the heartbeat monitoring mechanism 240-1 of the node 10-1 detects the heartbeat (HB) transmitted from the HB transmission unit 212-1 of the cluster control mechanism 210-1, the value of the Tick counter 231 at that time is detected. Tc, the value Sc of the system clock 232, and the value Rc of the real-time clock 230, that is, time information (Tc, Sc, Rc) are added to the detected heartbeat (first function). The time information (Tc, Sc, Rc) is added by the heartbeat monitoring mechanism 240-1, and the heartbeat is transmitted from the communication mechanism 14-1 to the node 10-2 via the communication path 20, and is sent to the node 10-2. It is received by the communication mechanism 14-2.

  The heartbeat monitoring mechanism 240-2 of the node 10-2 monitors the heartbeat from the node 10-1 received by the communication mechanism 14-2. Time information (Tc, Sc, Rc) is added to the heartbeat.

  When the heartbeat monitoring mechanism 240-2 detects a heartbeat transmitted from the node 10-1, the heartbeat monitoring mechanism 240-2 performs a determination process performed when the heartbeat transmitted from the node 10-1 is detected in the first modified example ( A determination process (determination process at heartbeat reception) similar to the determination process at heartbeat transmission is performed. However, in the heartbeat reception determination process by the heartbeat monitoring mechanism 240-2, the slowdown state (type 3 or type 4 failure) of the heartbeat transmitting node 10-1 (that is, another node) is determined. For the determination by the heartbeat monitoring mechanism 240-2, time information (Tc, Sc, Rc) added to the detected heartbeat is used.

  That is, the heartbeat monitoring mechanism 240-2 uses the time information (Tc, Sc, Rc) added to the heartbeat when the heartbeat was detected last time as time information (Tp, Sp, Rp) as a time information storage unit. (Corresponding to the time information storage unit 221 shown in FIG. 3), this time information (Tp, Sp, Rp) and the time information (Tc, Sc, Rc) added to the heartbeat detected this time Based on the above, a slight slow-down state (type 3 or type 4 failure) of the node 10-1 (other node) is determined (second function). When the heartbeat monitoring mechanism 240-2 determines the type 3 or type 4 failure of the node 10-1 (other node), the heartbeat monitoring mechanism 240-2 (cluster control mechanism 210-1 or OS 131-1 in the node 10-1 (other node)). (S) Notify the user of a slight slowdown via the client terminal.

  As described above, according to the second modification, the heartbeat monitoring mechanism 240-1 adds the time information to the heartbeat transmitted from the node 10-1 to the node 10-2 and transmits it. The second heartbeat monitoring mechanism 240-2 uses the time information added to the heartbeat to determine the slowdown state (type) of the node 10-1 (other node) by the same determination process as when the heartbeat is transmitted. 3 or type 4 failure) and can notify the user.

  Since the node 10-1 (another node) has made a determination in the heartbeat transmission mechanism determination process of the heartbeat monitoring mechanism 240-1, it is normally thrown in the hardbeat reception determination process of the heartbeat monitoring mechanism 240-2. Although it is not determined as a down state (type 1 or type 2 failure), it is longer than the heartbeat timeout time Thb due to a transient failure of the communication path 20, for example, other than the node failure of the node 10-1 (other node). If the heartbeat monitoring mechanism 240-2 has not received a heartbeat because no inter-heartbeat has arrived, it is determined as a slow-down state (type 1 or type 2 failure) of the node 10-1 (another node) . In such a case, a forced stop process is performed to stop the node 10-1 (another node). Thereby, the split-brain generation | occurrence | production state can be rapidly eliminated similarly to the said embodiment.

  Also in the heartbeat monitoring mechanism 240-2 of the node 10-2, when the heartbeat is transmitted to the node 10-1 (other node), the time information ( The operation of adding (Tc, Sc, Rc) is performed. The heartbeat monitoring mechanism 240-1 of the node 10-1 also performs the heartbeat monitoring mechanism 240-2 of the node 10-2 when a heartbeat from the node 10-2 (another node) is detected. The same heartbeat reception time determination process as that described above is performed.

  In addition, this invention is not limited to the said embodiment or its modification example as it is, A component can be deform | transformed and embodied in the range which does not deviate from the summary in an implementation stage. In addition, various inventions can be formed by appropriately combining a plurality of constituent elements disclosed in the embodiment or its modification. For example, you may delete a some component from all the components shown by embodiment or its modification.

The block diagram which shows the hardware constitutions of the cluster system which concerns on one Embodiment of this invention. FIG. 2 is a block diagram mainly showing a functional configuration of a cluster system having the hardware configuration of FIG. 1. The block diagram which shows the structure of the heartbeat monitoring mechanism mainly of the node shown by FIG. The flowchart which shows the operation | movement procedure of the heartbeat monitoring mechanism in the embodiment. The flowchart which shows the operation | movement procedure of the heartbeat monitoring mechanism in the 1st modification of the embodiment. The block diagram which mainly shows the function structure of the cluster system in the 2nd modification of the embodiment.

Explanation of symbols

  10-1, 10-2 ... nodes (computers), 13-1, 13-2 ... external storage devices, 14-1, 14-2 ... communication mechanism, 20 ... communication path, 131-1, 131-2 ... OS (Operating system), 132-1, 132-2 ... program, 210-1, 210-2 ... cluster control mechanism, 211-1, 211-2 ... main control unit, 212-1, 212-2 ... HB transmission unit (Heartbeat transmitter), 213-1, 213-2 ... HB receiver (Heartbeat transmitter), 220-1, 220-2, 240-1, 240-2 ... Heartbeat monitoring mechanism, 221 ... Time information Storage unit 222... Heartbeat detection unit 223 Failure failure processing unit 224 Forced stop processing unit 230 Real time clock 231 Tick counter (time counter) 232 System clock

Claims (7)

In the computers that make up the cluster system,
Heartbeat transmission for notifying other computers constituting the cluster system of the state of the computer is performed at regular time intervals, and heartbeat reception from the other computers is monitored. A cluster that performs failover to continue the service that was executed on the computer of the other computer by determining the failure of the other computer when the heartbeat reception is interrupted beyond a predetermined heartbeat timeout period A control mechanism;
Time keeping means for keeping time or date and time,
The heartbeat transmission to the other computer by the cluster control mechanism is monitored, the time interval of the heartbeat transmission is measured using the time measuring means, and the measured time interval of the heartbeat transmission is the heartbeat transmission time. A heartbeat monitoring mechanism that determines that a split brain has occurred due to slowdown of the computer when the beat timeout time is exceeded, and forcibly stops the computer. Computers that make up the system. The time measuring means is a time counter managed by an operating system, and includes a time counter that counts an elapsed time from the startup of the operating system,
The heartbeat monitoring mechanism measures the time interval of the heartbeat transmission to the other computer by the cluster control mechanism using the time counter, and the measured time interval of the heartbeat transmission is the heartbeat timeout. 2. The computer constituting the cluster system according to claim 1, wherein when the time is exceeded, it is determined that a split brain is generated due to a slowdown of the cluster control mechanism. 3. The timekeeping means further includes a real-time clock that times the date and time independently of the operating system,
The heartbeat monitoring mechanism measures the time interval of the heartbeat transmission to the other computer by the cluster control mechanism using the real time clock, and measures the heartbeat transmission time measured using the time counter. Even if the first time interval that is the interval does not exceed the heartbeat timeout time, the second time interval that is the time interval of the heartbeat transmission measured using the real-time clock is the heartbeat timeout time. 3. The computer constituting the cluster system according to claim 2, wherein the computer is determined to be in a split brain occurrence state due to a slowdown of the operating system. The timekeeping means further includes a system clock that is managed by the operating system and counts the date and time, and can be changed by the operating system.
The real-time clock is operated by the operating system to synchronize with the system clock at the time of the system clock operation by the operating system,
The heartbeat monitoring mechanism measures the time interval of the heartbeat transmission to the other computer by the cluster control mechanism using the system clock, and the first time interval exceeds the heartbeat timeout time. Even if the second time interval exceeds the heartbeat timeout time, the third time interval that is the time interval of the heartbeat transmission measured using the system clock is the second time interval. 4. The computer constituting the cluster system according to claim 3, wherein the computer is determined to be normal if the time intervals coincide with each other. Although the measured heartbeat transmission time interval does not exceed the heartbeat timeout time, the heartbeat monitoring mechanism is longer than the fixed time interval and shorter than the heartbeat timeout time. When the preset slowdown detection time is exceeded, the client computer connected to the computer is notified that a slowdown that has not reached the split brain occurrence state has occurred in the computer. The computer constituting the cluster system according to claim 1.   When the heartbeat monitoring mechanism detects the heartbeat transmitted to the other computer by the cluster control mechanism, the heartbeat monitoring mechanism adds the time or date / time information indicated by the timing means at that time to the heartbeat. When the heartbeat added with the time or date / time information transmitted from the other computer is received by the computer, the time / date / time information added to the received heartbeat and the previous time Based on the received time or date / time information added to the heartbeat from the other computer, the reception time interval of the heartbeat transmitted from the other computer is calculated, and the calculated reception time Slow-down detection preset at a time longer than the fixed time interval and shorter than the heartbeat timeout time. 2. The computer constituting a cluster system according to claim 1, wherein when the time is exceeded, the client computer connected to the computer is notified that a slowdown has occurred in the other computer. . A computer constituting a cluster system, wherein heartbeat transmission for notifying other computers constituting the cluster system of the state of the computer is performed at regular time intervals, and heartbeat reception from the other computers is performed. When the heartbeat reception from the other computer is interrupted after exceeding a predetermined heartbeat timeout time, the failure of the other computer is determined and executed by the computer of the other computer. To a computer having a cluster control mechanism that performs failover to continue the service, a time measuring means for measuring time or date, and a heartbeat monitoring mechanism,
The heartbeat monitoring mechanism measures the heartbeat transmission time interval by using the time measuring means by monitoring the heartbeat transmission to the other computer by the cluster control mechanism;
The heartbeat monitoring mechanism comparing the measured heartbeat transmission time interval with the heartbeat timeout time;
When the heartbeat monitoring mechanism determines that the measured brainbeat transmission time interval exceeds the heartbeat timeout time, it is in a split brain occurrence state due to slowdown of the computer;
A program for causing the heartbeat monitoring mechanism to forcibly stop the computer when it is determined that the split brain has occurred.

Images