CN103370694B - Restart the data processing system - Google Patents

Abstract

Techniques are disclosed including a computer-implemented method that includes sending a message (604) through a process level that includes at least first and second processes that are executing as one or more tasks in response to a predetermined event (506), the message indicating an abort of execution of the one or more tasks, and initiating, by one or more of the processes upon receiving the message, the abort of execution of the one or more tasks (606).

Description

Restart the data processing system

Cross References to Related Applications

This application claims priority to US Application Serial No. 13/031,078, filed February 18, 2011, entitled "Restarting Data Processing Systems," the entire contents of which are hereby incorporated by reference.

technical field

This description involves restarting a data processing system.

Background technique

The speed of computation provided by single-processor computers has increased enormously over the past few decades. However, many applications performed by such processors may require computing power beyond that of even the fastest single-processor computers. For example, in a transactional system such as an airline reservation system, multiple users may have simultaneous access to computer resources. These users typically expect low response times. Single-processor computers may not be able to keep up with such demands. Various architectures, such as parallel processing systems, have been developed to handle such applications in order to improve performance. Typically, parallel processing systems use multiple processors that may be located at a single site or distributed remotely. Because of their processing capabilities, such parallel processing systems have come to be relied upon for applications that process large amounts of data, which in some cases may involve substantially continuous and near real-time processing. Such processing capabilities are expected to be robust and resistant to system failures, ie, fault-tolerant. These capabilities are useful for computer networks of all types and sizes, ranging from large-scale Internet-based data processing to private networking and communication systems (eg, "intranets" within enterprises, etc.).

Contents of the invention

In one aspect, in general, a computer-implemented method includes sending a message through a process level including at least a first and a second process that is executing as one or more tasks in response to a predetermined event, the message indicating that the one or more tasks and one or more of the processes, upon receipt of said message, initiates the suspension of execution of one or more tasks.

Aspects may include one or more of the following.

The computer-implemented method may include storing information related to an initial state of each of the first and second processes while the first and second processes are being initialized. Execution of a process may include executing at least one execution phase of said process, and storing information representative of an end status of said execution phase upon completion of the corresponding execution phase. The computer-implemented method may include resuming execution of one or more of the first and second processes from one of the saved end states without closing the process.

A predetermined event may represent a loss of connection to an external device. A scheduled event may represent an error of an external device. Process execution may resume when the connection to the external device has been re-established. When the external device's error has been cleared, process execution can be resumed. Process execution can be resumed from an end state stored in a stage of execution prior to the occurrence of a scheduled event. Execution of one or more of the first and second processes may resume from the initial state if the predetermined event occurs substantially immediately after starting the process.

Process execution may include performing one or more processing actions on a received data stream in order to generate output data. The computer-implemented method may include sending a checkpoint message through the process-level first and second processes, the checkpoint message including instructions for storing current information about the process, and terminating the process upon receipt of the checkpoint message at the process and initiates storing information about the current execution state of the process into the storage area. The computer-implemented method may include overwriting a previously stored initial or final state with a new initial or final state.

Each of the first and second processes can communicate with one or more data queues for receiving and queuing data for the process. The computer-implemented method may include generating a checkpoint message in response to a network-related event. Network-related events can represent a network shutdown. The computer-implemented method may include periodically generating checkpoint messages. The computer-implemented method may include generating a checkpoint message in response to one or more data values within or derived from an incoming data record to be processed by a process. The computer-implemented method may include resuming execution of one or more of the first and second processes from one of the saved end states based in part on information contained in the resume process message. The computer-implemented method may include receiving one or more messages during a first execution phase of the first process substantially immediately following initialization of the first process, and resuming execution of the first process from a saved initial state, and There is no need to shut down and restart the first process.

In another aspect, generally, a computer-readable storage medium storing a computer program includes instructions for causing a computing system to: send a message through at least a first and a second process including performing one or more tasks in response to a predetermined event At the process level, the message indicates the suspension of one or more tasks being executed, and upon receipt of the message by one or more of the processes, the suspension of the execution of the one or more tasks is initiated.

In another aspect, in general, a computing system includes a device or port configured to send a message through process levels including at least first and second processes performing one or more tasks in response to a predetermined event, the message indicating that a task is being performed and at least one processor configured to, upon receipt of said message by one or more of the processes, initiate suspension of execution of the one or more tasks.

In another aspect, generally, a computing system includes means for sending a message through process levels including at least first and second processes performing one or more tasks in response to a predetermined event, the message indicating that one or more tasks are being performed. Abort of tasks, and means for initiating, by one or more of the processes upon receipt of said message, the abort of execution of one or more tasks.

Aspects can include one or more of the following advantages.

Processes in a multiprocessing system can be executed in different stages of execution. In the event of a system failure, terminating and restarting the processing system from the most recently completed checkpoint may consume an excessive amount of processing time and resources. After a processing system has terminated its activities in response to an abnormal condition, it may be necessary to manually reinitialize the processing system by an information technology specialist with extensive experience with such systems. This can cause considerable system downtime. In some examples, a separate process may also be required to detect system failures and notify experts. As such, for increased efficiency and reduced processing resource consumption, a process within a processing system may be executed from its most recently recorded checkpoint rather than restarting the entire system when reestablishing a failed connection to the process. In one implementation, rather than terminating and restarting the entire system, individual processes in the system may be notified to suspend processing until the failed connection is re-established.

Other features and advantages of the invention will be apparent from the following description and from the claims.

Description of drawings

Figure 1 is a block diagram of a multi-process data processing system.

2 and 3 illustrate example multi-process data processing systems.

4 is a flowchart illustrating an example checkpoint operation process.

5 and 6 are flowcharts of example restoration mechanisms.

detailed description

Referring to FIG. 1, a data processing system 100 provides a plurality of processes arranged in a pipeline for processing data. In the example system 100, data is received from a data source 102 (e.g., an application executing on a server 104 acting as a web server) and sent to an application executing on a computer system 108 or in a distributed fashion (e.g., with two or The multi-process data processing module 106 executed by a plurality of networked computer terminals) transmits the data. Data processing module 106 monitors, controls, and executes the data processing aspects of system 100 . To provide for such processing, the data processing module 106 includes one or more queues 110 , 112 , 114 capable of storing data to be processed by one or more processes 116 , 118 . In this example, data received from data source 102 is stored in initial data queue 110 and provided to initial process 116 periodically, as shown. Process 116 processes data (eg, transforms, filters, validates content, etc.) and provides processed data to one or more downstream data queues 112, 112'. Subsequent processes 118, 118' may be provided with data from the queues 112, 112' and perform other (or similar) processing before passing results to other downstream data queues 114, 114'. The illustrated queue and process layout of the data processing module 106 is one of many possible processing schemes that may be used. For example, the data processing module 106 may include additional processes that may be located upstream, downstream, or independent of the illustrated processes (eg, for parallel or serial execution). In some examples, data from the last set of queues (eg, queues 114 and 114') can be output to target application 120 (or applications), such as a relational database management system (RDBMS).

Processes included in data processing module 106 may communicate with external devices and/or other processing systems (eg, computer system 122 ). For example, a process can communicate with a Java Message Service (JMS) queue that provides messages to processes from other systems. In some cases, processes in data processing module 106 may communicate with one or more databases (eg, located in external system 122 ). For example, module 106 may perform an update of the customer's financial account in the bank's database based on information received from a corresponding customer session at one or more automated teller machines (ATMs).

By way of example, FIG. 2 illustrates a processing system 200 having a remotely executed processing module 202 (executed by an ATM 204 ) for providing data for processing at a central location. In the illustrated example, an initial process such as the read ATM process 206 can receive customer account data (eg, associated with a transaction) from the ATM and pass the data to the verify account process 208 for authenticating account details. In this example, the verify account process 208 may check the PIN entered by the customer against a personal identification number (PIN) database 210 . Once the identity of the customer has been authenticated, a further data record may be passed downstream to a check balance process 212 which may communicate with a second, different database 214 eg for checking the balance of the identified customer account. After further transactions are completed, additional data may be sent downstream to the update balance process 216, which may communicate with the third database 218, eg, to update balance information associated with the customer account. Create answer process 220 may prepare an output summary of the transaction that may be provided to output display process 222 (eg, for display to a user at ATM 204 ). Databases 210 , 214 , and 218 may communicate with master data server 224 for system level monitoring (eg, system quality assurance) or other applications. In some implementations, the database can be executed, for example, by a separate computer system 226 .

Processes in such a multiprocessing system may be executed in different execution stages. Execution of a process may include the execution of one or more tasks within the process in different stages of execution. By segmenting execution into such phases, different execution phases can be terminated, for example, by multiple logical endpoints or breakpoints in the data processing. Each execution phase can have one or more processing activities to achieve the goals of that execution phase. As an example, the verify account process 208 may be performed in one or more ways in different stages of execution. For example, as a first stage of execution, the verify account process 208 may initially receive personal identification number (PIN) information from a customer. Various processing activities upon receiving PIN information from a customer may include, for example, displaying a prompt on an ATM display, and running routines to verify data entries for the PIN information. In the next stage of execution, process 208 may establish a connection to database 210 and use the PIN information as a key to identify the customer's record. Once the process 208 has completed its recorded transaction with the customer, the connection to the database 210 may be terminated. In a final execution stage, process 208 may generate results based on the transactions described above. As such, each of these execution phases includes a distinct logical endpoint (or processing breakpoint) at which the verify account process 208 can be temporarily suspended and/or resumed.

Under certain circumstances, one or more events may occur that tend to affect the normal course of system operation. Such events may be anomalies or errors caused by hardware or software modules included in the processing system. For example, a hardware exception or error may include a reset, interrupt, or other signal from one or more hardware units. Exceptions may also be generated by the arithmetic logic unit for numerical errors such as division by zero, overflow, instruction decode error, undefined instruction, and the like. In some cases, a device connected to the network may fail or go offline temporarily, causing other devices on the network to fail. Based on the occurrence of one or more of these events, it may be necessary to temporarily cease operation of one or more of the databases for corrective action (eg, maintenance, switching to a second system, etc.).

Other events that may require cessation of operations and corrective action may include detecting a failure of one or more of the databases 210-224. Such failures may occur for various reasons. For example, there may be an error in the memory allocation or there may be a conflict writing to the memory space. There may also be errors in the underlying data manipulation, such as when a process attempts to withdraw funds from a depleted account. In addition to events where there are temporary failures, there may also be events triggered by operator intervention. In some implementations, the operator can correct the condition that caused the failure, or the system can correct the condition in time. Examples of such events may include, without limitation, failure of one or more devices connected to the network, shutdown of one or more devices or software services for maintenance, failure or switching of devices or software services, such as Exhaustion of storage resources, load on processing units, timeout of one or more software services.

To detect and resolve such events, data processing systems may employ one or more techniques commonly referred to as checkpointing techniques to ensure minimal system downtime in the event of a failure or when taking the system offline for maintenance or switchover. Checkpointing techniques generally involve storing details of the current state of a process as a checkpoint record so that the process can use the stored information to later restart from that state. For example, verify account process 204 may save its current state in a checkpoint record as it completes each execution stage (prior to initiating execution of the next execution stage or other processing).

Checkpoint records can include various types of information, such as process values, information about successfully processed records, and other details about the current stage of execution of the process. For example, a checkpoint record may include information about the current position in a data queue (eg, the data queue of FIG. 1 ) from which data is being processed. As such, processing can be resumed from this queue position after the operation has been stopped. Along these lines, after recovering from a system failure, a process can restart from a stored intermediate checkpoint state rather than from an initial state.

As an example, if the PIN database 210 fails, the verify account process 208 may generate an exception causing the entire processing system to terminate. On restart, processes (or parts of processes) in the processing system can continue processing from their most recent checkpointed state. In this example, since the failure and reboot occurred at a point in time after the customer provided his PIN information, the PIN information is reverted to the process and does not need to be recollected from the customer. As such, there may not be a need to prompt the customer again for his PIN information.

In the event of a system failure, terminating and restarting the processing system from the most recently completed checkpoint may consume an excessive amount of processing time and resources. After a processing system has terminated its activities in response to an abnormal condition, the processing system may need to be manually reinitialized by an information technology specialist with extensive experience with such systems. This can cause considerable system downtime. In some examples, a separate process may need to be designed to detect system failures and notify experts. In US Patent No. 6,584,581 entitled "Continuous Flow Checkpointing Data Processing", in US Patent No. 5,819,021 entitled "Overpartitioning System and method for increasing checkpoints in component-based parallel applications", and in "Methods and Systems An example of a checkpoint operating system is described in US Patent No. 5,712,971 for Reconstructing the State of a Computation," the contents of each of which are hereby incorporated in their entirety.

To improve efficiency and reduce processing resource consumption, a process is executed from its most recently recorded checkpoint rather than restarting the entire system when a failed connection is re-established. In one implementation, individual processes in the system may be notified to suspend processing until the failed connection is re-established, rather than terminating and restarting the entire system.

Figure 3 shows a block diagram of a multi-process system 300 comprising a data source process 302, processes 304a-n, a data receiving process 306, and communicating with each of the other processes (processes 302, 304a-n) fault tolerance manager 308 . In some implementations, fault tolerance manager 308 may execute as another process within multi-process system 300 . In some cases, fault tolerance manager 302 may be an application running on a separate computer system (not shown) or implemented in a special-purpose processor such as the one in, for example, the "Continuous Flow Checkpointing The checkpoint processor described in US Patent No. 6,584,581 of "Data Processing", the contents of which are incorporated herein in their entirety.

One or more techniques may be implemented to establish communication between processes 302-306 and fault tolerance manager 308. For example, separate exception channels 310a-n may be used to convey information about abnormal conditions that may occur in processes 302-306. Channels 310a-n may be part of a wired network system, a wireless network system, or a combination of wired or wireless network systems. Channels 310a-n may be used by processes 302-306 to communicate error information about processes 302-306 to fault tolerance manager 308. For example, if an external device communicating with process 304a fails, process 304a may immediately generate an error flag and communicate the error to fault tolerance manager 308 via exception channel 310b.

In addition to exception channels 310a-n, fault tolerance manager 308 may send command messages (eg, abort/pause command messages and checkpoint command messages) to processes 302-306 over corresponding communication channels 312a-e. Communication channels 312a-e are arranged to transmit command messages from fault tolerance manager 308 to each of processes 302-306 in sequence. For example, a message from fault tolerance manager 308 may first be communicated to data source process 302 via channels 312b-d, and then passed sequentially through each of processes 304a-n and data receiving process 306. Data receiving process 306 may transmit command messages to fault tolerance manager 308 using channel 312e.

Channel Channel process 304a may further communicate with external database 318 (executing on computer system 320). From time to time, the connection to database 318 may fail, or database 318 may be taken offline for maintenance. The failure may be a hardware failure of the computer system 320 executing the database 318 . In such a case, process 304a may generate an error flag on exception channel 310a to notify fault tolerance manager 308 that the connection was lost.

Upon receiving notification of an error, fault tolerance manager 308 may generate and propagate an abort command message through processes 302-306. In some implementations, the abort command message 322 notifies each of the processes 302-306 to suspend operations and abort any work in progress. Abort command message 322 may be a special message packet that causes processes to abort their current processing.

The abort message is typically transmitted first to data source process 302 via channel 312a, then through each of processes 302-306 via channels 312b-d, and finally back to fault tolerance manager 308 via channel 312e. Upon receipt of an abort message 322, each of the processes 302-306 aborts its current activity with a relatively small delay (if any) and flushes/discards any outstanding tasks that may have been processed since the most recent checkpointed state or record. After a process has suspended its activity, it may pass an abort message 322 to the next downstream process. In this manner, the abort message 322 propagates to the receiving process 306 before returning to the fault tolerance manager 308 . The fault tolerance manager 308 waits until it receives an abort message 322 from the receiving process 306, which ensures that all processes 302-306 have aborted the current processing task (eg, are in a quiescent state).

When database 318 fails due to a hardware failure in computer system 320, processes 302-306 are instructed to abort their processing. In some implementations, after the system has completely suspended its processing, process 302 may wait a specifiable amount of time that reflects the average amount of time required to correct a failure, and begin processing again from the most recent saved checkpoint state . In some implementations, the process 304a may periodically poll the database 318 to check its status (ie, check whether the database 318 is operational). In some examples, computer system 320 may be configured to automatically notify process 304a when database 318 is returned to an operational state. When the connection to database 318 is re-established, processing system 300 may again begin processing from the most recently saved checkpoint state.

In this regard, process 304a notifies fault tolerance manager 308 that the connection has been re-established. Fault tolerance manager 308 determines, for each of processes 302-306, the state of the most recently successfully completed checkpoint and sends a recovery process message (not shown) to each of processes 302-306. As with other command messages, resume processing messages propagate sequentially through each of processes 302-306 via communication channels 312a-e.

In an embodiment, the resume processing message specifies a checkpoint state from which the process 302-306 will resume processing. A checkpoint operation involves storing multiple checkpoint states to corresponding storage areas. To store checkpoint state data associated with each of the processes 302, 304a-n, 306, a storage area (eg, memory) may be allocated to each process. Each process periodically suspends its current operation at the end of a different phase of execution and stores its checkpoint data in an associated storage area. For example, data source process 302 may periodically suspend its current operation at the end of different execution stages in processing (eg, introducing a logical breakpoint in the data flow) and store checkpoint information in storage area 311 . In this manner, as each of the processes 302, 304a-n, 306 executes, the corresponding storage area 311, 314a-n, 316 periodically holds checkpoint data. Checkpoint data may include information about the current state and/or data associated with the processes 302-306 to allow reconstruction of those states at a later time. Storage areas 311-316 may be implemented using various types of storage technologies, for example on non-volatile storage on magnetic media such as hard drives.

Fault tolerance manager 308 manages checkpoint operations by generating and sequentially delivering checkpoint command messages (not shown) to each process 302-306 over communication channels 312a-e. A more detailed description of the checkpoint command messaging system is provided in co-pending US Patent Application No. 13/030,998, the contents of which are hereby incorporated by reference in their entirety. A checkpoint command message is passed through each process 302-306 so that the process can checkpoint its current state when the message is received. As such, the checkpoint command message travels to the data source process 302 , then through each process 304 a - n and the data receiving process 306 in sequence before returning to the fault tolerance manager 308 . This checkpoint operation can be started automatically at regular intervals. For example, fault tolerance manager 308 may initiate checkpoint command messages at a predetermined periodic rate, such as every five minutes. The cycle rate can be set as a default, or adjusted by the user. In some examples, one or more external triggers may initiate operations for storing checkpoint information. In one example, a network message may notify fault tolerance manager 308 of an impending network shutdown, thereby triggering a checkpoint operation. In some implementations, a checkpoint operation may be triggered in response to a value within or derived from a data record being processed. For example, a processed data record may include a timestamp or a breakpoint value that may be considered a logical point at which a checkpoint operation may occur.

As the checkpoint information is stored during the period in which the data is being processed by the system, the information may be stored prior to processing the data. In one embodiment, an initial checkpoint operation may be triggered when the multi-process system 300 is first initialized, eg, during startup. Fault tolerance manager 308 may pass initial checkpoint command messages through each of processes 302-306. In the example shown in FIG. 3 , an initial checkpoint message is first transmitted to the data source process 302 . The data source process 302 immediately performs a checkpoint operation, such as storing data representing the initial state of the data source process 302 in the associated data storage space 311, and passing an initial checkpoint message to the downstream next process 304a. This initial checkpoint state is called checkpoint state zero. Similarly, in serial fashion, each of the processes 304-306 may respectively store its initial state and associated data values to the appropriate information area as checkpoint state zero. In an example, the initial state and associated data values may include initial values of global variables, reference data information, and audit variables including initial values of counters.

After each of processes 302-306 have stored their initial state, an initial checkpoint command message is returned to fault tolerance manager 308 via channel 312e. Based on the message returned to the fault tolerance manager 308 after its round trip through the processes 302, 304a-n, 306, the fault tolerance manager processes 302-306 are alerted that they have completed checkpoint status zero. In some implementations, while downstream processes are saving their current state, the source and other upstream processes can continue to receive data and perform other functions without waiting for all processes to save their state.

Similarly, additional checkpointing operations may be performed for each distinct stage of execution of processes 302-306. As such, along with storing data representing initial checkpoint information, fault tolerance manager 308 may initiate, for example, an Storage of additional information. To initiate storage of subsequent checkpoint information, techniques such as propagating further checkpoint command messages through processes 302, 304a-n, 306 may be used. Upon receiving the checkpoint command message, process 304a may complete any ongoing tasks or suspend any outstanding tasks. In some examples, process 304a may delete previously created checkpoint records stored in data store 314 and re-request storage space. Process 304 may then create a new checkpoint record for its current state and associated data. In some cases, earlier checkpoint records are permanently stored in memory and are not overwritten by new checkpoint records. Additional examples of information stored in checkpoint operation records are provided in US Patent No. 6,584,581, the contents of which are incorporated herein in their entirety.

In some instances, fault tolerance manager 308 may initiate additional checkpoint operations while a previous checkpoint operation is currently being performed. For example, while process 304n is processing any checkpoint state (e.g., checkpoint state N corresponding to checkpoint command message N), fault tolerance manager 308 may generate and transmit subsequent checkpoint command message N+ 1 to start the subsequent checkpoint state (e.g. checkpoint state N+1). Along these lines, while checkpoint command message N is still traveling through processes 302-306, it is possible to generate a new subsequent checkpoint command message N+1 and pass said checkpoint command message N+1 through processes 302-306 of. In this manner, fault tolerance manager 308 can cause more frequent checkpointing of process states without having to wait until a previous checkpointed state is complete.

In some cases, a system failure may occur while one or more checkpoint command messages are transmitted through the processes 302, 304a-n, 306. For example, consider the case where fault-tolerance manager 308 has initiated a checkpoint state N by generating a checkpoint command message N . While checkpoint command message N is being processed by processes 302-306, the connection between one of the processes (eg, process 304a) and the external system (eg, database 312) may fail. Upon being alerted of the situation, fault tolerance manager 308 may respond through processes 302-306 by passing an abort command message 322. Abort command message 322 may arrive at a process (eg, process 304n ) that is still processing checkpoint state N (eg, storing checkpoint information associated with checkpoint N). Based on receipt of the abort command, process 304n may take one or more actions. For example, process 304n may complete checkpoint state N and abort all further processing. In another case, process 304n may discard results associated with the current and subsequent states since the previous checkpoint state N-1, and abort further processing. As a result, each of processes 302-306 may be in a different checkpoint state when system 300 achieves quiescence. For example, all processes upstream of process 304n (e.g., data receiving process 306) may have completed checkpoint state N, while all processes downstream of process 304n (e.g., process 304a and data source process 302) may have only completed checkpoint state N -1.

When system 300 is ready to resume processing, fault tolerance manager 308 transmits one or more resume processing messages to each process via communication channels 312a-e. The resume processing message indicates to the process the earliest, fully committed (or completed) checkpoint state (eg, checkpoint state N-1 ) from which the process will execute. In an example, a process that may have completed checkpoint state N may only reproduce results from checkpoint state N−1 to checkpoint state N. In this way, processes 302-306 can avoid repeating their earlier efforts. In an example, replaying results from checkpoint state N−1 to checkpoint state N involves reproducing the results of earlier processing actions that may have occurred between the two checkpoint states.

In an example, system failure may occur substantially immediately after startup. In such a case, many of the processes 302-306 may have only completed checkpoint state zero. These processes 302-306 may resume processing from a checkpoint state of zero based on initialization data and startup values stored in the corresponding checkpoint records.

4 is a flowchart depicting an example execution of a process (eg, process 302 of FIG. 3 ) within a multi-process system. Upon startup, the process immediately stores its initial state into data storage as checkpoint state zero (step 402). The process can then execute in different execution phases (eg, execution phase 1, 2, . . . , N-1). At the end of each stage of execution, a process can save its end state to data memory as a checkpoint state. For example, after the first execution phase, the process may save the end state of the first execution phase as checkpoint state 1 (state 404 ). Similarly, after a subsequent execution phase, the process may save the end state of the execution phase as checkpoint states 2, . . . , N-1, and N (steps 406-410).

5 is a flow diagram depicting example steps performed when storing and restoring from checkpoints while executing a process. For example, when a process is initialized, information related to the initial state of the process is stored in an associated memory area (step 502). The process is then executed in different execution phases. As such, at the end of each execution phase, the process stores information representing the end state of the execution phase (step 504). When a predetermined event occurs, eg, loss of connection to an external device (step 508), execution of the process is aborted (step 506). At the same time, the process checks to see if the event that triggered the pause has cleared (for example, reestablishing a connection to an external device). During this time, the process is not closed, but aborted until the event is considered cleared. Execution of the process resumes from the most recently saved initial or end state (step 510).

6 is a flowchart depicting example steps performed in the event of an external system failure or taking the external system offline for maintenance. For example, an external system may be a database (eg, database 318 of FIG. 3 ) in communication with processes in the processing system. When an external system is taken offline for maintenance or an external system fails, an error flag may be generated, for example, by a process communicating with the failed external system (step 602 ). An abort message (eg, abort command message 322 of FIG. 3 ) may be generated and sent through the process in response to the error flag (step 604 ). One or more techniques may be employed for such message transmission. For example, different transmission paths and different types of messages or combinations of messages can be implemented. When the process receives the abort command message, the current activity of each of the processes in the process is aborted (step 606 ). One or more techniques may be implemented for stopping such processes. An abort may include the execution of one or more functions, for example, a process may discard any transactions executed since the most recent checkpointed state. Further actions are aborted until the failed connection to the external system is re-established (step 608). When the connection is re-established, a resume processing message is transmitted to each of the processes (step 610). A resume processing message indicates a checkpoint state from which the process will resume processing. As such, each of the processes retrieves relevant information about the state of the checkpoint from their associated storage area (step 612).

Restarting of the processes described above can be accomplished using software for execution on a computer. For example, the software forms a process in one or more computer programs executing on one or more programmed or programmable computer systems (which may be of various architectures such as distributed, client/server, or grid), Each of said computer systems includes at least one processor, at least one data storage system (including volatile and non-volatile memory and/or storage elements), at least one input device or port, and at least one output device or port. The software may form one or more modules within a larger program, eg, that provides other services related to the design and configuration of dataflow graphs. The nodes and elements of a graph may be implemented as data structures stored in a computer readable medium, or other organized data conforming to a data model stored in a data repository.

The software may be provided on a storage medium, such as a CD-ROM, which can be read by a general or special purpose programmable computer, or may be delivered (encoded in a propagated signal) via a communication medium to a network of computers executing the software the software. All functions can be performed on a special purpose computer, or using special purpose hardware such as coprocessors. The software can be implemented in a distributed fashion with different computers performing different parts of the calculations specified by the software. Each such computer program is preferably stored or downloaded to a storage medium or device (e.g., solid-state memory or media, or magnetic or optical media) that can be read by a general-purpose or special-purpose programmable computer for use when programmed by the computer system Configure and operate a computer while reading a storage medium or device to perform the processes described herein. The inventive system may also be considered to be implemented as a computer-readable storage medium configured using a computer program, wherein the storage medium is configured such that the computer system is operated in a specific and predetermined manner to perform the functions described herein.

A number of embodiments of the invention have been described. However, it will be understood that various modifications may be made without departing from the spirit and scope of the invention. For example, some of the steps described above may be order independent, and thus may be performed in an order different from that described.

It should be understood that the above description is intended to illustrate, and not to limit, the scope of the invention, which is defined by the scope of the appended claims. For example, many of the functional steps described above may be performed in a different order without materially affecting the overall process. Other embodiments are within the scope of the following claims.

Claims (42)

1. A computer-implemented method comprising: sending a message to at least a first process and a second process configured to perform one or more tasks in response to a predetermined event, the message indicating suspension of execution of the one or more tasks, wherein the first process is configured to provide data to the second process; and Upon receipt of the message, an abort of execution of the one or more tasks to be aborted by one or more of at least the first process and the second process is initiated. 2. The computer-implemented method of claim 1, comprising: Information about an initial state of each of the first process and the second process is stored while the first process and the second process are being initialized. 3. The computer-implemented method of claim 1, wherein execution of a process includes executing at least one execution phase of the process, and storing information representative of an end status of the execution phase upon completion of the corresponding execution phase. 4. The computer-implemented method of claim 3, comprising: resuming execution of one or more of the first process and the second process from one of the saved end states without closing the one of the first process and the second process or multiple processes. 5. The computer-implemented method of claim 1, wherein the predetermined event represents a loss of connection to an external device. 6. The computer-implemented method of claim 1, wherein the predetermined event represents an error of an external device. 7. The computer-implemented method of claim 5, wherein a recovery process is performed when a connection to the external device has been re-established. 8. The computer-implemented method of claim 6, wherein execution of the first process or the second process is resumed when an error of the external device has been cleared. 9. The computer-implemented method of claim 3, wherein process execution is resumed from an end state stored by a stage of execution prior to the occurrence of the predetermined event. 10. The computer-implemented method of claim 2, wherein if the predetermined event occurs after initiation of one or more of the first process and the second process and before completion of an execution phase, Then resume execution of the one or more processes in the first process and the second process from the initial state. 11. The computer-implemented method of claim 1, wherein process execution includes performing one or more processing actions on a received data stream to generate output data. 12. The computer-implemented method of claim 1, comprising: sending a checkpoint message through the first process and the second process, the checkpoint message including instructions for storing current information about the first process and the second process, and suspending operation upon receipt of said checkpoint message at said first process or said second process and initiating storage of information related to a current execution state of said first process or said second process to a storage area . 13. The computer-implemented method of claim 12, comprising: Overwrites a previously stored initial or final state with a new initial or final state. 14. The computer-implemented method of claim 1 , wherein each of the first process and the second process communicates with one or more data queues for at least the first process and the second process The second process receives data and queues the data. 15. The computer-implemented method of claim 12, further comprising: Checkpoint messages are generated in response to network-related events. 16. The computer-implemented method of claim 15, wherein the network-related event represents a network shutdown. 17. The computer-implemented method of claim 12, further comprising: The checkpoint messages are generated periodically. 18. The computer-implemented method of claim 12, further comprising: The checkpoint message is generated in response to one or more data values within or derived from an incoming data record to be processed by at least the first process and the second process. 19. The computer-implemented method of claim 2, further comprising: Execution of one or more of the first process and the second process is resumed from one of the saved end states based in part on information contained in the resume processing message. 20. The computer-implemented method of claim 1, further comprising: receiving the message during a first execution phase of the first process after initialization of the first process and prior to completion of the first execution phase of the first process, and Resuming execution of the first process from the saved initial state without shutting down and restarting the first process. 21. The method of claim 1, wherein the second process is configured to process data received from the first process by at least one of transforming, filtering, validating data content. 22. A computing system comprising: a device or port configured to send a message to at least a first process and a second process performing one or more tasks in response to a predetermined event, the message indicating an abort of the one or more tasks being performed, wherein the the first process is configured to provide data to the second process; and at least one processor configured to, upon receipt of said message, initiate execution of said one or more tasks to be aborted by one or more of at least said first process and said second process suspension. 23. The computing system of claim 22 , wherein the at least one processor is configured to store information related to the first process and the second process when the first process and the second process are being initialized. Information about the initial state of each process in . 24. The computing system of claim 22, wherein execution of a process includes executing at least one execution phase of the process, and storing information representative of an end status of the execution phase upon completion of the corresponding execution phase. 25. The computing system of claim 22 , wherein the at least one processor is configured to resume one or more of the first process and the second process from one of the saved end states without closing the one or more processes of the first process and the second process. 26. The computing system of claim 22, wherein the predetermined event represents at least one of a loss of connection to the external device or an error of the external device. 27. The computing system of claim 22, wherein process execution is resumed from an end state stored by a stage of execution prior to the occurrence of the predetermined event. 28. The computing system of claim 22 , wherein the at least one processor is configured to send a checkpoint message through the first process and the second process, the checkpoint message including information for storing information about the An instruction describing the current information of the first process and the second process. 29. The computing system of claim 22 , wherein each of the first process and the second process communicates with one or more data queues for at least the first process and the second process Two processes receive data and queue the data. 30. The computing system of claim 22, wherein the second process is configured to process data received from the first process by at least one of transforming, filtering, validating data content. 31. A computing system comprising: means for sending a message to at least a first process and a second process executing one or more tasks in response to a predetermined event, the message indicating an abort of the executing one or more tasks, wherein the first process is terminated by configured to provide data to the second process; and means for, upon receipt of said message, initiating abort of execution of said one or more tasks to be aborted by one or more of at least said first process and said second process. 32. The computing system of claim 31 , comprising means for storing an initial process associated with each of the first process and the second process when the first process and the second process are being initialized. A device for state-related information. 33. The computing system of claim 31 , comprising means for resuming execution of one or more of the first process and the second process from one of the saved end states without shutting down A means for the one or more processes of the first process and the second process. 34. The computing system of claim 31, wherein the predetermined event represents at least one of a loss of connection to the external device or an error of the external device. 35. The computing system of claim 31, wherein the second process is configured to process data received from the first process by at least one of transforming, filtering, validating data content. 36. A computer implemented method comprising: sending an abort command message from a manager executing on the computer system in response to a predetermined event; receiving an abort command message at a first process configured to execute one or more tasks; suspending execution of one or more tasks of the first process; sending an abort command message from said first process to a second process configured to perform one or more tasks; suspending execution of one or more tasks of the second process; and A return abort command message is sent to the manager. 37. The method of claim 36, wherein the first process is configured to provide data to the second process, and the second process is configured to process the data to complete the one or more Task. 38. The method of claim 36 , comprising sending an abort command message from the second process to a sequence of downstream processes, wherein the returned abort command message is from the last of the downstream processes to the manager send. 39. The method of claim 36, comprising, after said manager receives said returned abort command message, sending a resume process message from said manager, and receiving said Resume processing messages. 40. A computing system comprising: one or more processors that execute instructions to implement a manager configured to send an abort command message in response to a predetermined event; and a series of processes configured to perform one or more tasks; Wherein the first process in the series of processes is configured to suspend the execution of one or more tasks of the first process when receiving an abort command message, and send an abort to the second process in the series of processes a command message, the second process being downstream of the first process; Wherein the second process is configured to suspend execution of one or more tasks of the second process when receiving a suspend command message; Wherein the last process in the series of processes is configured to send back an abort command message to the manager. 41. The system of claim 40, wherein the first process is configured to provide data to the second process, and the second process is configured to process the data to complete the one or more Task. 42. The system of claim 40, wherein the manager is configured to send a resume processing message to the series of processes after receiving the returned abort command message.