JP2016091429A - Information process system and control method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide data of a processing treatment result with invariably high reliability without manually performing a processing treatment on data accumulated in the past when adding the processing treatment.SOLUTION: An information processing system according to one embodiment of the present invention includes a data storage server 102 and an ETL management server 101. The data storage server 102 stores data of a plurality of network devices gathered through a network and ETL execution-completed data, and provides the ETL execution-completed data for an analysis system which takes analysis. The ETL management server 101 manages execution of ETL and rules of ETL. The ETL management server 101 performs processing according to first and second rules when the second rule related to the first rule managed by a rule management part 302 is added. Further, even input data having been processed according to the first rule before the second rule is added is processed according to the second rule.SELECTED DRAWING: Figure 1

Description

  The present invention relates to an information processing system and a control method.

  Currently, logs of applications operating on the server and information on a large amount of devices operating on the market are collected and accumulated on the server. The accumulated data is used for improving the quality of applications and devices, improving usability, or predicting failures, and is aggregated and analyzed. An application that runs on a server is a Web service provided to a customer or an in-house system for a company that supports the business of its own users. In addition, devices include a wide range of devices such as MFPs (Multi Function Peripherals), printers, mobile phones, smartphones, and PCs (Personal Computers). In addition to data collected from devices, service person dispatch history data for support such as device repair, and complaint information to a call center are also analyzed. As described above, the sizes and types of data to be analyzed are increasingly diversified and scaled up.

In order to efficiently analyze the data, the accumulated data is processed. The processing includes removal of illegal data included in data, character code correction, code addition, normalization, and format unification. Such processing is generally called ETL (Extract Transform Load). The ETL needs to be added or updated flexibly according to the requirements of the user or system using the data, and is required to manage a plurality of output data for one input data.
Patent Document 1 discloses a database system that employs a configuration in which a purpose-specific database is extracted from a basic database. In Patent Document 1, a difference between a newly generated purpose-specific database and a purpose-specific database at the previous extraction time is managed, and when there is a reproduction request, the purpose-specific database collected at a certain past time is reproduced according to the difference.

JP-A-5-257765

  However, in Patent Document 1, since desired data is reproduced when data is used, it is considered that the user or the system is in a waiting state during the reproduction process. Further, when changing the processing, the administrator needs to manually apply the processing to the difference data and the latest data.

  The present invention provides an information processing system that always provides highly reliable processing result data without manually performing processing processing on data accumulated in the past when processing processing is added. With the goal.

  An information processing system according to an embodiment of the present invention is suitable for analysis of first storage means for storing data of a plurality of network devices collected via a network and data stored in the first storage means. Management means for managing a plurality of rules when processing to make data, processing means for processing the data stored in the first storage means according to the managed rules, and processed data Second storage means for storing, and providing means for providing processed data stored in the second storage means to an analysis system for performing analysis. The processing means performs processing according to each of the first and second rules when a second rule related to the first rule managed by the management means is added. Furthermore, before the second rule is added, the data stored in the first storage means processed according to the first rule is also processed according to the second rule.

  According to the information processing system of the present invention, when machining processing is added, data of machining processing results with high reliability is always provided without manually performing processing processing on data accumulated in the past. An information processing system can be provided.

It is a figure which shows the structure of the information processing system which concerns on 1st Embodiment of this invention. It is a figure which shows the hardware constitutions of the server in 1st Embodiment. It is a software block diagram of an ETL management server. It is a software block diagram of a data storage server. It is a software block diagram of a data collection server. It is a figure which shows the list screen which an ETL management server displays on a management terminal. It is a figure which shows the new addition screen which an ETL management server displays on a management terminal. It is a flowchart explaining execution of ETL of an ETL management server. It is a flowchart explaining the process at the time of a rule being added. It is a flowchart explaining the process which deletes non-primary data. It is a figure which shows the structure of the information processing system which concerns on 2nd Embodiment of this invention. It is a figure which shows the server number designation | designated screen in 2nd Embodiment. It is a figure which shows the ETL time limit designation | designated screen in 3rd Embodiment of this invention. It is a flowchart explaining the calculation process of the number of servers required for ETL execution.

Hereinafter, embodiments for carrying out the present invention will be described with reference to the drawings.
(First embodiment)
FIG. 1 is a diagram showing a configuration of an information processing system according to the first embodiment of the present invention.
In FIG. 1, the Internet 100 represents the Internet. The servers 101 to 104 and 106 described below, the management terminal 105, and the analysis terminal 107 are connected to each other via the Internet. The Internet 100 is not limited to the Internet, and may be a corporate network (intranet).

  The ETL management server 101 is a server that manages ETL execution and ETL rules. Before using data, it is common to process the data in a form that is easy to use. Specifically, there are processing processes such as removal of illegal data included in data, character code correction, sign assignment, normalization, and format unification. In the present embodiment, such processing is referred to as ETL. Each time the data collection server 103 periodically collects data, the ETL management server 101 executes ETL on the newly collected data stored in the data storage server 102. In addition, the ETL-executed data is stored in the data storage server 102. Furthermore, a rule change from the management terminal 105 is accepted, and if necessary, ETL is automatically executed on past data stored in the data storage server 102. By automatically executing the ETL based on the rule change, it is possible to reduce the trouble of the administrator manually executing and storing the ETL on past data every time the rule is changed.

  The data accumulation server 102 accumulates data collected by the data collection server 103 and output data obtained as a result of ETL performed by the ETL management server. In addition, ETL-executed data is provided in accordance with requests from the analysis terminal 107 and the data use server group 106. The data storage server 102 can accept input / output of data by a general protocol such as FTP, HTTP, SMB, or database connection, but is not limited to this as long as it is a protocol capable of data input / output.

  The data collection server 103 collects data necessary for data analysis from a network device such as the business server group 104. At this time, the data collection server 103 collects data from each server of the business server group 104 by a data transfer protocol such as FTP, HTTP, or SMB, or database connection. The protocol used for data collection follows the protocol provided by each server of the business server group 104. In this embodiment, the protocol is not limited. In addition, since the business server group 104 adds and updates data during operation, the data collection server 103 periodically collects data from the business server group 104. This may be performed by using a standard function of the OS such as Linux (registered trademark) crontab or Windows (registered trademark) task scheduler, or software having a schedule management function.

  The business server group 104 is a set of server machines that operate for purposes other than data analysis. For example, a server that provides a Web service to a customer via the Internet, or a server that operates an in-house system for a company that supports the business of the company user. In recent years, mobile terminals operating in the user environment and business devices such as printers can be connected to the Internet. It may be a server connected from such a device and managing it. The business server group 104 may be a physical server machine or a server machine constructed in a virtual environment. Since these servers are constructed for each business purpose, they do not have a data analysis function. Therefore, data analysis is possible only when the data collection server 103 collects these data in an integrated manner.

  The management terminal 105 is a terminal used by an administrator. The administrator uses the management terminal 105 to instruct the ETL management server 101 to change the rule. Here, the management terminal 105 can be connected to the ETL management server 101 via a Web browser or the like. However, the management terminal 105 is not limited to this as long as an instruction to change the rule can be given.

  The data use server group 106 is a server group for using ETL-executed data stored in the data storage server 102. For example, the data utilization server group 106 can perform data analysis for improving the quality of the business system by collecting and monitoring business system errors provided to the customer. In another example, in the data use server group 106, a terminal that requires maintenance is identified from usage logs transmitted by a large number of terminals, so that the customer is warned or the customer is visited for maintenance. It can be used to implement.

The analysis terminal 107 is a terminal for performing data analysis by referring to data stored in the data storage server 102 directly by an analyst. The data use server group 106 is a server that performs data analysis for some specific purposes, whereas in the analysis terminal 107, an analyst performs data analysis using various tools and know-how. The first step in utilizing data is to start by verifying the data. Therefore, data can be effectively used by designing and constructing the data use server group 106 based on the data knowledge obtained by the analysis terminal 107.
In FIG. 1, the servers 101 to 104 and 106 are illustrated as physically separate servers, but each server may be arranged as a virtual server function in the same physical server.

FIG. 2 is a diagram showing a hardware configuration of the ETL management server 101, the data storage server 102, the data collection server 103, the business server group 104, and the data use server group 106 according to the present embodiment.
In FIG. 2, a central processing unit (CPU) 202 controls the entire apparatus. The CPU 202 executes an application program, an OS, and the like stored in the hard disk (HDD) 205, and performs control to temporarily store information, files, and the like necessary for executing the program in the random access memory (RAM) 203. . A read only memory (ROM) 204 is a storage unit, and stores various data such as a basic I / O program therein. A RAM 203 is temporary storage means, and functions as a main memory, work area, and the like for the CPU 202. The HDD 205 is one of external storage means, functions as a large-capacity memory, and stores application programs such as a Web browser, a service group program, an OS, and related programs. A display 206 is a display means for displaying commands and the like input from the keyboard 207. An interface 208 is an external device I / F, and connects a printer, a USB device, a peripheral device, or the like. The keyboard 207 is an instruction input unit. The system bus 201 controls the flow of data in the apparatus. A network interface card (NIC) 209 exchanges data with an external device via the interface 209 and the Internet 100.
The configuration of the computer is an example, and is not limited to the configuration example of FIG. For example, the storage destination of data and programs can be changed to the ROM 204, the RAM 203, the HDD 205, or the like according to the characteristics.

FIG. 3 is a software block diagram for explaining functional modules provided by the ETL management server 101.
The control unit 301 is a main execution unit that calls the functional modules described below and executes ETL. The rule management unit 302 manages rules for controlling execution of ETL. The rule management unit 302 holds the rules as a text file such as a database or XML.

Here, rules managed by the rule management unit 302 will be described using Table A. FIG.
The schedule column represents a schedule for each rule for periodically executing ETL. For example, in Table A, what is indicated as Per Day is a schedule for executing ETL every day. Also, what is indicated as Per 3hour is a schedule for executing ETL every 3 hours. It is preferable to set a schedule according to the frequency of data collection by the data collection server 103 so that the analyst can analyze data in real time as much as possible.

  The input path rule column indicates a file path rule of input data. For example, in the rule in the first row of Table A, / data / device / DEVICE_ALARM_ *. Files corresponding to csv are subject to ETL. Here, * represents an arbitrary character string. In Table A, the input path rule, / data / device / DEVICE_ALARM_ *. Since the ETL corresponding to csv is the first to third lines, the ETL management server 101 executes three ETLs.

  The output path rule column indicates a file path rule for output data. The ETL management server 101 stores the data of the execution result by the ETL execution file in the path defined by the output path rule string. Here, it can be seen that the input path rule in the fifth row of Table A matches the output path rule in the fourth row. This is / data / device / DEVICE_ENVIRONMENT_ *. Regarding csv, after the ETL on the fourth line is executed, the ETL on the fifth line is further executed.

The ETL execution file path column indicates the file path of the execution module for executing the ETL. The ETL management server 101 can apply the ETL to the input file by executing the execution file indicated by the ETL execution file path.
The primary column indicates whether or not to execute the ETL according to the checked rule for the input data that has been accumulated in the past and has already been subjected to the ETL according to the rule managed by the rule management unit 302. The processing when the primary column is checked will be described in detail with reference to the flowchart of FIG.
The data reading unit 303 provides a function of reading ETL target data from the data storage server 102. The data storage server 102 has one or more data disclosure methods such as HTTP, FTP, or a general database connection method. The data reading unit 303 reads data in accordance with the disclosure method of the data storage server 102 and temporarily stores it in the intermediate data management unit 305.

The data storage unit 304 has a function for storing the output data on which ETL has been executed in the data storage server 102. Similar to the data reading unit 303, the data that has been subjected to ETL execution is stored in accordance with the disclosure method of the data storage server 102.
The intermediate data management unit 305 is a storage area for storing data during execution of ETL.
The execution file storage unit 306 is a storage area for storing an ETL execution file. The ETL execution file has a file format such as a script or an execution file (exe, jar), but is not limited thereto.

The ETL execution unit 307 activates the ETL execution file stored in the execution file storage unit 306 and executes the ETL. The ETL is, for example, character code conversion, line feed code deletion, full-width / half-width conversion, null value insertion, digit alignment, simple tabulation, and the like, but is not limited thereto. Further, a combination of these processes may be used.
The rule reception unit 308 receives the addition of a rule in accordance with a request from the management terminal 105. Here, a management screen for the rule receiving unit 308 to accept addition of a rule from the management terminal 105 will be described with reference to FIGS.

FIG. 6 is a diagram showing a list screen on which the rules managed by the rule management unit 302 of the ETL management server are displayed on the management terminal 105.
In the rule list 601, rules managed by the rule management unit 302 can be displayed in a list. When the user presses (operates) the reflection button 603, the rule receiving unit 308 executes ETL on the input data stored in the data storage server 102 in the past with respect to the rule whose primary column is checked. The above process will be described in detail with reference to the flowchart of FIG.

  In this specification, the primary is a true / false value indicating that the data stored in the data storage server 102 is the most suitable data for use of the data. Data set as primary is referred to as primary data. Since primary data is used by many data utilization server groups 106 and analysis terminals 107, it is desirable that primary data can always be acquired. Conventionally, it has been necessary to manually execute ETL for all data accumulated in the past. However, when designated as primary as in this embodiment, the ETL management server 101 automatically executes ETL. By doing so, manual labor can be reduced.

  The primary data is data that needs to be acquired permanently, but data that is not primary data only needs to be temporarily acquired. Here, the reason why ETL continues to be executed even for data that is not primary data is to enable continuous use of the data use server group 106 even when the data use server group 106 is already system-coupled. . However, it is not always necessary to accumulate all data that is not primary data over the past. Therefore, it is necessary to delete periodically. Deletion of data that is not primary data will be described later with reference to the flowchart of FIG.

When the user presses the new addition button 602, the rule receiving unit 308 displays a new addition screen 700 shown in FIG.
In the rule input field 701 in FIG. 7, the user can input information necessary for adding a new rule. A check box 702 is a check box for designating whether or not to register result data obtained by executing ETL as primary data. When the user presses an OK button 703, the rule is registered in the rule management unit 302.
Here, when the check box 702 is checked, as in the case where the primary column is checked in the list screen 600 of FIG. Also performs ETL. The processing at this time is also the same as the flowchart shown in FIG.

FIG. 4 is a software block diagram for explaining the functional modules provided by the data storage server 102.
The data storage unit 501 is a storage area that functions as a first storage unit that stores input data before ETL execution and a second storage unit that stores processed data after ETL execution. The data storage unit 501 is generally an HDD or database that is a physical storage medium, but may be a virtualized object storage, for example. Hereinafter, examples of input data and output data stored in the data storage unit 501 will be described using Tables B to E.

  Table B is an example of alarm history data collected from the business server group 104 by the data collection server 103. The alarm history is an alarm history notified to the business server group 104 when any problem occurs in a large number of devices connected to the business server group 104. In the present embodiment, the data represented in Table B is stored in the / data / device / DEVICE_ALARM_ *. It is stored in the path indicated by csv.

  The country code string is a code representing the country in which the device connected to the business server group 104 is operating. The occurrence date / time column is the date / time at which the alarm occurred. The cancellation date / time column is the date / time when the alarm is canceled by the user or the service person. The data transmission date / time column is the date / time when the device transmits the alarm history to the business server group 104. The model column is a code for identifying the model of the device. The machine number column is an ID (identifier) of the device. The alarm code string is an alarm occurrence classification code. The alarm level column is a numerical value representing the seriousness of the alarm.

In the alarm history data, the line feed code <LF> is included in the second and fifth rows of the country code column and the model column. The fifth line further includes a line feed code <CR>. This is an example in which an invalid line feed code is included. The ETL management server 101 executes ETL to remove this invalid line feed code. In the present embodiment, the first row of Table A is a rule for executing the above-described ETL on the data of Table B. The ETL executable file path is / var / etl / transform_rcd. The result of executing this ETL execution file on the data in Table B is the data in Table C.

  In Table C, it can be seen that an incorrect line feed code <LF> has been removed from the second and fifth lines compared to Table B. Such an incorrect line feed code is mixed due to a software bug when data is generated in the device, an OS mismatch between the device and the data collection server 103, or the like. When an illegal line feed code is mixed, for example, when data is analyzed by the analysis terminal 107 or the data use server group 106, an event in which the data cannot be read correctly occurs. By removing such an illegal line feed code by the ETL management server 101, the analysis terminal 107 and the data use server group 106 can correctly read the data.

Here, it can be seen that the line feed code <CR> is not removed in the fifth line of Table C. This is the ETL executable file / var / etl / transform_rcd. When the jar is implemented, since there is no assumption that the line feed code <CR> is included in the data, the line feed code remains without being removed by the ETL described above. In such a case, it is necessary to newly install an ETL executable file and add rules. The rule in which the third row of Table A is added is shown. ETL execution file of the rule on the third line / var / etl / transform_rcd2. In the ETL process by jar, it is possible to delete a line feed code <CR>. Here, the primary data in the ETL management server 101 may be data obtained by executing ETL according to the rule in the third row.

  Table D is an example of in-device environment data collected from the business server group 104 by the data collection server 103. In the present embodiment, the data represented in Table D is sent to the data storage server 102 as / data / device / DEVICE_ENVIRONMENT_ *. It is stored in the path indicated by csv.

  The country code string is a code representing the country in which the device connected to the business server group 104 is operating. The data generation date / time column is the date / time when the in-machine temperature, the in-machine humidity, and the fixing roller surface temperature are measured and recorded. The data transmission date and time column is the date and time when the device transmitted the in-device environment data to the business server group 104. The model column is a code for identifying the model of the device. The machine number column is an ID (identifier) of the device. The in-machine temperature column is the temperature inside the device. The in-machine humidity column is the humidity inside the device. The fixing roller surface temperature column is a surface temperature of the fixing roller in the device.

In the in-device environment data, the values of the fixing roller surface temperature columns in the second and sixth rows are abnormally high. This is an example in which the temperature measurement device in the device is out of order and the data is not correct because no parts have been exchanged by the service person. The ETL management server 101 executes an ETL that converts all the invalid data into null values.

Table E is an example of in-device environment data as a result of the ETL management server 101 executing ETL. In Table E, it can be seen that the data in each column of the in-machine temperature, the in-machine humidity, and the fixing roller surface temperature in the second and seventh rows are null values as compared with Table D. When an abnormal value as shown in Table D is mixed in the data, a correct analysis result cannot be obtained at the time of analysis by the data use server group 106 and the analysis terminal 107. For example, outlier detection by statistical analysis cannot be performed. Therefore, by setting the null value in advance by the ETL by the ETL management server 101, it can be excluded from the analysis target in the data use server group 106 and the analysis terminal 107, and improvement in accuracy of the analysis result can be expected.
The above is an example of data stored in the data storage unit 501 shown in FIG.

  Returning to the description of the data storage server 102 in FIG. The data transfer unit 502 transfers the data stored in the data storage unit 501 in response to a data input / output request from the ETL management server 101 and a data input request from the data collection server 103. Further, the data stored in the data storage unit 501 is provided in response to a data output request from an analysis system that performs analysis such as the data use server group 106 and the analysis terminal 107. The data transfer protocol is, for example, HTTP, FTP, database connection, or the like, but is not limited thereto.

FIG. 5 is a software block diagram for explaining functional modules provided by the data collection server 103.
The control unit 401 periodically checks the collection rule management unit 404 to check whether there is data to be collected. If there is data to be collected, the data is collected from the business server group 104 and stored in the data storage server 102.

The data collection unit 402 acquires data from the business server group 104. The protocol used for data acquisition follows the protocol provided by the business server group 104. The data storage unit 403 stores the data acquired by the data collection unit 402 in the data storage server 102. The protocol for storing data in the data storage unit 403 follows the data storage server 102. The collection rule management unit 404 is a database that manages collection rules for collecting data from the business server group 104. Here, the collection rule will be described using Table F.

The schedule column represents a schedule for executing data collection. For example, in Table F, what is indicated as Per Day is a schedule for collecting data every day. Also, what is indicated as Per 3hour is a schedule for collecting data every 3 hours. It is desirable to collect data according to the frequency and schedule at which the business server group 104 provides data.
The acquisition destination column indicates the end point of the business server group 104 that is the data acquisition destination. The first line exemplifies the acquisition source by HTTP, and the second line exemplifies the acquisition destination by FTP.

The accumulation path rule column indicates a storage destination path to the data accumulation server 102. When storing data, it is desirable to insert the date and time of acquisition as a character string at the location indicated by *. For example, according to the collection rule shown in the first row, data acquired on September 30, 2014 is / data / device / DEVICE_ALARM_20140930. It is stored in a path called csv.
The protocol column indicates a protocol used when the data collection unit 402 collects data from the business server group 104. In this way, data can be acquired from various business server groups 104 by switching protocols.

Next, ETL execution by the ETL management server 101 will be described with reference to the flowchart of FIG.
The control unit 301 of the ETL management server 101 periodically calls up the processing from step S801 to step S806. Although the period is not limited, it is about several minutes in this embodiment.

In step S801, the control unit 301 of the ETL management server 101 refers to the rule by inquiring of the rule management unit 302.
In step S802, the control unit 301 of the ETL management server 101 applies the processing in steps S803 to S805 to the rule that matches the schedule. For example, in Table A, if the rule is described as Per Day in the schedule column, it is assumed that the rule matches the schedule when one or more days have passed since the previous ETL execution. The determination of schedule match may be implemented as an original function, or a Crontab function in Linux (registered trademark) or a task scheduler function in Windows (registered trademark) may be used.

In step S803, the control unit 301 of the ETL management server 101 inquires of the data storage server 102 and acquires input data before ETL execution.
In step S804, the control unit 301 of the ETL management server 101 calls the ETL execution unit 307. The ETL execution unit 307 acquires the ETL execution file stored in the execution file storage unit 306 and executes ETL on the input data. Here, the ETL execution file is stored in the path described in the ETL execution file path column of Table A on the execution file storage unit 306.

  In step S <b> 805, the control unit 301 of the ETL management server 101 transfers the output data obtained by executing the ETL to the data storage server 102. The data storage server 102 receives the output data and stores it in the data storage unit 501. At this time, the storage destination in the data storage unit 501 follows the output path rule string in Table A.

  As described above, the ETL management server 101 continuously executes ETL on data acquired by the data collection server 103 by periodically executing ETL according to the rules stored in the rule management unit 302. be able to.

  Next, processing when the user of the management terminal 105 presses the reflection button 603 on the list screen 600 provided by the rule management unit 302 of the ETL management server 101 will be described with reference to the flowchart of FIG. Note that the same processing is applied when the OK button 703 on the new addition screen 700 is pressed.

In step S <b> 901, the control unit 301 of the ETL management server 101 registers a newly registered or changed rule in the rule management unit 302.
In step S902, the rule management unit 302 of the ETL management server 101 determines whether a newly registered or changed rule is designated primary. If the primary is designated, the process branches to step S903. If the primary is not designated, the process of this flowchart is terminated.
In step S903, the control unit 301 of the ETL management server 101 acquires all data that matches the input path rule of Table A among the input data stored in the data storage server 102 in the past. Then, the control unit 301 of the ETL management server 101 calls the ETL execution unit 307. The ETL execution unit 307 executes the ETL execution file acquired from the execution file storage unit 306 with respect to all the input data acquired in step S903.
In step S <b> 904, the control unit 301 of the ETL management server 101 transfers all output data for which ETL has been executed in step S <b> 903 to the data storage server 102. The data storage server 102 receives the output data and stores it in the data storage unit 501. At this time, the storage destination follows the output path rule of Table A.

  When a newly registered or changed rule is designated as primary, if the input data acquired in step S903 is the result of executing another rule, the related rule is also set in step S903. -S904 may be executed. For example, in Table A, the rule on the 5th row is designated primary, and the input path rule on the 5th row matches the output path rule on the 4th row. In such a case, even if the fourth row rule is not designated as primary, the fourth row rule may be executed on data accumulated in the past.

  As described above, when the user adds a rule via the management terminal 105 by the process of FIG. 9, if the rule to be added is a rule designated primary, all the data accumulated in the past ETL can be automatically executed for For example, when a second rule related to the first rule already managed by the rule management unit 302 is newly added, the processing is performed according to the first rule before the second rule is added. Processing according to the second rule can also be performed on the input data. As a result, two related rules can be executed on input data in the same period, so that it is possible to always provide highly reliable and useful data. In the conventional technique, it is necessary to manually execute the ETL and manage the data as a result, but this embodiment can reduce the management effort.

Next, processing for deleting output data (non-primary data) not designated as primary after a predetermined period will be described with reference to the flowchart of FIG.
The control unit 301 of the ETL management server 101 periodically calls up the processing from step S1001 to step S1006. Although the period is not limited, it is about several minutes in this embodiment.

In step S <b> 1001, the control unit 301 of the ETL management server 101 repeats the processing of steps S <b> 1002 to S <b> 1005 for all output data as a result of executing the ETL stored in the data storage server 102.
In step S1002, the control unit 301 of the ETL management server 101 refers to the generation date and time of output data. The generation date and time of output data can be obtained by referring to the properties of the output data file.
In step S1003, the control unit 301 of the ETL management server 101 obtains a rule for generating output data by referring to the rule management unit 302 and referring to the rule. For example, if the output data is output path rule / data / device / etl / DEVICE_ALARM_ETL_TRANSFORM_RCD_2_ *. If it matches csv, it is found that the output data is generated by the rule in the third row of Table A.
In step S1004, the control unit 301 of the ETL management server 101 determines whether the output data is a deletion target. What is to be deleted is that the generation date and time acquired in step S1002 is in the past for a certain period and the rule acquired in step S1003 is not designated as primary. Here, for example, a period of about one month is assumed as being past for a certain period or longer, but the present invention is not limited to this, and may be set to a predetermined time. The period to be deleted is preferably set in advance according to the acquisition frequency of the analysis terminal 107 that uses data and the data use server group 106. If it is a deletion target, the process proceeds to step S1005. If it is not a deletion target, the process branches to step S1006.
In step S1006, the control unit 301 of the ETL management server 101 deletes the output data file to be deleted.

  As described above, the ETL management server 101 automatically deletes data that is not designated as primary by the processing of FIG. This is because the analysis terminal 107 and the user of the data use server group 106 need to continue to use data that is not designated as primary, but it is not primary data and does not need to be retained permanently. As described above, it is possible to reduce the data storage capacity without affecting the existing user or system while allowing the primary data to be always acquired.

  As described above, according to the present embodiment, when adding a rule for executing an ETL, the administrator does not need to manually execute the ETL, and always provides output data of a highly reliable ETL execution result. Can do.

(Second Embodiment)
Next, an information processing system according to the second embodiment of the present invention will be described. In the first embodiment, when an ETL rule is added, it is possible to automatically execute ETL on data accumulated in the past. In the present embodiment, a method for executing the processing executed in the first embodiment on the number of servers desired by the administrator will be described. In addition, the same code | symbol is attached | subjected to the thing of the structure same as 1st Embodiment, and description is abbreviate | omitted.

FIG. 11 is a diagram showing a system configuration according to the present embodiment.
The ETL execution server group 108 is a server group for executing ETL. The ETL management server 101 can execute the ETL for the input data stored in the data storage server 102 in the past executed at the time of adding the rule to the ETL execution server group 108 in a short time. Since application logs and device data accumulated in the past are very large, it is desirable to execute ETL for data accumulated in the past in a plurality of servers.

FIG. 12 is a diagram showing a server number designation screen 1200 on which the administrator sets the number of servers that execute ETL.
The server number designation screen 1200 is displayed when the administrator presses the reflection button 603 on the list screen 600 provided by the rule management unit 302 of the ETL management server 101 or when the OK button 703 is pressed on the new addition screen 700. Is done.

The server number input field 1201 is an input field for the administrator to specify a desired number of servers for executing ETL. The administrator designates the number of servers in consideration of the start deadline for using data output based on the added ETL rule and the cost of starting a plurality of servers.
An OK button 1202 is a button for starting ETL for data accumulated in the past. When the administrator presses the OK button 1202, the ETL shown in the flowchart of FIG. 9 is started.

  As described above, according to the present embodiment, it is possible to execute ETL on data accumulated in the past with the number of servers designated by the administrator, and execution of ETL can be completed in a short time.

  Next, a system according to a third embodiment of the present invention will be described. In the second embodiment, it is possible to execute ETL on data accumulated in the past in the server of the number of managers desired. On the other hand, in this embodiment, when executing ETL on data accumulated in the past, the administrator inputs the time limit until ETL is completed, so that the number of ETL execution servers is automatically set. A method for calculating and executing ETL will be described. In addition, the same code | symbol is attached | subjected to the thing of the same structure as 1st and 2nd embodiment, and description is abbreviate | omitted.

FIG. 13 is a diagram showing an ETL time limit designation screen 1300 for setting a time limit until the administrator ends the ETL.
The ETL deadline designation screen 1300 is displayed when the administrator presses the reflection button 603 on the list screen 600 provided by the rule management unit 302 of the ETL management server 101 or when the OK button 703 is pressed on the new addition screen 700. Is done.
The time limit input field 1301 is an input field for designating a time limit until the administrator finishes the ETL execution. The administrator can input the time limit according to the start time limit for using the data output based on the added ETL rule.
The calculation result 1302 is a screen that displays the number of servers necessary for completing the ETL by the time limit specified by the administrator in the time limit input field 1301.
An OK button 1303 is a button for starting ETL for data accumulated in the past. When the administrator presses an OK button 1303, the ETL shown in the flowchart of FIG. 9 is started with the calculated number of servers.
The time required for ETL execution varies depending on the processing contents of the ETL execution file. Therefore, the number of servers required to complete the ETL also differs depending on the processing content of the ETL executable file. Here, a method for calculating the number of servers will be described below.

FIG. 14 is a flowchart for explaining processing for calculating the number of servers necessary for completing the execution of the ETL by the designated time limit.
In step S1401, the rule management unit 302 of the ETL management server 101 refers to the added rule.
In step S1402, the rule management unit 302 of the ETL management server 101 acquires trial data based on the input path rule of the rule referenced in step S1401. The trial data is input data for measuring the ETL execution time in order to calculate the number of servers required for ETL execution. Arbitrary data can be acquired as trial data as long as the data is specified in the input path rule.
In step S1403, the rule management unit 302 of the ETL management server 101 calls the ETL execution unit 307 to execute ETL on the trial data.
In step S1404, the rule management unit 302 of the ETL management server 101 calculates an ETL execution time per MB from the time required for ETL execution in step S1403 and the amount of trial data.
In step S <b> 1405, the rule management unit 302 of the ETL management server 101 calls the data reading unit 303 to acquire the data amount of the ETL target data stored in the data storage server 102 in the past.
In step S1406, the number of servers required for ETL execution is calculated based on the ETL execution time per MB calculated in step S1404 and the data amount of the ETL target data acquired in step S1405. Since the ETL only needs to process the ETL for each row by a separate server, the required number of servers can be easily calculated from the ETL execution time per 1 MB. For example, if the ETL execution time per MB is 0.2 seconds and the data amount of the ETL target data is 100 GB, the execution time when executed by one unit is 0.2 * 100,000 = 10,000 seconds. If the user wants to complete the ETL within one hour (3600 seconds), 10,000 / 3600 = 2.78, so the number of servers is three.

  As described above, according to the present embodiment, by calculating the ETL execution time for the trial data, the ETL can be completed and acquired by the time limit specified by the administrator.

(Other embodiments)
The present invention supplies a program that realizes one or more functions of the above-described embodiments to a system or apparatus via a network or a storage medium, and one or more processors in a computer of the system or apparatus read and execute the program This process can be realized. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

  As mentioned above, although preferable embodiment of this invention was described, this invention is not limited to these embodiment, A various deformation | transformation and change are possible within the range of the summary.

Claims (9)

First storage means for storing data of a plurality of network devices collected via a network;
Management means for managing a plurality of rules when processing the data stored in the first storage means to obtain data suitable for analysis;
Processing means for processing the data stored in the first storage means according to the managed rules;
Second storage means for storing processed data;
Providing means for providing processed data stored in the second storage means to an analysis system for performing analysis;
The processing means performs processing according to each of the first rule and the second rule when a second rule related to the first rule managed by the management means is added,
Furthermore, the processing according to the second rule is performed on the data stored in the first storage means that has been processed according to the first rule before the second rule is added. A featured information processing system.   The information processing system according to claim 1, further comprising a setting unit configured to set the number of servers used for the processing unit to perform processing according to the managed rule.   The information processing system according to claim 2, wherein the setting unit calculates the number of the servers based on a time limit for finishing the processing according to the managed rule and a data amount of data to be processed.   The information according to claim 3, wherein the setting unit displays a screen for setting the number of servers used for performing processing according to the managed rule or a time limit for finishing the processing. Processing system. The management means manages a schedule for processing according to the rule for each rule to be managed,
The information processing system according to claim 1, wherein the processing unit performs processing according to the rule based on the schedule.   The information processing system according to claim 1, wherein the management unit displays a screen for changing the managed rule or adding a newly managed rule. .   The management means, the rule that has made the change or addition to the data that has been processed according to the managed rules before making the change or addition according to the operation on the screen displayed by the management means 7. The information processing system according to claim 6, wherein whether or not to perform processing according to is set.   In the screen displayed by the management means, if it is set not to perform the process according to the rule that has been changed or added to the data that has been processed according to the managed rule, the managed The information processing system according to claim 7, further comprising a deletion unit that deletes the data processed according to the rule after a predetermined time has elapsed. A first storage step of storing data of a plurality of network devices collected via a network;
A management step for managing a plurality of rules when processing the data stored in the first storage step to obtain data suitable for analysis;
A processing step for performing processing according to the managed rule for the data stored in the first storage step;
A second storage step for storing processed data;
Providing the processed data stored in the second storage step to an analysis system for performing analysis,
In the processing step, when a second rule related to the first rule managed in the management step is added, processing according to each of the first rule and the second rule,
Furthermore, the processing according to the second rule is also performed on the data stored in the first storage step that has been processed according to the first rule before the second rule is added. A control method of the system.