JP4634096B2 - Data processing apparatus, data processing method, and program - Google Patents

Description

  The present invention relates to a data storage technique for storing data in a predetermined data storage area.

In a data storage method in a system that needs to store continuously generated data in real time, conventionally, when the number of data in the data storage queue exceeds a certain number, a buffer prepared separately By storing several data from the buffer area in a batch so that they can be stored in the area, even when a large amount of data has occurred, the number of data storage processes has been reduced, improving processing capacity. (For example, patent document 1).
JP-A-4-126493

  The conventional data storage method improves the processing capability by collecting data in the buffer area and processing in batches when the amount of data increases, but if a large amount of data temporarily exceeds the processing capability, data arrival There is a non-negligible delay in system processing from (storage request) to storage completion. This delay directly affects the system as a subsequent data storage delay, and remains as a delay until the data storage catches up with the latest data even after the amount of data returns to the steady state. For this reason, in systems where real-time performance is more important than completeness of data and processing using the latest data periodically, if a large amount of data is generated, data storage delays will occur. In addition, there is a problem in that even after the data amount returns to the steady state, a period during which the latest data cannot be processed occurs due to the delay.

  The present invention has been made to solve the above-described problems, and by adjusting the amount of stored data when a large amount of data is generated in continuous storage processing of data that occurs continuously in real time. The main object is to obtain a data storage system that prevents data storage delays from spreading to subsequent processing, and that guarantees data storage without delay in a steady state other than when a large amount of data occurs.

The data processing apparatus according to the present invention
A data storage section that is a data storage area;
A data acquisition unit that acquires storage candidate data that is candidates for storage in the data storage unit;
It is determined whether the storage candidate data acquired by the data acquisition unit can be stored in the data storage unit within a predetermined time limit, and the storage candidate data is stored when the storage candidate data can be stored within the time limit A data storage adjustment unit that designates the storage target data to be processed and suspends the storage candidate data storage processing when the storage candidate data cannot be stored within the time limit;
And a data storage processing unit that stores data to be stored designated by the data storage adjustment unit in the data storage unit.

  According to the present invention, when a large amount of data occurs, it is possible to detect the possibility of a delay in data storage and to suspend the storage of a part of the large amount of data. Can be prevented, and data storage without delay in a steady state can be ensured.

Embodiment 1 FIG.
FIG. 1 is a diagram illustrating a configuration example of a data storage system (data processing apparatus) 100 according to the present embodiment.

  In FIG. 1, a data storage system 100 receives a generated data, a data input unit 101, analyzes data contents, converts data into an actual storage format, and adjusts the amount of stored data. A data amount adjustment unit 103 that specifies data to be stored in the database 109, system setting information 104 that is referred to when adjusting the data amount, a queue control unit 106 that manages a queue of data blocks to be stored, and data A data writing unit 108 for writing data and a database 109 which is a data storage area.

  The data input unit 101 is an example of a data acquisition unit, the data amount adjustment unit 103 is an example of a data storage adjustment unit, the data writing unit 108 is an example of a data storage processing unit, and the database 109 is a data storage unit. It is an example.

  The reserved data column 105 is a column of data blocks whose data storage order has been postponed (held) by the data amount adjusting unit 103, and is managed by the data amount adjusting unit 103. The write data string 107 is a queue of data blocks that are designated as a storage processing target in the database 109 by the data amount adjustment unit 103 and wait for writing to the database 109, and are managed by the queue control unit 106.

  Note that data that is input by the data input unit 101 and that has not been determined by the data amount adjustment unit 103 as to be allocated to the pending data sequence 105 or the write data sequence 107 is data that is a candidate for storage in the database 109. Yes, corresponding to storage candidate data. The data in the write data string 107 corresponds to storage target data.

  The system setting information 104 includes data write time information necessary for the data writing unit 108 to write the contents of the unit data block to the database 109 and the time from when the data arrives at the data input unit 101 until it is written to the database 109. The delay allowable time (time limit) information indicating the maximum time in which the delay is allowed is included.

  Next, the operation will be described.

  First, as a preparatory stage before the operation of the data storage system 100, a data writing time and a delay allowable time (time limit) are set in the system setting information 104. The value of the data writing time is derived by the system administrator (user) in consideration of the system configuration and the H / W (Hard Hardware) specification to be executed. Further, statistical information on the data writing time may be collected using test data or the like and automatically derived based on the statistical information. The value of the allowable delay time is derived by the system administrator (user) based on the data writing time and the system configuration.

  Next, the flow of processing from the reception of data at the data input unit 101 in the data storage system 100 to the writing of data into the database 109 will be described with reference to the flowcharts of FIGS.

  FIG. 2 is a flowchart showing a data input procedure (data acquisition step) by the data input unit 101 and the data analysis / normalization unit 102 of the present embodiment.

  In step 121, the data input unit 101 receives input data to be stored in the database 109. Depending on the system configuration, the data input unit 101 may receive input data via a network, or may directly receive data from a data generation source. The input data is designated as an instruction such as an SQL sentence, or is passed to the data input unit 101 as data in a predetermined format. The data input unit 101 may receive a plurality of input data at a time. The data input unit 101 adds acquisition time (time stamp) information when data is received, and passes the input data to the data analysis / normalization unit 102 one by one.

  In step 122, the data analysis / normalization unit 102 checks whether the input data is invalid. When the data is correct, the input data is converted into an internal data format when it is actually written in the database 109.

  In step 123, the data converted into the internal data format is sent to the data amount adjustment unit 103.

  FIG. 3 is a flowchart showing a data amount adjustment procedure (data storage adjustment step) by the data amount adjustment unit 103 of this embodiment.

  In step 131, the data amount adjustment unit 103 waits for data to arrive from the data analysis / normalization unit 102 for a predetermined time. Here, the fixed time refers to the time (for example, data write time × 1.2) in which one or more of the data blocks of the write data string 107 are written based on the data write time set in the system setting information 104. Is automatically calculated.

  If it is determined in step 132 that data has been received within a predetermined time, the process proceeds to step 133. If not, the process proceeds to step 138.

  In step 133, the data received from the data analysis / normalization unit 102 is accumulated in the data buffer and blocked. If the data fills the buffer and data for one block of the unit data block arrives, the process returns to step 134. If the data does not reach one block, the process returns to step 131.

  In step 134, the data amount adjustment unit 103 acquires the number of data blocks waiting in the write data string 107 from the queue control unit 106. Based on the number of data blocks and the data writing time of the system setting information 104, the time required to complete the writing when the data block held by the data amount adjusting unit 103 is added to the writing data string 107 (scheduled storage time) Is calculated. Here, since the amount of data for each data block is constant, the number of blocks of the waiting data block and the number of blocks of the data block to be judged are added, and this added value is multiplied by the data write time. If this is done, the estimated storage time can be obtained. For this reason, the estimated storage time is based on the data amount of the data block to be determined and the data amount of the data block waiting in the write data string.

  In step 135, the scheduled storage time is compared with the acquisition time (time stamp) of the head data of the held data block, and it is checked whether the data storage is completed within the delay allowable time (time limit) of the system setting information 104. If the data storage is completed within the allowable delay time (time limit), the process proceeds to step 136; otherwise, the process proceeds to step 137.

  The data block held in step 136 is transferred to the queue control unit 106. The queue control unit 106 adds the data block to the end of the write data string 107.

  In step 137, the held data block is added to the end of the reserved data string 105.

  If the input data does not arrive within a predetermined time in step 132, it is checked in step 138 whether a data block exists in the pending data string 105. If there is a data block in the reserved data string 105, the process proceeds to step 139.

  In step 139, the first data block is extracted from the reserved data string 105. Thereafter, the process proceeds to step 136, and the extracted data block is added to the end of the write data string 107.

  FIG. 4 is a flowchart showing a data writing procedure (data storage processing step) by the data writing unit 108 of the present embodiment.

  In step 141, the data writing unit 108 requests a data block from the queue control unit 106. When there is a data block in the write data string 107, the queue control unit 106 passes the head data block to the data write unit 108. If no data block exists, the process waits until the data block is delivered from the data amount adjustment unit 103.

  In step 142, the data writing unit 108 writes the received data block into the database 109.

  FIG. 15 is a timing chart for explaining the processing of the data storage system 100 described above.

  If input data is input within a certain period of time, if the scheduled storage time of the data block is before the storage deadline obtained by adding the delay allowable time to the acquisition time (time stamp) of the head data of the data block The data block is added to the end of the write data string. On the other hand, when the scheduled storage time of the data block is after the storage deadline, the data block is added to the pending data string. When there is no input data within a certain time, the data block of the reserved data string is added to the write data string.

  As described above, the data is blocked and evaluated whether the storage is completed within the allowable delay time. If the storage is not completed, the storage of the data block is put on hold. It is possible to detect a possibility of delay and postpone storing a part of a large amount of data. As a result, it is possible to prevent the data storage delay from spreading beyond the point in time when a large amount of data is generated, and to guarantee data storage without a delay in a steady state.

  In addition, because the system automatically detects and processes the suspension of partial data storage due to a large amount of data and the storage of the pending data when the data volume returns, an unexpected temporary increase in data without human intervention It can correspond to.

  In the above description, the allocation to the write data string or the reserved data string is performed for each data block. However, the distribution to the write data string or the reserved data string can be performed for each data. In this case, the data writing time information in the system setting information 104 is changed from information corresponding to the number of blocks to information corresponding to the data amount, and the processing of FIG. 3 is changed as shown in FIG.

  FIG. 16 is a flowchart showing a data amount adjustment procedure (data storage adjustment step) by the data amount adjustment unit 103 in such a case.

  In step 151, the data amount adjustment unit 103 waits for data to arrive from the data analysis / normalization unit 102 for a predetermined time.

  If it is determined in step 152 that data has been received within a predetermined time, the process proceeds to step 153. Otherwise, the process proceeds to step 157.

  In step 153, the data amount adjustment unit 103 acquires the data amount of data waiting in the write data string 107 from the queue control unit 106, and calculates the data amount of the input data received in step 151. Writing is completed when the input data held by the data amount adjustment unit 103 is added to the write data sequence 107 based on the data amount of the data in the write data sequence, the data amount of the input data, and the data write time of the system setting information 104 Calculate the time required for storage (scheduled storage time).

  In step 154, the scheduled storage time is compared with the acquisition time (time stamp) of the input data to check whether the data storage is completed within the delay allowable time (time limit) of the system setting information 104. If the data storage is completed within the allowable delay time (time limit), the process proceeds to step 155. Otherwise, the process proceeds to step 156.

  The input data held in step 155 is transferred to the queue control unit 106. The queue control unit 106 adds the input data to the end of the write data string 107.

  In step 156, the stored input data is added to the end of the reserved data string 105.

  If the input data does not arrive within a predetermined time in step 152, it is checked in step 157 whether there is data in the pending data string 105. If there is data in the reserved data string 105, the process proceeds to step 158.

  In step 158, the first data is extracted from the reserved data string 105. Thereafter, the process proceeds to step 155, and the extracted data is added to the end of the write data string 107.

  Thereafter, the same data writing process as in FIG. 4 is performed.

  As described above, it is possible to distribute the data to the write data string or the reserved data string.

Embodiment 2. FIG.
In the first embodiment described above, the writing of data that has not been stored within the allowable delay time is suspended, but the embodiment in which the index information of the data is managed when the storage is suspended next time. Show.

  FIG. 5 is a diagram illustrating a configuration example of the data storage system (data processing apparatus) 200 according to the present embodiment.

  In FIG. 5, a data storage system 200 receives a generated data, a data input unit 101, analyzes data contents, converts data into an actual storage format, and adjusts the amount of stored data. A data amount adjustment unit 201 that specifies data to be stored in the database 109, system setting information 202 that is referred to when adjusting the data amount, a queue control unit 106 that manages a queue of data blocks to be stored, and data A data writing unit 108 for writing data and a database 109 which is a data storage area.

  The data input unit 101 is an example of a data acquisition unit, the data amount adjustment unit 201 is an example of a data storage adjustment unit, the data writing unit 108 is an example of a data storage processing unit, and the database 109 is a data storage unit. It is an example.

  The index table 203 is a table for managing index information related to data whose data storage is suspended in the data amount adjustment unit 201, and is managed by the data amount adjustment unit 201.

  The reserved data sequence 204 is a sequence of data blocks whose data storage order has been postponed by the data amount adjustment unit 201, and is managed by the data amount adjustment unit 201. The write data string 107 is a queue of data blocks waiting to be written to the database 109, and is managed by the queue control unit 106.

  The system setting information 202 includes data write time information necessary for the data writing unit 108 to write the contents of the data block to the database 109 and the time from when the data arrives at the data input unit 101 until it is written to the database 109. Information on allowable delay time (time limit) indicating the maximum time allowed for delay and index definition information indicating column names of data for which index information needs to be created when storage is put on hold is included.

  In FIG. 5 and FIG. 1 of the first embodiment, elements having the same reference numerals perform the same operation.

  FIG. 6 is a diagram illustrating an example of the index table 203. The block ID is an identifier for uniquely identifying the data block, the table name is the name of the data for which the index definition is performed in the system setting information 202, the column name is the column name of the data for creating the index information, and the value is the actual column Is the value of By performing a search using the index column for the index table 203, it is possible to check the existence of the data in the pending data column 204.

  Next, the operation will be described.

  First, as a preparatory stage before the operation of the data storage system 200, data write time, delay allowable time (time limit), and index definition information are set in the system setting information 202. The data writing time and the allowable delay time are the same as in the first embodiment. The index definition information defines column information used for confirming the existence of data when storage is suspended for the data to be stored defined in the database 109.

  Next, the flow of data amount adjustment processing in the data storage system 200 will be described. The data input flow is the same as the procedure in FIG. 2 of the first embodiment, and the data write flow is the same as the procedure in FIG. 4 of the first embodiment.

  FIG. 7 is a flowchart showing a data amount adjustment procedure (data storage adjustment step) by the data amount adjustment unit 201 of the present embodiment.

  In step 221, the data amount adjustment unit 201 waits for data to arrive from the data analysis / normalization unit 102 for a predetermined time. Here, the fixed time is the same as that in step 131 of the first embodiment.

  If it is determined in step 222 that data has been received within a predetermined time, the process proceeds to step 223, and if not, the process proceeds to step 228.

  In step 223, the data received from the data analysis / normalization unit 102 is accumulated in the data buffer and blocked. In addition to the data area, an index area is prepared in the data buffer. When data having an index definition in the system setting information 202 is received, the data of the column specified as the index column is copied to the index area. If the data fills the buffer and data for one block arrives, the process returns to step 224. If the data does not reach one block, the process returns to step 221.

  In step 224, the data amount adjustment unit 201 acquires the number of data blocks waiting in the write data string 107 from the queue control unit 106. Based on the number of data blocks and the data writing time of the system setting information 202, the time required to complete the writing (scheduled storage time) when the data block held by the data amount adjustment unit 201 is added to the writing data string 107 calculate.

  In step 225, the scheduled storage time is compared with the acquisition time (time stamp) of the head data of the held data block, and it is checked whether the data storage is completed within the delay allowable time of the system setting information 202. If the data storage is completed within the allowable delay time, the process proceeds to step 226. If not, the process proceeds to step 227.

  In step 226, the held data block is transferred to the queue control unit 106. The queue control unit 106 adds the data block to the end of the write data string 107.

  In step 227, information on the index area of the stored data block is added to the index table 203 together with the block ID, and the data block is added to the end of the reserved data string 204.

  If the input data does not arrive within a predetermined time in step 222, it is checked in step 228 whether a data block exists in the pending data string 204. If there is a data block in the reserved data string 204, the process proceeds to step 229.

  In step 229, the first data block is extracted from the reserved data string 204, information matching the block ID of the data block extracted from the index table 203 is deleted, and the process proceeds to step 226.

  As described above, the data is blocked and evaluated whether the storage is completed within the allowable delay time. If the storage is not completed, the storage of the data block is put on hold. It is possible to detect a possibility of delay and postpone storing a part of a large amount of data. As a result, it is possible to prevent the data storage delay from spreading beyond the point in time when a large amount of data is generated, and to guarantee data storage without a delay in a steady state.

  In addition, since the index information of the data that is pending storage is managed, if there is no data in the data search, whether the storage just arrived or does not exist Can be screened for absence.

  In addition, because the system automatically detects and processes the suspension of partial data storage due to a large amount of data and the storage of the pending data when the amount of data returns, there is an unexpected temporary increase in data without human intervention. It can correspond to.

  In the above description, the allocation to the write data string or the reserved data string is performed for each data block. However, the distribution to the write data string or the reserved data string can be performed for each data. In this case, the data ID for each data is used instead of the block ID of the index table, and the processing of FIG. 7 is changed as shown in FIG.

  FIG. 17 is a flowchart showing a data amount adjustment procedure (data storage adjustment step) by the data amount adjustment unit 201 in this case.

  In step 231, the data amount adjustment unit 201 waits for data to arrive from the data analysis / normalization unit 102 for a predetermined time.

  If it is determined in step 232 that data has been received within a predetermined time, the process proceeds to step 233. If not, the process proceeds to step 237.

  In step 233, the data amount adjustment unit 201 obtains the data amount of data waiting in the write data string 107 from the queue control unit 106 and calculates the data amount of input data input in step 231. Writing is completed when the input data held by the data amount adjustment unit 201 is added to the write data sequence 107 based on the data amount of the data in the write data sequence, the data amount of the input data, and the data write time of the system setting information 202. The time required for storage (scheduled storage time) is calculated.

  In step 234, the scheduled storage time is compared with the acquisition time (time stamp) of the held input data, and it is checked whether the data storage is completed within the delay allowable time of the system setting information 202. If the data storage is completed within the allowable delay time, the process proceeds to step 235. Otherwise, the process proceeds to step 236.

  In step 235, the stored input data is transferred to the queue control unit 106. The queue control unit 106 adds input data to the end of the write data string 107.

  In step 236, the information on the index area of the stored input data is added to the index table 203 together with the data ID, and the input data is added to the end of the pending data string 204.

  If no input data arrives within a predetermined time in step 232, it is checked in step 237 whether data exists in the pending data string 204. If there is data in the pending data string 204, the process proceeds to step 238.

  In step 238, the first data is extracted from the reserved data string 204, information that matches the data ID of the data extracted from the index table 203 is deleted, and the process proceeds to step 235.

  Thereafter, the same data writing process as in FIG. 4 is performed.

  As described above, it is possible to distribute the data to the write data string or the reserved data string.

Embodiment 3 FIG.
In the first embodiment described above, the input data to be stored is received on the memory. Next, an embodiment in which the input data is received as a file will be described.

  FIG. 8 is a diagram illustrating a configuration example of a data storage system (data processing apparatus) 300 according to the present embodiment.

  In FIG. 8, the data storage system 300 receives a data file 302 in which generated data is recorded, adjusts the amount of stored data, and adjusts the amount of data that specifies the data to be stored in the database 109. Unit 303, data setting / reference unit 305 that reads input data from system setting information 104 and data file 302 that are referred to when adjusting the amount of data, analyzes data contents, and converts the data into a storage format; A queue control unit 106 that manages a queue of data blocks to be processed, a data writing unit 108 that writes data, and a database 109 that is a data storage area. The hold list 304 is a list in which file names of data whose data storage order has been postponed by the data amount adjustment unit 303 are arranged in order of file reception time, and are managed by the data amount adjustment unit 303. The write data string 107 is a queue of data blocks waiting to be written to the database 109, and is managed by the queue control unit 106. In addition, the same operation | movement is carried out about the element of the same code | symbol in FIG. 8 and FIG. 1 of Embodiment 1. FIG.

  The data file 302 is a file in which input data to be stored in the database 109 is recorded. It is created according to a predetermined file naming rule and data recording format. The maximum number of data recorded in one file is determined by the system, and the data in one file becomes one data block when writing to the database 109.

  The data reception unit 301 and the data analysis / normalization unit 305 are examples of data acquisition units, the data amount adjustment unit 303 is an example of data storage adjustment unit, and the data writing unit 108 is an example of data storage processing unit. Yes, the database 109 is an example of a data storage unit.

  Next, the operation will be described.

  First, as a preparatory stage before the operation of the data storage system 300, a data writing time and a delay allowable time are set in the system setting information 104. The setting contents are the same as in the first embodiment.

  Next, the flow of processing from data reception in the data receiving unit 301 in the data storage system 300 to data block addition to the write data string 107 in the data amount adjustment unit 303 will be described with reference to the flowcharts of FIGS. 9 and 10. . Note that the flow of writing data in the write data string 107 is the same as the procedure in FIG. 4 of the first embodiment.

  FIG. 9 is a flowchart showing a data reception procedure by the data receiving unit 301 of the present embodiment.

  In step 321, the data receiving unit 301 detects that the data file 302 has been generated as input data. The data file 302 is transferred and generated from the data generation source via the network. Or an application program etc. may create directly. The data file 302 is detected by the data receiving unit 301 periodically monitoring a specific location in the file system and checking whether a new file has been generated. A program for generating the data file 302 (file transfer program, application program, etc.) may notify the data receiving unit 301 after the file is generated.

  In step 322, information regarding the detected data file 302 is collected. The storage destination data is specified from the file name and the like, and an approximate value of the write data amount is obtained from the file size. The acquired information is added with acquisition time (time stamp) information when the data file 302 is detected to obtain file information of the input data. The file information indicates the data amount of data that is a candidate for storage, and the file information corresponds to the data amount information.

  In step 323, the file information of the data file 302 is passed to the data amount adjustment unit 303.

  FIG. 10 is a flowchart showing a data amount adjustment procedure by the data amount adjustment unit 303 of the present embodiment.

  In step 331, the data amount adjustment unit 303 waits for the arrival of file information from the data reception unit 301 for a predetermined time. Here, the fixed time is the same as that in step 131 of the first embodiment. If the file information has arrived, the process proceeds to step 332; if not, the process proceeds to step 336.

  In step 332, the data amount adjustment unit 303 acquires the number of data blocks waiting in the current write data string 107 from the queue control unit 106. Necessary to complete writing when a data block obtained from the data file 302 is added to the write data string 107 using the number of data blocks, the amount of data indicated in the file information, and the data write time of the system setting information 104 Time (scheduled storage time) is calculated.

  In step 333, the scheduled storage time is compared with the file information acquisition time (time stamp), and it is checked whether or not the data storage is completed within the delay allowable time set in the system setting information 104. If the data storage is completed within the allowable delay time, the process proceeds to step 334; otherwise, the process proceeds to step 335.

  In step 334, the data amount adjustment unit 303 passes the file information to the data analysis / normalization unit 305 to request acquisition of target data. The data analysis / normalization unit 305 opens the data file 302 based on the file information and reads the data. The data is inspected for illegal data, converted to an internal data format when actually written to the database 109, and copied to the data buffer. All data in the data file 302 is processed to create a data block, and the data block is passed to the data amount adjustment unit 303.

  In step 338, the data amount adjustment unit 303 passes the data block to the queue control unit 106, and the queue control unit 106 adds the data block to the end of the write data string 107.

  On the other hand, if it is determined in step 333 that the storage is not completed within the allowable delay time, the file information is added to the end of the hold list 304 in step 335.

  If the file information does not arrive within a predetermined time in step 331, the data amount adjustment unit 303 checks whether the file information exists in the hold list 304 in step 336. If file information exists, the process proceeds to step 337.

  In step 337, the file information at the head of the hold list 304 is extracted, and the process proceeds to step 334.

  As described above, it is evaluated whether the file data can be stored within the allowable delay time, and if it cannot be stored, the data storage process of the file is suspended, so that a large amount of data can be delayed. It is possible to postpone storing a part of a large amount of data by detecting the sex. As a result, it is possible to prevent the data storage delay from spreading beyond the point in time when a large amount of data is generated, and to guarantee data storage without a delay in a steady state.

  In addition, since the data whose storage is suspended is stored as a file as it is, it is possible to save system resources necessary for executing processing such as memory.

  In addition, because the system automatically detects and processes the suspension of partial data storage due to a large amount of data and the storage of the pending data when the amount of data returns, there is an unexpected temporary increase in data without human intervention. It can correspond to.

Embodiment 4 FIG.
In the first embodiment described above, data for which writing is suspended is performed in units of data blocks. Next, an embodiment in which data is performed in units of data will be described.

  FIG. 11 is a diagram illustrating a configuration example of a data storage system (data processing apparatus) 400 according to the present embodiment.

  In FIG. 11, a data storage system 400 receives a generated data, a data input unit 401, analyzes the contents of the data, converts the data into an actual storage format, and adjusts the amount of stored data. A data amount adjustment unit 403 that specifies data to be stored in the database 109, system setting information 404 that is referred to during data classification and data amount adjustment, a data writing unit 406 that writes data, and data storage It has a database 109 that is a region. The reserved data buffer 405 is a buffer that stores data whose data storage order has been postponed by the data amount adjustment unit 403.

  The data input unit 401 is an example of a data acquisition unit, the data amount adjustment unit 403 is an example of a data storage adjustment unit, the data writing unit 406 is an example of a data storage processing unit, and the database 109 is a data storage unit. It is an example.

  The system setting information 404 includes adjustment unit time information, maximum data amount information, and data classification condition information. The adjustment unit time indicates a unit time interval of processing when adjusting the amount of data to be stored, and adjustment processing of the data amount is performed every time. The maximum data amount indicates the amount of data stored per adjustment unit time. The maximum data amount corresponds to the limit data amount. The data classification condition is a condition definition used to classify data in order to classify input data and adjust the data amount for each data system. The definition contents of the data classification conditions include, for example, the column names of data used in the judgment of the classification, the total number of classifications, the ID of each classification, the range of column values in each classification, and the ratio of the amount of stored data in each classification (maximum The ratio of the category data to the data volume).

  In addition, the same operation | movement is carried out about the element of the same code | symbol in FIG. 11 and FIG. 1 of Embodiment 1. FIG.

  FIG. 12 is a diagram showing an example when the data classification condition of the system setting information 404 is defined in a table. The number of data in the table name 410 represents the total number of categories. A value range 411 indicates a range of column values in each section, and is used to determine whether data corresponds to a section by a simple operation such as comparison of values. The ratio 412 is a relative data amount within the adjustment unit time as a ratio of the stored data amount of each section.

  Next, the operation will be described.

  First, as a preparatory stage before the operation of the data storage system 400, an adjustment unit time, a maximum data amount, and a data classification condition are set in the system setting information 404. The value of the adjustment unit time is derived by the system administrator (user) in consideration of the system configuration to be executed and the H / W specification. The value of the maximum data amount is derived by a system administrator (user) based on the adjustment unit time, H / W specification, etc., and the amount of data that can be actually stored in the system. The data classification condition is defined by the system administrator (user) based on information about the generated data.

  At the start of system operation, the data amount adjustment unit 403 calculates the maximum data amount in the adjustment unit time in each data division from the ratio of the maximum data amount and the stored data amount of each division in the data classification condition. Note that the maximum data amount for each segment corresponds to the limited data amount for each segment.

  Next, the flow of processing from the reception of data at the data input unit 401 in the data storage system 400 to the writing of data into the database 109 will be described with reference to the flowcharts of FIGS.

  FIG. 13 is a flowchart showing a data input procedure (data acquisition step) by the data input unit 401 and the data analysis / normalization unit 402 of this embodiment.

  In step 421, the data input unit 401 receives input data to be stored in the database 109. Depending on the system configuration, the data input unit 401 may receive input data via a network, or may directly receive data from a data generation source. The input data is designated as an instruction such as an SQL sentence, or is passed to the data input unit 401 as data in a predetermined format. The data input unit 401 may receive a plurality of input data at a time. The data input unit 401 passes the input data to the data analysis / normalization unit 402 one by one.

  In step 422, the data analysis / normalization unit 402 checks whether the input data is invalid. When the data is correct, the input data is converted into an internal data format when it is actually written in the database 109.

  In step 423, the input data is classified based on the data classification condition of the system setting information 404. The category to which the input data belongs is determined, and information on the category ID is added to the data.

  In step 424, the data converted into the internal data format is sent to the data amount adjustment unit 403.

  FIG. 14 is a flowchart showing a data amount adjustment procedure (data storage adjustment step) by the data amount adjustment unit 403 of this embodiment.

  In step 431, the data amount adjustment unit 403 waits for the arrival of data from the data analysis / normalization unit 402 for the adjustment unit time.

  If data is received within the adjustment unit time in step 432, the process proceeds to step 433, and if not received, the process proceeds to step 437.

  In step 433, the data division ID received from the data analysis / normalization unit 402 is referred to, and it is checked whether or not the total data amount of the division has reached the division maximum data amount during the adjustment unit time. If the data has not yet reached the segment maximum data amount, the process proceeds to step 434. If the data has already been reached, the process proceeds to step 435.

  In step 434, the data is transferred to the data writing unit 406 and the process proceeds to step 436.

  In step 435, the data is added to the end of the pending data buffer 405, and the process proceeds to step 436.

  In step 436, it is checked whether the processing time exceeds the adjustment unit time. If not, the process returns to step 431. If so, the process proceeds to step 437.

  In step 437, it is checked whether the total data amount (stored data amount) passed to the data writing unit 406 within the adjustment unit time has reached the maximum data amount (restricted data amount). If not, the process proceeds to step 438.

  In step 438, it is checked whether there is data in the pending data buffer 405. If there is data, the process proceeds to step 439.

  In step 439, the data in the hold data buffer 405 is transferred to the data writing unit 406, and the data is deleted from the hold data buffer 405. The data is transferred to the data writing unit 406 until there is no data in the reserved data buffer 405 or the stored data amount reaches the maximum data amount.

  The data transferred to the data writing unit 406 in step 434 and step 439 is stored in the database 109 by the data writing unit 406. At this time, the data may be written one by one every time the data is transferred, or may be buffered and written by the data writing unit 406.

  In the example of FIGS. 13 and 14, the input data is divided into sections, and the data to be written is determined by comparing the total data amount and the maximum section data amount for each section. However, the input data is not divided into sections. It is also possible to determine data to be written by comparing the total data amount of the entire input data with the maximum data amount.

  Further, in step 438 and step 439 in FIG. 14, the data write target is set in order from the first data in the hold data buffer 405 regardless of the division of the data buffered in the hold data buffer 405. When the total amount of data is less than the maximum data amount, the data buffered in the pending data buffer 405 is only subject to data in the relevant segment, or the data is subject to data writing in order from the data in the relevant segment. If there is no more data in that category, data in another category may be set as a data write target.

  As described above, data storage processing is divided every unit time, and when the maximum amount of data is exceeded, data storage is suspended, so when large amounts of data occur, the possibility of data storage delay is detected As a result, a part of a large amount of data can be stored later. As a result, it is possible to prevent the data storage delay from spreading beyond the point in time when a large amount of data is generated, and to guarantee data storage without a delay in a steady state.

  In addition, since the amount of data is adjusted every unit time, the data that is suspended due to the generation of a large amount of data is scattered on an average basis every unit time. The data averaged over time can be stored.

  Furthermore, since data is classified and the amount of data is adjusted for each category, even when a large amount of data is generated in a certain category, data storage in other categories can be guaranteed.

  And since the system automatically detects and processes the suspension of partial data storage due to a large amount of data and the storage of the pending data when the amount of data returns, an unexpected temporary increase in data without human intervention It can correspond to.

  In each of the embodiments described above, the data storage system can be realized by a computer.

  Although not shown, the data storage system includes a CPU (Central Processing Unit) that executes a program.

  For example, a CPU is connected to a ROM (Read Only Memory), a RAM (Random Access Memory), a communication board, a display device, a K / B (keyboard), a mouse, an FDD (Flexible Disk Drive), and a CDD (Compact Disc) via a bus. Drive), magnetic disk device, optical disk device, printer device, scanner device and the like.

  The RAM is an example of a volatile memory. ROM, FDD, CDD, magnetic disk device, and optical disk device are examples of nonvolatile memory. These are examples of a storage device or a storage unit.

  The data and information handled by the data storage system of each embodiment described above are stored in a storage device or storage unit, and are recorded and read out by each unit of the data storage system.

  The communication board is connected to a WAN (Wide Area Network) such as a LAN, the Internet, or ISDN.

  The magnetic disk device stores an operating system (OS), a window system, a program group, and a file group (database).

  The program group is executed by a CPU, OS, and window system.

  Each part of the data storage system may be partly or entirely configured by a program operable by a computer. Alternatively, it may be realized by firmware stored in the ROM. Alternatively, it may be implemented by software, hardware, or a combination of software, hardware, and firmware.

  The program group stores a program that causes the CPU to execute the processing described as “˜unit” in the description of the embodiment. These programs are created by computer languages, such as C language, HTML, SGML, and XML, for example.

  The program is stored in another recording medium such as a magnetic disk device, FD (Flexible Disk), optical disk, CD (compact disk), MD (mini disk), DVD (Digital Versatile Disk), and read by the CPU. And executed.

  Here, the characteristics of the data storage system described in the above embodiment will be described below.

The data storage system described in the first embodiment is
In data storage processing that occurs continuously, the delay allowance that indicates the maximum time allowed for the delay between the data write time that indicates the time required for writing the data block to the database and the time from data arrival to the database. With time,
A data input means for receiving data to be stored;
A data analysis / normalization means for analyzing the data and converting it into an actual storage format;
A data amount adjusting unit that blocks data and determines whether or not the data block is completely written within the allowable delay time, and if not completed, temporarily stores data to adjust the amount of data to be written; and
Data writing means for writing the data block into the database is provided.

The data storage system described in the first embodiment is
In data storage processing that occurs continuously, the delay allowance that indicates the maximum time allowed for the delay between the data write time that indicates the time required for writing the data block to the database and the time from data arrival to the database. System setting information for storing the time, the data writing time and the delay allowable time, a pending data string for managing the data block whose storage is suspended, and a write data string that is a queue of data blocks to be written And
A data input means for receiving data to be stored;
A data analysis / normalization means for analyzing the data and converting it into an actual storage format;
Data that blocks data and determines whether or not the data block has been written within the allowable delay time. If the data block is not completed, the storage data is temporarily held in the reserved data string to adjust the number of data blocks in the write data string A quantity adjusting means;
Queue control means for managing the write data string;
Data write means for writing the data block of the write data string into a database is provided.

The data storage system described in the second embodiment is
In data storage processing that occurs continuously, the delay allowance that indicates the maximum time allowed for the delay between the data write time that indicates the time required for writing the data block to the database and the time from data arrival to the database. Index definition information indicating definition information of an index to be created for data whose storage is suspended, system setting information for storing the data write time, the delay allowable time, and the index definition information, and the index definition An index table that stores index information created for data whose storage has been suspended based on information, a pending data column that manages data blocks whose storage has been suspended, and a queue of data blocks that perform write processing A write data string, and
A data input means for receiving data to be stored;
A data analysis / normalization means for analyzing the data and converting it into an actual storage format;
The data is blocked and it is determined whether or not the data block has been written within the allowable delay time. If the data block is not completed, the number of data blocks in the write data string is adjusted by temporarily holding the data in the reserved data string. A data amount adjusting means for adding index information of the held data to the index table;
Queue control means for managing the write data string;
Data write means for writing the data block of the write data string into a database is provided.

The data storage system described in Embodiment 3 is
In data storage processing that occurs continuously, the delay allowance that indicates the maximum time allowed for the delay between the data write time that indicates the time required for writing the data block to the database and the time from data arrival to the database. System setting information for storing the time, the data writing time and the allowable delay time, a hold list for storing the file name of the data whose storage has been held,
A write data sequence that is a queue of data blocks to be written, and
Data receiving means for detecting a file containing data to be stored;
A data amount adjusting means for determining whether the data of the file is to be written within the allowable delay time and, if not completed, registering the file in the hold list to adjust the number of data blocks of the write data string;
Data analysis and normalization means to analyze the data of the file and convert it into an actual storage format, and create a data block,
Queue control means for managing the write data string;
Data write means for writing the data block of the write data string into a database is provided.

The data storage system described in the fourth embodiment is
In the storage process of data that occurs continuously, the adjustment unit time indicating the time interval for adjusting the amount of data to be stored, and the maximum data amount indicating the data amount stored in the adjustment unit time,
A data input means for receiving data to be stored;
A data analysis / normalization means for analyzing the input data and converting it into an actual storage format;
A data amount adjustment means for evaluating the amount of data and temporarily suspending data storage when the maximum amount of data is exceeded,
Data writing means for writing data into a database is provided.

The data storage system described in the fourth embodiment is
In the storage process of continuously generated data, the adjustment unit time indicating the time interval for adjusting the amount of data to be stored, the maximum data amount indicating the data amount stored in the adjustment unit time, and the data for each data system Data partition condition indicating the condition definition used to classify the data to adjust the amount, system setting information storing the adjustment time time, the maximum data amount, and the data partition condition, and storage pending A pending data buffer for storing data,
A data input means for receiving data to be stored;
Data analysis / normalization means for analyzing input data and converting it into an actual storage format, and determining data classification according to the above data classification conditions;
A data amount adjusting means for temporarily suspending data storage by evaluating the data amount for each data section and adding data to the suspend data buffer when the maximum data amount is exceeded;
Data writing means for writing data into a database is provided.

1 is a diagram illustrating a configuration example of a data storage system according to a first embodiment. FIG. 3 is a flowchart showing data input processing according to the first embodiment. FIG. 3 is a flowchart showing data amount adjustment processing according to the first embodiment. FIG. 3 is a flowchart showing write processing according to the first embodiment. FIG. 4 is a diagram illustrating a configuration example of a data storage system according to a second embodiment. FIG. 10 shows an example of an index table according to the second embodiment. FIG. 9 is a flowchart showing data amount adjustment processing according to the second embodiment. FIG. 9 is a diagram illustrating a configuration example of a data storage system according to a third embodiment. FIG. 10 is a flowchart showing input data reception processing according to the third embodiment. FIG. 10 is a flowchart showing data amount adjustment processing according to the third embodiment. FIG. 9 is a diagram illustrating a configuration example of a data storage system according to a fourth embodiment. FIG. 10 is a diagram illustrating an example of data classification conditions according to the fourth embodiment. FIG. 9 is a flowchart showing data input processing according to the fourth embodiment. FIG. 9 is a flowchart showing data amount adjustment processing according to the fourth embodiment. FIG. 3 is a timing chart illustrating data amount adjustment processing according to the first embodiment. FIG. 3 is a flowchart showing data amount adjustment processing according to the first embodiment. FIG. 9 is a flowchart showing data amount adjustment processing according to the second embodiment.

Explanation of symbols

  100 data storage system, 101 data input unit, 102 data analysis / normalization unit, 103 data amount adjustment unit, 104 system setting information, 105 reserved data sequence, 106 queue control unit, 107 write data sequence, 108 data write unit, 109 Database, 200 Data storage system, 201 Data amount adjusting unit, 202 System setting information, 203 Index table, 204 Reserved data column, 301 Data receiving unit, 302 Data file, 303 Data amount adjusting unit, 304 Reserved list, 305 Data analysis / Normalization unit, 401 data input unit, 402 data analysis / normalization unit, 403 data amount adjustment unit, 404 system setting information, 405 reserved data buffer, 406 data write unit.

Claims (13)

A data storage section that is a data storage area;
A data acquisition unit that acquires storage candidate data that is candidates for storage in the data storage unit;
It is determined whether the storage candidate data acquired by the data acquisition unit can be stored in the data storage unit within a predetermined time limit, and the storage candidate data is stored when the storage candidate data can be stored within the time limit A data storage adjustment unit that designates the storage target data to be processed and suspends the storage candidate data storage processing when the storage candidate data cannot be stored within the time limit;
Have a data storage processing unit for storing the storage target data specified by the data storage adjustment unit in the data storage unit,
The data storage adjustment unit
Indicates the attribute of storage candidate data for which storage processing has been suspended, and index information that enables confirmation whether the search target data is storage candidate data for which storage processing has been suspended or whether the search target data does not exist A data processing apparatus which records in a predetermined index table . The data processing device includes:
Based on the index information, it is confirmed whether the data to be searched is storage candidate data whose storage processing is suspended or whether the data to be searched does not exist in the data processing apparatus. The data processing apparatus according to claim 1. The data storage adjustment unit
Within a predetermined time limit based on the data amount of the storage candidate data and the data amount of the storage target data waiting for the storage processing in the data storage unit by the data storage processing unit when determining the storage candidate data The data processing apparatus according to claim 1, wherein it is determined whether storage candidate data can be stored in the data storage unit. The data storage adjustment unit
The storage candidate data for which storage processing has already been suspended is designated as storage target data when no new storage candidate data is acquired by the data acquisition unit for a certain period of time. Data processing device. The data storage processing unit
Storing the data to be stored for each predetermined unit data block in the data storage unit;
The data storage adjustment unit
The storage candidate data is input from the data acquisition unit, and it is determined whether storage candidate data for a unit data block is input from the data acquisition unit, and the storage candidate data for the unit data block is input from the data acquisition unit. Is input, it is determined whether or not the storage candidate data for the unit data block can be stored in the data storage unit within a predetermined time limit, and the storage candidate data for the unit data block within the time limit If the storage candidate data for the unit data block is designated as the storage target data to be stored and the storage candidate data for the unit data block cannot be stored within the time limit, the unit data block The data processing apparatus according to claim 1, wherein the storage processing of the storage candidate data for the remaining minutes is suspended. The data acquisition unit
Every time you get storage candidate data, measure the acquisition time,
The data storage adjustment unit
The storage candidate data is input from the data acquisition unit, and it is determined whether the input storage candidate data can be stored in the data storage unit within a predetermined time limit from the acquisition time of the storage candidate data of the data acquisition unit. The data processing apparatus according to claim 1. The data storage adjustment unit
When new storage candidate data is not acquired by the data acquisition unit for a certain period of time, storage candidate data for which storage processing has already been suspended is specified as storage target data, and storage candidate data specified as storage target data The data processing apparatus according to claim 1 , wherein the index information is deleted from the index table. The data acquisition unit
Prior to acquisition of storage candidate data, acquire data amount information indicating the amount of storage candidate data,
The data storage adjustment unit
Whether or not the storage candidate data can be stored in the data storage unit within a predetermined time limit based on the data amount of the storage candidate data indicated in the data amount information is input from the data acquisition unit The data processing apparatus according to claim 1, wherein the determination is made. The data storage adjustment unit
Based on the data amount of the storage candidate data indicated in the data amount information and the data amount of the storage target data waiting for storage processing in the data storage unit by the data storage processing unit when determining the storage candidate data 9. The data processing apparatus according to claim 8, wherein it is determined whether or not storage candidate data can be stored in the data storage unit within a predetermined time limit. The data storage adjustment unit
When it is determined whether or not the storage candidate data can be stored in the data storage unit within a predetermined time limit, if the storage candidate data can be stored within the time limit, the storage candidate data is stored as a target of the storage process. The data processing apparatus according to claim 8, wherein the data processing apparatus requests the data acquisition unit to acquire the storage candidate data specified as the storage target data and specified as the target data. The data storage processing unit
Storing the data to be stored for each predetermined unit data block in the data storage unit;
The data acquisition unit
Prior to acquisition of storage candidate data, data amount information indicating the data amount of storage candidate data for a unit data block is acquired,
The data storage adjustment unit
Data amount information is input from the data acquisition unit, and based on the data amount of the storage candidate data for the unit data block indicated in the data amount information, the storage candidate data for the unit data block is within the predetermined time limit. Storage target data for which the storage candidate data for the unit data block is subjected to storage processing when it is determined whether or not the data storage unit can store the storage candidate data for the unit data block within the time limit. 9. The data processing according to claim 8, wherein the storage candidate data for the unit data block is suspended when the storage candidate data for the unit data block cannot be stored within the time limit. apparatus. A data acquisition step of acquiring storage candidate data that is a candidate for storage in the data storage area;
It is determined whether the storage candidate data acquired in the data acquisition step can be stored in the data storage area within a predetermined time limit, and the storage candidate data is stored when the storage candidate data can be stored within the time limit A data storage adjustment step that designates storage target data to be processed and suspends storage processing of storage candidate data when the storage candidate data cannot be stored within the time limit;
Have a data storage processing step of storing the storing object data specified by the data stored adjustment step in the data storage area,
The data storage adjustment step includes:
Indicates the attribute of the storage candidate data for which storage processing has been suspended, and index information that can be used to confirm whether the search target data is storage candidate data for which storage processing has been suspended or whether the search target data does not exist A data processing method characterized by recording in a predetermined index table . A data acquisition process for acquiring storage candidate data that is a candidate for storage in the data storage area;
Determine whether the storage candidate data acquired by the data acquisition process can be stored in the data storage area within a predetermined time limit, and store the storage candidate data when the storage candidate data can be stored within the time limit Data storage adjustment processing that is designated as storage target data to be stored, and storage of the storage candidate data is suspended when the storage candidate data cannot be stored within the time limit,
Causing the computer to execute a data storage process for storing the storage target data designated by the data storage adjustment process in the data storage area ;
In the data storage adjustment process,
In the computer,
Indicates the attribute of the storage candidate data for which storage processing has been suspended, and index information that can be used to confirm whether the search target data is storage candidate data for which storage processing has been suspended or whether the search target data does not exist A program for recording in a predetermined index table .

Images