CN106156028A - A kind of data processing method and system - Google Patents

Abstract

The embodiment of the invention discloses a kind of data processing method and system, when detection has the first data file, download described first data file, described first data file is for needing the data file retransmitting；Carry out to described first data file changing loading processing, obtain the second data file；According to triggering rerun rule determine described second data file meet rerun condition when, perform the operation of reruning to described second data file.

Description

A data processing method and system

technical field

The invention relates to data warehouse technology, in particular to a data processing method and system.

Background technique

Data warehouse technology ETL is used to describe the process of extracting, cleaning, transforming, and loading data from the source to the destination, which is an important part of building a data warehouse.

ETL can be regarded as a chained workflow. Whenever interface data arrives, the interface data is cleaned, converted, loaded, and data extracted in sequence. In practical applications, traditional ETL usually performs chain operations on interface data in a timing-driven manner according to the characteristics of interface transmission, such as calculating interface A data at 8:00 every night. In this way, after an ETL task chain is completed, when interface A data changes and needs to be retransmitted, traditional ETL requires maintenance personnel to manually perform subsequent data recalculation work, such as interface data cleaning, conversion, loading, data extraction, etc. , That is to use manual intervention to deal with subsequent data recalculation work, which will bring a lot of trouble to ETL maintenance personnel.

Contents of the invention

In view of this, the embodiments of the present invention expect to provide a data processing method and system, which can intelligently complete data recalculation without manual intervention when interface data changes and needs to be retransmitted.

In order to achieve the above object, the technical solution of the embodiment of the present invention is achieved in this way:

An embodiment of the present invention provides a data processing method, the method comprising:

When detecting that there is a first data file, download the first data file, the first data file is a data file that needs to be retransmitted;

performing conversion and loading processing on the first data file to obtain a second data file;

When it is determined according to the triggering recalculation rule that the second data file satisfies the recalculation condition, a recalculation operation on the second data file is performed.

In the above scheme, the detection of the first data file includes:

Filter out the data files that meet the rules according to the preset configuration information;

judging whether the data file has been changed;

When the data file has been changed, it is determined that the data file is the first data file.

In the above solution, the conversion and loading process on the first data file to obtain the second data file includes:

performing conversion processing on the first data file to obtain a converted data file;

Loading is performed on the converted data file to obtain a second data file.

In the above solution, the converting the first data file to obtain the converted data file includes:

Acquiring a conversion strategy corresponding to the first data file;

Perform conversion processing on the first data file according to the conversion strategy to obtain a converted data file.

In the above solution, the loading processing of the converted data file to obtain the second data file includes:

Loading the converted data file and database table fields to obtain a second data file.

The embodiment of the present invention also provides a data processing system, and the system includes: a file transfer protocol FTP scanner, a file scanner and a stored procedure trigger;

The FTP scanner is configured to download the first data file when detecting the first data file, and the first data file is a data file that needs to be retransmitted;

The file scanner is configured to convert and load the first data file to obtain a second data file;

The stored procedure trigger is configured to perform a recalculation operation on the second data file when it is determined according to a trigger recalculation rule that the second data file satisfies a recalculation condition.

In the above solution, the FTP scanner is used to filter out data files conforming to the rules according to preset configuration information; determine whether the data file has changed; when the data file has changed, determine the data file The file is a first data file.

In the above solution, the file scanner includes a conversion scanner and a loading scanner;

The conversion scanner is configured to convert the first data file to obtain a converted data file;

The loading scanner is configured to load the converted data file to obtain a second data file.

In the solution above, the conversion scanner is configured to obtain a conversion strategy corresponding to the first data file; perform conversion processing on the first data file according to the conversion strategy to obtain a converted data file.

In the solution above, the loading scanner is configured to load the converted data file and database table fields to obtain a second data file.

In the data processing method and system provided by the embodiments of the present invention, when a first data file is detected, the first data file is downloaded, and the first data file is a data file that needs to be retransmitted; The file is converted and loaded to obtain the second data file; when it is determined that the second data file satisfies the recalculation condition according to the trigger recalculation rule, the recalculation operation on the second data file is executed. In this way, when the interface data changes and needs to be retransmitted, the data recalculation can be completed intelligently without manual intervention.

Description of drawings

Fig. 1 is a schematic diagram of the implementation flow of the data processing method of the embodiment of the present invention;

FIG. 2 is a schematic diagram of an implementation process for detecting the presence of a first data file according to an embodiment of the present invention;

FIG. 3 is a schematic diagram of an implementation flow for converting and loading the first data file according to an embodiment of the present invention;

4 is a schematic diagram of the composition and structure of a data processing system according to an embodiment of the present invention;

FIG. 5 is a schematic diagram of the composition and structure of a document scanner according to an embodiment of the present invention.

detailed description

In an embodiment of the present invention, when a first data file is detected, download the first data file, the first data file is a data file that needs to be retransmitted; perform conversion and loading processing on the first data file, Obtaining a second data file; performing a recalculation operation on the second data file when determining that the second data file satisfies a recalculation condition according to a triggering recalculation rule.

The present invention will be further described in detail below in conjunction with the accompanying drawings and specific embodiments.

Embodiment one

Fig. 1 is a schematic diagram of the implementation flow of the data processing method of the embodiment of the present invention. As shown in Fig. 1, the data processing method of the embodiment of the present invention includes:

Step S101: downloading the first data file when it is detected that there is a first data file;

Wherein, the first data file is a data file that needs to be retransmitted.

Specifically, a File Transfer Protocol (File Transfer Protocol, FTP) scanner needs to detect whether there is a first data file, and download the first data file when detecting that the first data file exists. Here, as shown in Figure 2, the first data file detected includes:

Step S1011: filter out the data files that meet the rules according to the preset configuration information;

Specifically, the FTP scanner invokes a file filter through predetermined configuration information to filter out data files that meet the rules. Here, the predetermined configuration information and the specific requirements for data files conforming to the rules will be preset in the system for implementing the data processing method.

Step S1012: judging whether the data file has been changed;

Specifically, the FTP scanner judges whether changes such as date and file size have occurred in the data file.

Step S1013: When the data file has been changed, determine that the data file is the first data file.

Specifically, when the data file has been changed, the downloader is invoked to download; at the same time, the information of the data file can also be recorded, and the downloaded data file can be broadcast in a file-driven manner.

Step S102: Perform conversion and loading processing on the first data file to obtain a second data file;

Specifically, as shown in FIG. 3, the conversion and loading process is performed on the first data file to obtain a second data file, including:

Step S1021: Perform conversion processing on the first data file to obtain a converted data file;

Specifically, a conversion strategy corresponding to the first data file is acquired by a conversion scanner; then, conversion processing is performed on the first data file according to the conversion strategy to obtain a converted data file.

Step S1022: Perform loading processing on the converted data file to obtain a second data file.

Specifically, the converted data file and database table fields are loaded through a loading scanner, such as one-to-one mapping and conversion processing are performed on the converted data file and database table fields to obtain the second data document.

Further, the loading scanner will change the state of the interface corresponding to the second data file, and broadcast the second data file, so as to provide the system for implementing the data processing method to perform a Scheduling services, so that the second data file can be re-calculated as soon as possible.

In practical applications, the result analyzer in the loading scanner can also be used to conduct in-depth analysis of the loading feedback result information, such as judging reading, loading, discarding and other index information, so as to make accurate results for this loading Judgment; when a high-level error occurs during this loading process, the interrupt processor will perform corresponding interrupt processing based on the feedback result of this loading to prevent table locks, deadlocks, etc. from occurring.

Step S103: When it is determined according to the triggering recalculation rule that the second data file satisfies the recalculation condition, perform a recalculation operation on the second data file.

Here, the triggering recalculation rule usually consists of configuration information, resource pool information, dependency information, and the like. Specifically, when the stored procedure trigger determines that the second data file satisfies the recalculation condition according to configuration information, resource pool information, and dependency information, it recalculates the second data file in sequence. It should be noted that, during the recalculation process, recalculation operations can be performed simultaneously for non-conflicting schedules.

In this way, through the data processing method of the embodiment of the present invention, each device in the system used to implement the data processing method can execute the file driver, and use broadcast communication to transfer the recalculated file. When the interface data changes and needs to be retransmitted, It can intelligently complete data recalculation work without manual intervention; further increase the recalculation condition judgment process to ensure the safety of recalculation.

Embodiment two

FIG. 4 is a schematic diagram of the composition and structure of a data processing system according to an embodiment of the present invention. As shown in FIG. 4 , the system includes an FTP scanner 401, a file scanner 402, and a stored procedure trigger 403;

The FTP scanner 401 is configured to download the first data file when detecting that there is a first data file, and the first data file is a data file that needs to be retransmitted;

Specifically, the FTP scanner 401 screens out data files conforming to the rules according to preset configuration information; judges whether the data file has changed; when the data file has changed, determines that the data file is the first a data file.

In practical applications, the FTP scanner 401 invokes a file filter through predetermined configuration information to filter out data files that meet the rules; When a change occurs, the downloader is invoked to download; at the same time, the information of the data file can also be recorded, and the downloaded data file can be broadcast in a file-driven manner.

The file scanner 402 is configured to convert and load the first data file to obtain a second data file;

The stored procedure trigger 403 is configured to perform a recalculation operation on the second data file when it is determined according to the trigger recalculation rule that the second data file meets a recalculation condition.

Here, the triggering recalculation rule usually consists of configuration information, resource pool information, dependency information, and the like. Specifically, when the stored procedure trigger 403 determines that the second data file satisfies the recalculation condition according to configuration information, resource pool information, and dependency information, it recalculates the second data file in sequence. It should be noted that, during the recalculation process, recalculation operations can be performed simultaneously for non-conflicting schedules.

In one embodiment, as shown in FIG. 5 , the file scanner 402 includes a conversion scanner 4021 and a loading scanner 4022;

The conversion scanner 4021 is configured to perform conversion processing on the first data file to obtain a converted data file;

Specifically, the conversion scanner 4021 acquires a conversion strategy corresponding to the first data file; and then performs conversion processing on the first data file according to the conversion strategy to obtain a converted data file.

The loading scanner 4022 is configured to load the converted data file to obtain a second data file.

Specifically, the loading scanner 4022 loads the converted data files and database table fields, such as performing one-to-one mapping and conversion processing on the converted data files and database table fields to obtain the first Two data files.

Further, the loading scanner 4022 will change the state of the interface corresponding to the second data file, and broadcast the second data file, so as to provide the system for implementing the data processing method with the second data file The service is scheduled so that the second data file can be recalculated as soon as possible.

In practical applications, the result analyzer in the loading scanner can also be used to conduct in-depth analysis of the loading feedback result information, such as judging reading, loading, discarding and other index information, so as to make accurate results for this loading Judgment; when a high-level error occurs during this loading process, the interrupt processor will perform corresponding interrupt processing based on the feedback result of this loading to prevent table locks, deadlocks, etc. from occurring.

In practical applications, each part of the data processing system provided in the embodiment of the present invention can be realized by a processor in the system, or can be realized by a specific logic circuit; Central processing unit (CPU), microprocessor (MPU), digital signal processor (DSP), or field-programmable gate array (FPGA) in the implementation.

The above descriptions are only preferred embodiments of the present invention, and are not intended to limit the protection scope of the present invention.

Claims (10)

1. a data processing method, it is characterised in that described method includes:

When detection has the first data file, downloading described first data file, described first data file is Need the data file retransmitting；

Carry out to described first data file changing loading processing, obtain the second data file；

According to triggering rerun rule determine described second data file meet rerun condition when, perform to described the The operation of reruning of two data files.

2. method according to claim 1, it is characterised in that described detection has the first data file bag Include:

Filter out legal data file according to default configuration information；

Judge whether described data file changes；

When described data file has occurred change, determine that described data file is the first data file.

3. method according to claim 1, it is characterised in that described described first data file is entered Row conversion loading processing, obtains the second data file, comprising:

Conversion process is carried out to described first data file, obtains the data file after conversion process；

Loading processing is carried out to the data file after described conversion process, obtains the second data file.

4. method according to claim 3, it is characterised in that described described first data file is entered Row conversion process, obtains the data file after conversion process, comprising:

Obtain the switching strategy corresponding to described first data file；

According to described switching strategy, conversion process is carried out to described first data file, after obtaining conversion process Data file.

5. method according to claim 3, it is characterised in that described to the number after described conversion process Carry out loading processing according to file, obtain the second data file, comprising:

Data file after described conversion process and database literary name section are carried out loading processing, obtains the second number According to file.

6. a data handling system, it is characterised in that described system includes: FTP FTP Scanner, document scanner and storing process trigger；

Described FTP scanner, for when detection has the first data file, downloads described first data file, Described first data file is for needing the data file retransmitting；

Described document scanner, for carrying out to described first data file changing loading processing, obtains second Data file；

According to triggering rule of reruning, described storing process trigger, for determining that described second data file meets During condition of reruning, perform the operation of reruning to described second data file.

7. system according to claim 6, it is characterised in that

Described FTP scanner, for filtering out legal data file according to the configuration information preset； Judge whether described data file changes；When described data file has occurred change, determine described Data file is the first data file.

8. system according to claim 6, it is characterised in that described document scanner includes that conversion is swept Retouch device and load scanner；

Described conversion scan device, for carrying out conversion process to described first data file, obtains conversion process After data file；

Described loading scanner, for carrying out loading processing to the data file after described conversion process, obtains Second data file.

9. system according to claim 8, it is characterised in that

Described conversion scan device, for obtaining the switching strategy corresponding to described first data file；According to institute State switching strategy and conversion process is carried out to described first data file, obtain the data file after conversion process.

10. system according to claim 8, it is characterised in that

Described loading scanner, for carrying out the data file after described conversion process and database literary name section Loading processing, obtains the second data file.