CN111259082A - Method for realizing full data synchronization in big data environment - Google Patents

Abstract

The invention relates to a method for realizing full data synchronization in a big data environment. The method comprises the steps of: executing a module for inserting source data, executing a module for generating intersection data, acquiring and inserting a record module to be added based on the intersection data, and generating a module to be updated based on the intersection data. Record module, update data module based on updated record, delete record module to be deleted based on intersection data. The present invention not only supports full data synchronization of single table and large data volume, but also supports business rollback during data exchange, and also supports providing an implementation method for full data synchronization of multiple tables and large data volume in an asynchronous manner. The most important thing is that after the full data synchronization is completed, it can well support the business application and data analysis of the original data.

Description

Method for realizing full data synchronization in big data environment

technical field

The invention relates to the field of computer software, in particular to the field of enterprise application, in particular to a method for realizing full data synchronization in a big data environment.

Background technique

In the field of enterprise-level data governance, metadata management is the basis of data governance. It is responsible for the integration and management of various source system data involved in the enterprise, and provides support for the development, operation and maintenance of enterprise business systems and data analysis. Metadata collection, as the basis of metadata management, is responsible for collecting the above types of metadata, integrating and processing these data and storing them in the metadata warehouse to support the unified management of metadata. Metadata collection needs to regularly update a certain category of metadata in full synchronously, that is, to collect the source system data of this category and update it to the metadata warehouse in full to ensure the quasi-real-time and validity of the data. With the development of enterprise business, more and more source systems are constructed, and the complexity of source system data is also increasing. The amount of data that needs to be collected for metadata collection is also increasing, and it supports full synchronous update of large data metadata. Data becomes the bottleneck of the problem.

In order to solve the above problems, there are currently a variety of solutions, as follows:

Comparing the above solutions, either cannot meet business needs, or there are many operation steps and low efficiency. Therefore, there is an urgent need for a stable and efficient method to support full data synchronization of large data volumes. After the full data synchronization is completed, the target metadata remains the same as The source metadata is consistent, which solves the problem of metadata collection supporting the business application and data analysis of metadata.

SUMMARY OF THE INVENTION

The main purpose of the present invention is to provide a stable and efficient method for full data synchronization supporting a large amount of data. and data analysis issues.

In order to achieve the above purpose, the present invention provides a method for realizing full data synchronization in a big data environment, the method comprising the following steps:

(1) Execute the insert source data module, which is responsible for inserting the source metadata into the newly created metadata table of the metadata warehouse in batches;

(2) Execute the generating intersection data module, combine the source metadata and the target metadata in the metadata warehouse, perform a repeatable union operation, and group the results according to the business primary key to count the entries;

(3) Obtain and insert the record module to be added based on the intersection data, connect the intersection data and the source metadata according to the business primary key, query and obtain all the data to be newly added, and insert into the target metadata table;

(4) Generate the record module to be updated based on the intersection data, connect the intersection data, source metadata, and target metadata according to the business primary key, and compare all attributes of the source metadata and all attributes of the target metadata to obtain the required updated records;

(5) Update the data module based on the updated record, connect the record to be updated and the source metadata according to the business primary key, and execute the update;

(6) Delete the record module to be deleted based on the intersection data, connect the intersection data and target metadata according to the business primary key, query to obtain all the data to be deleted, and obtain all the data to be deleted and execute the deletion.

The described inserting source data module is responsible for inserting the source metadata into the newly created metadata table of the metadata warehouse in batches, including the following steps:

(11) Determine whether there is already a table T` in the metadata warehouse. If T` already exists, it means that the last full data synchronization task was interrupted for some reasons. At this time, skip the following steps and go directly to (2) Generate secondary data Operation of the module; if T' does not exist, enter step (12);

(12) Copy and create a new metadata table (denoted as T`) in the metadata warehouse, and the table structure is completely consistent with the structure of the table (denoted as T) storing the target metadata;

(13) the metadata of the source system is inserted into the newly created metadata table T' of the metadata warehouse in batches;

(14) Add a business primary key index for the newly created metadata table T`;

(15) Check the validity of the module, if the verification is passed, enter the next module, otherwise enter (16);

(16) Delete the table T', and the task is terminated, and an error message is given.

The described generating intersection data module includes the following steps:

(21) Judge whether the table TMP_UNION already exists in the metadata warehouse, and delete it if it already exists;

(22) Combine the source metadata and the target metadata in the metadata warehouse, perform a repeatable union operation, and group the results into statistical entries according to the business primary key;

(23) the statistical data obtained by the query is stored in the newly created temporary table TMP_UNION;

(24) Create a business primary key index and a statistical entry index for the temporary table TMP_UNION respectively;

(25) Check the validity of the module, if the verification is passed, enter the next module, otherwise enter (26);

(26) Delete the temporary table TMP_UNION, and the task is terminated, and an error message is given.

The described acquisition and insertion of the recording module to be added based on the intersection data includes the following steps:

(31) Connect the intersection data and source metadata according to the primary key of the business, and if the entry is 1 is all the data that needs to be added, obtain all the data that needs to be newly added, and insert it into the target metadata table;

(32) Check the validity of the module, if the verification passes, enter the next module, otherwise enter (33);

(33) Connect the intersection data and source metadata according to the primary key of the business, and if the entry is 1 is all the data to be added,

Get all the data that needs to be newly added, and delete these data in the target metadata table;

(34) Continue to execute step (26).

The described generation of the record module to be updated based on the intersection data, connects the intersection data, source metadata, and target metadata according to the business primary key, and compares all the attributes of the source metadata and all the attributes of the target metadata to obtain the required data. Updated records, including the following steps:

(41) Connect the intersection data, source metadata, and target metadata, and compare all attributes of the source metadata and all attributes of the target metadata. The method is to first splicing all attributes of the metadata ATT_ATTRS, and then take MD5 Operation to obtain a unique hash value, and then compare the hash value MD5_T_FEATURES of the source metadata attribute and the hash value MD5_T`_FEATURES of the target metadata attribute. The status is no change, otherwise, the attribute is considered to have changed, and the attribute status is marked as changed. The data is stored in the newly created temporary table TMP_UPDATE;

(42) adding an index to the business primary key of the record table to be updated;

(43) adding an index for the attribute state of the record table to be updated;

(44) Check the validity of the module, if the verification is passed, enter the next module, otherwise enter (45);

(45) delete the temporary table TMP_UPDATE;

(45) Continue to execute step (33).

The described updating data module based on the updated record includes the following steps:

(51) connect the record table TMP_UPDATE to be updated and the source metadata table, and perform the update in the target metadata table;

The described deletion of the record module to be deleted based on the intersection data includes the following steps:

(61) Connect the intersection data and target metadata according to the primary key of the business, and at the same time the entry of 1 is all the data to be deleted, and all the data to be deleted are obtained;

(62) Delete is performed on the target metadata table.

(63) Check the validity of the module, if the check passes, delete the temporary table TMP_UPDATE;

(64) delete the temporary table TMP_UNION;

(65) Delete table T'.

So far, the full data synchronization is completed, the target metadata remains consistent with the source metadata, and the business application and data analysis of the metadata are not affected.

It can be seen from the above technical solutions that the method for realizing full data synchronization in a big data environment provided by the present invention not only provides a method for realizing full data synchronization of single table large data amount, but also supports business rollback and asynchronous mode. Provides an implementation method for full data synchronization of multiple tables and large amounts of data. The most important thing is that after the full data synchronization is completed, the target metadata remains consistent with the source metadata, and can support business applications and data analysis of the original metadata.

Description of drawings

Fig. 1 is a flow chart of full data synchronization provided by the present invention;

FIG. 2 is a full data synchronization class diagram provided by the present invention.

Detailed ways

In order to make the objectives, technical solutions and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the specific implementation steps and the accompanying drawings.

As shown in Figure 1, Figure 1 is a flow chart of full data synchronization provided by the present invention, and the flow specifically includes the following steps:

(1) Insert the source data module, which is responsible for inserting the source metadata into the newly created metadata table of the metadata warehouse in batches;

(2) Generate the intersection data module, combine the source metadata and the target metadata in the metadata warehouse, do a repeatable union operation, and group the results according to the business primary key to count the entries;

(3) Obtain and insert the record module to be added based on the intersection data, connect the intersection data and the source metadata according to the business primary key, query and obtain all the data to be newly added, and insert into the target metadata table;

(4) Generate the record module to be updated based on the intersection data, connect the intersection data, source metadata, and target metadata according to the business primary key, and compare all attributes of the source metadata and all attributes of the target metadata to obtain the required updated records;

(5) Update the data module based on the updated record, connect the record to be updated and the source metadata according to the business primary key, and execute the update;

(6) Delete the record module to be deleted based on the intersection data, connect the intersection data and target metadata according to the business primary key, query to obtain all the data to be deleted, and obtain all the data to be deleted and execute the deletion.

In a more preferred embodiment, the inserting source data module is responsible for inserting the source metadata into the newly created metadata table of the metadata warehouse in batches, including the following steps:

(11) Determine whether there is already a table T` in the metadata warehouse. If T` already exists, it means that the last full data synchronization task was interrupted for some reasons. At this time, skip the following steps and go directly to (2) Generate secondary data Operation of the module; if T' does not exist, enter step (12);

(12) Copy and create a new metadata table (denoted as T`) in the metadata warehouse, and the table structure is completely consistent with the structure of the table (denoted as T) storing the target metadata;

(13) the metadata of the source system is inserted into the newly created metadata table T' of the metadata warehouse in batches;

(14) Add a business primary key index for the newly created metadata table T`;

(15) Check the validity of the module, if the verification is passed, enter the next module, otherwise enter (16);

(16) Delete the table T', and the task is terminated, and an error message is given.

In a more preferred embodiment, the described generating intersection data module includes the following steps:

(21) Judge whether the table TMP_UNION already exists in the metadata warehouse, and delete it if it already exists;

(22) Combine the source metadata and the target metadata in the metadata warehouse, perform a repeatable union operation, and group the results into statistical entries according to the business primary key;

(23) the statistical data obtained by the query is stored in the newly created temporary table TMP_UNION;

(24) Create a business primary key index and a statistical entry index for the temporary table TMP_UNION respectively;

(25) Check the validity of the module, if the verification is passed, enter the next module, otherwise enter (26);

(26) Delete the temporary table TMP_UNION, and the task is terminated, and an error message is given.

In a more preferred embodiment, the described acquisition and insertion of the recording module to be added based on the intersection data includes the following steps:

(31) Connect the intersection data and source metadata according to the primary key of the business, and if the entry is 1 is all the data that needs to be added, obtain all the data that needs to be newly added, and insert it into the target metadata table;

(32) Check the validity of the module, if the verification passes, enter the next module, otherwise enter (33);

(33) Connect the intersection data and source metadata according to the primary key of the business, and if the entry is 1 is all the data to be added,

Get all the data that needs to be newly added, and delete these data in the target metadata table;

(34) Continue to execute step (26).

In a more preferred embodiment, the recording module to be updated is generated based on the intersection data, the intersection data, source metadata, and target metadata are connected according to the business primary key, and all attributes and target metadata of the source metadata are compared. Compare all attributes of , and get the records that need to be updated, including the following steps:

(41) Connect the intersection data, source metadata, and target metadata, and compare all attributes of the source metadata and all attributes of the target metadata. The method is to first splicing all attributes of the metadata ATT_ATTRS, separated by a specific character (such as "_") to concatenate all attribute fields, and then take the MD5 operation to obtain a unique hash value, and then compare the hash value MD5_T_FEATURES of the source metadata attribute and the hash value MD5_T`_FEATURES of the target metadata attribute. If the column values are the same, it is considered that there is no change in the attributes of the two, and the status of the marked attribute is no change; otherwise, the attribute is considered to have changed, and the status of the marked attribute is changed. Insert the query operation result into the newly created temporary table TMP_UPDATE. The records that satisfy TMP_UPDATE.MD5_T_FEATURES<>TMP_UPDATE.MD5_T`_FEATURES are the records that need to be updated;

The MD5 operation is as follows:

(42) adding an index to the business primary key of the record table to be updated;

(43) adding an index for the attribute state of the record table to be updated;

(44) Check the validity of the module, if the verification is passed, enter the next module, otherwise enter (45);

(45) delete the temporary table TMP_UPDATE;

(45) Continue to execute step (33).

In a more preferred embodiment, the described updating data module based on the updated record comprises the following steps:

(51) Connect the record table TMP_UPDATE to be updated and the source metadata table T, combined with TMP_UPDATE.

MD5_T_FEATURES<>TMP_UPDATE.MD5_T`_FEATURES condition, all records obtained from the query are updated to the T table;

In a more preferred embodiment, the described deletion of the record module to be deleted based on the intersection data includes the following steps:

(61) Connect the intersection data and target metadata according to the primary key of the business, and at the same time the entry of 1 is all the data to be deleted, and all the data to be deleted are obtained;

(62) Delete is performed on the target metadata table.

(63) Check the validity of the module, if the check passes, delete the temporary table TMP_UPDATE;

(64) delete the temporary table TMP_UNION;

(65) Delete table T'.

Preferably, in order to ensure the consistency of data operations, it is necessary to add a business rollback operation to the above modules, and the business rollback of the current module should include the business rollback of the previous module to ensure that the data business rollback is complete and the data is consistent. The specific implementation is shown in Figure 2. Figure 2 is a full data synchronization class diagram provided by the present invention. The specific description for this diagram is as follows:

(2-1) Class 101 is the realization of step 101 in FIG. 1 , class 102 is the realization of step 102 in FIG. 1 , and so on;

(2-2) In order to realize the serial execution of the steps in FIG. 1, through the decorator pattern, class 101 holds class 102, class 102 holds class 103, and so on;

(2-3) The control of the operation sequence of steps 101 and 102 in Figure 1, and the control of the business rollback operation when the operation is abnormal, are implemented in the method AbstractStep.step, and the pseudo code is as follows

Through the above design, the execution sequence of the operation steps is guaranteed, and at the same time, in the event of abnormal situations (such as abnormal data insertion, abnormal power failure during the execution process, etc.), the business rollback operation can be guaranteed, thereby ensuring data consistency.

More preferably, FIG. 1 is directed to the operation flow of one table data. When multiple tables require the same operation flow, multiple sets of the operation flow of FIG. 1 need to be implemented. In order to deal with this scenario flexibly, consider parameterizing the source metadata table T', target metadata table T and their attributes involved in Figure 1, and add the temporary tables TMP_UNION and TMP_UPDATE generated during the operation steps of Figure 1 to this execution. Session ID, becomes TMP_UNION_${SID},

TMP_UPDATE_${SID}, where ${SID} is the session ID of this execution. At this point, the SQL to be executed in each step is dynamically assembled according to the parameters. The specific implementation is as follows:

(3-1) Add an annotation to the entity of the target metadata table T, similar to the following

(3-2) When obtaining the table name, dynamically obtain the table name from the entity class based on the annotation, as follows:

(3-3) When obtaining the fields of the table, dynamically obtain the table fields from the entity class based on the annotation, as follows:

(3-4) Dynamically assemble SQL based on the table name and field name obtained from the annotation, and execute the SQL. The assembled SQL is similar to the following:

Through the above design, it is possible to flexibly cope with the scenario of multi-table full data synchronization.

More preferably, for the scenario of multi-table full data synchronization, the operations between different tables do not affect each other. In order to further improve the operation efficiency, if the software and hardware conditions allow, the asynchronous method of thread pool multi-threading can be used. To achieve multi-table full data synchronization, the operation pseudocode is as follows:

Through the above design, the operation efficiency of multi-table full data synchronization is further improved.

So far, the full data synchronization is completed, the target metadata remains consistent with the source metadata, and the business application and data analysis of the metadata are not affected.

The specific embodiments of the present invention described above are not intended to limit the application, and any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention. within.

Claims (7)

1. A method for realizing full data synchronization in a big data environment comprises the following steps:

(1) the source data inserting module is executed and is responsible for inserting source metadata into a metadata table newly created by the metadata warehouse in batches;

(2) executing a module for generating intersection data, combining the source metadata and the target metadata in the metadata warehouse, performing repeatable union operation, and grouping and counting items according to the result by a service main key;

(3) acquiring and inserting a recording module to be added based on the intersection data, connecting the intersection data and the source metadata according to the service primary key, inquiring and acquiring all data to be newly added, and inserting the data into a target metadata table;

(4) generating a record module to be updated based on the intersection data, connecting the intersection data, the source metadata and the target metadata according to the main service key, and comparing all attributes of the source metadata with all attributes of the target metadata to obtain a record to be updated;

(5) based on the updated record update data module, connecting the record to be updated and the source metadata according to the service main key, and executing the update;

(6) and deleting the recording module to be deleted based on the intersection data, connecting the intersection data and the target metadata according to the main service key, inquiring and acquiring all data to be deleted, and obtaining all data to be deleted to execute deletion.

2. The method for achieving full data synchronization in a big data environment according to claim 1, wherein the inserting source data module is responsible for inserting source metadata into the metadata table newly created by the metadata warehouse in batches, and comprises the following steps:

(11) judging whether the table T 'already exists in the metadata warehouse, if T' already exists, indicating that the last task of full data synchronization is interrupted for some reasons, skipping the following steps, and directly entering the operation of (2) generating the secondary data module; if T' does not exist, entering step (12);

(12) copying and creating a new metadata table (denoted as T') in the metadata warehouse, wherein the table structure is completely consistent with the table (denoted as T) structure for storing target metadata;

(13) inserting the metadata of the source system into a newly created metadata table T' of a metadata warehouse in batches;

(14) adding a service main key index to the newly created metadata table T';

(15) carrying out validity check on the module, if the check is passed, entering the next module, otherwise, entering (16);

(16) the table T' is deleted and the task terminates and an error prompt is given.

3. The method for implementing full data synchronization in big data environment according to claim 1, wherein said module for generating intersection data comprises the following steps:

(21) judging whether a table TMP _ UNION already exists in the metadata warehouse or not, and deleting the table if the table TMP _ UNION already exists;

(22) combining the source metadata and the target metadata in the metadata warehouse, performing repeatable union operation, and grouping and counting the result according to the main key of the service;

(23) storing the statistical data obtained by query in a newly created temporary table TMP _ UNION;

(24) respectively creating a service main key index and a statistic entry index for the TMP _ UNION;

(25) carrying out validity check on the module, entering the next module if the check is passed, and entering (26) if the check is not passed;

(26) the temporary table TMP UNION is deleted and the task terminates and an error prompt is given.

4. The method for achieving full data synchronization in big data environment according to claim 1, wherein said obtaining and inserting a record module to be added based on intersection data comprises the following steps:

(31) connecting the intersection data and the source metadata according to the main key of the service, and simultaneously, obtaining all data needing to be newly added if the item is 1, and inserting the data into the target metadata table;

(32) the module is checked for validity, if the module passes the check, the next module is entered, otherwise, the next module is entered (33);

(33) the intersection data and the source metadata are connected according to the main key of the service, and the entry of 1 is all the data to be added,

obtaining all data needing to be newly added, and deleting the data in the target metadata table;

(34) and continuing to execute the step (26).

5. The method according to claim 1, wherein the step of generating a record module to be updated based on the intersection data, connecting the intersection data, the source metadata, and the target metadata according to the service primary key, and comparing all attributes of the source metadata and all attributes of the target metadata to obtain a record to be updated includes the steps of:

(41) connecting intersection data, source metadata and target metadata, comparing all attributes of the source metadata with all attributes of the target metadata, splicing all attributes ATT _ ATTRS of the metadata, then performing MD5 operation to obtain a unique hash value, then comparing the hash value MD5_ T _ FEATURES of the source metadata attribute with the hash value MD5_ T' FEATURES of the target metadata attribute, if the hash values are consistent, considering that the attributes of the source metadata attribute and the target metadata attribute have no change, marking the attribute state as no change, otherwise considering that the attributes have change, marking the attribute state as changed, and storing the data in a newly created temporary table TMP _ UPDATE;

(42) adding an index to a service main key of a record table to be updated;

(43) adding an index for the attribute state of the record table to be updated;

(44) carrying out validity check on the module, if the check is passed, entering the next module, otherwise, entering (45);

(45) deleting the temporary table TMP _ UPDATE;

(45) and continuing to execute the step (33).

6. The method for implementing full data synchronization in big data environment according to claim 1, wherein said update-based record update data module comprises the following steps:

(51) the record table TMP _ UPDATE to be updated is connected to the source metadata table and the UPDATE is performed in the target metadata table.

7. The method for implementing full data synchronization in big data environment according to claim 1, wherein said deleting the record module to be deleted based on the intersection data comprises the following steps:

(61) connecting the intersection data and the target metadata according to the main service key, and obtaining all data to be deleted if the entry is 1;

(62) performing deletion in the target metadata table;

(63) carrying out validity check on the module, and if the module passes the validity check, deleting the temporary table TMP _ UPDATE;

(64) deleting the temporary table TMP _ UNION;

(65) delete table T'.

Images