CN111782619A - A method, synchronization device and storage medium for incremental synchronization of documents between servers - Google Patents

Abstract

The invention relates to a method, a synchronization device and a storage medium for incremental synchronization of documents between servers, which are applied to the incremental synchronization of a source server to a target server, wherein the source server maintains a record of each synchronized object. Synchronizing the record table, the synchronization method is specifically: taking a certain directory on the source server as the root node, recursively sub-nodes, comparing the information in each node and the synchronization record table, obtaining difference information, generating a synchronization task table, The synchronization task table is disassembled into multiple subtasks that are performed concurrently, and are concurrently transmitted to the target server. After each subtask is completed, the execution result of each subtask is updated in the synchronization record table. Compared with the prior art, the present invention can calculate the difference locally without requesting the document structure from the peer server, greatly improving the calculation efficiency and greatly reducing the dependence on the network environment.

Description

A method, synchronization device and storage medium for incremental synchronization of documents between servers

technical field

The invention relates to the technical field of databases, and in particular, to a method, a synchronization device and a storage medium for incremental synchronization of documents between servers.

Background technique

As a basic function of the network disk, document synchronization has a relatively mature algorithm in each network disk product to ensure that the file is consistent with the server locally on the client side. Generally, file synchronization is a C/S architecture: the server saves the file list of each terminal, and the client obtains the file list information through the interface provided by the server and caches it locally. When the client needs to synchronize, it needs to obtain the directory structure and document object attributes from the server through a network request, and combine with the client's local file cache information to perform incremental calculations and make synchronization decisions.

However, due to compliance requirements, the network between servers is generally physically isolated, and files can only be transferred between servers through switching devices. In this environment, the client-server synchronization mode is not suitable for server-server document synchronization. Therefore, there is a need to develop a new method to be applied to server-to-server document synchronization.

SUMMARY OF THE INVENTION

The purpose of the present invention is to provide a method, a synchronization device and a storage medium for incremental synchronization of documents between servers in order to overcome the above-mentioned defects in the prior art, without requesting the document structure from the opposite server, easy to use, and short synchronization time.

The object of the present invention can be realized through the following technical solutions:

A method for incremental synchronization of documents between servers, which is applied to incremental synchronization from a source server to a target server. The source server maintains a synchronization record table that records each synchronized object. The synchronization method is specifically for:

Taking a directory on the source server as the root node, recursively sub-nodes, compare each node with the information in the synchronization record table, obtain the difference information, generate a synchronization task table, and disassemble the synchronization task table into concurrent Multiple subtasks are concurrently transmitted to the target server, and after each subtask is completed, the execution result of each subtask is updated in the synchronization record table.

Further, the difference information is obtained by incremental calculation, and the document object attributes compared during the incremental calculation include document id and version identifier.

Further, the specific process of the incremental calculation includes:

1) Read the version identification of each document object in the source server, and judge whether the version identification is in the synchronization record table, if so, execute step 4), if not, execute step 2);

2) Determine whether the document id associated with the version identifier is in the synchronization record table, if so, execute step 3), if not, write the corresponding document object into the synchronization task table, and mark it as uploading;

3) If the document object is a directory, recurse the sub-objects of the directory. When all objects at the same level as the document object are traversed, all records are taken from the synchronization record table, and the source server is searched according to the file name. For objects that do not exist, write these objects to the synchronization task table and mark them for deletion;

If the document object is a file, directly write the document object into the synchronization task table and mark it as upload;

4) No synchronization;

5) Recurse downwards from the root node until all files and directories are complete.

Further, disassembling the synchronization task table into a plurality of subtasks is specifically: disassembling each record in the synchronization task table in a small granularity.

Further, the small-granularity disassembly is specifically: according to the tags of each synchronization task, calling a corresponding processing interface that communicates with the target server to implement corresponding processing of the file.

The synchronization task marked as "upload" will call the interface for uploading files to the target server, and pass the file information for uploading;

Call the synchronization task marked "delete" to delete the file to the target server, and pass the file information to delete.

Further, the execution result of updating each subtask in the synchronization record table is specifically:

If the object to be operated is a file, when the object is marked for upload, the document id and version identifier are recorded in the synchronization record table at the same time, and when the object is marked for deletion, the record corresponding to the document id is deleted from the synchronization record table;

If the object to be operated is a directory, when the object is marked for upload, record the document id and version ID of each file in the upload directory in the synchronization record table, and delete the corresponding document id from the synchronization record table when the object is marked for deletion record, and delete the record of the document id of the child object in the synchronization record table.

Further, when the document id and version identifier are recorded in the synchronization record table, if the document id exists in the synchronization record table, the corresponding version identifier is overwritten; otherwise, a synchronization record containing the document id and version identifier is added.

Further, the source server transmits the information to the source server through the switching device.

The present invention also provides a document incremental synchronization device between servers, which is set on the source server, including:

The incremental calculation module is used to maintain a synchronization record table that records each synchronized object. When a synchronization task is performed, a certain directory on the source server is used as the root node, recursive child nodes, and each node is compared and synchronized. Record the information in the table, obtain the difference information, and generate the synchronization task table;

The synchronization module is used to disassemble the synchronization task table into multiple subtasks that are performed concurrently, and transmit them to the target server concurrently. After each subtask is completed, update the execution result of each subtask in the synchronization record table. .

The present invention also provides a computer-readable storage medium on which a computer program is stored, the computer program being executed by a processor to implement the method.

Compared with the prior art, the present invention has the following beneficial effects:

1. The present invention records the document synchronization result on the source server, and can realize the incremental synchronization of the document without the need to communicate with the target end. The difference is calculated, the calculation efficiency is greatly improved, and the dependence on the network environment is greatly reduced.

2. In the environment of network physical isolation, the present invention facilitates file transmission through switching devices.

3. By calculating the file difference between the source server and the target server, the present invention can realize only the synchronization of the changed part, reduce the synchronization time, and save the synchronization bandwidth usage.

4. By disassembling the synchronization task into subtasks, a synchronization task can be decomposed and performed concurrently. When an exception occurs and a single subtask fails, it can be retried for a single subtask, or it can be included in the next incremental synchronization. Failed subtasks.

5. The present invention classifies files and directories during incremental calculation and synchronization processing, and has high synchronization precision and reliability.

Description of drawings

1 is a schematic diagram of a synchronization process of the present invention;

Fig. 2 is the schematic diagram of incremental calculation process of the present invention;

Fig. 3 is the update schematic diagram of the synchronization record table of the present invention;

FIG. 4 is a schematic diagram of transmission through a switching device in an embodiment.

Detailed ways

The present invention will be described in detail below with reference to the accompanying drawings and specific embodiments. This embodiment is implemented on the premise of the technical solution of the present invention, and provides a detailed implementation manner and a specific operation process, but the protection scope of the present invention is not limited to the following embodiments.

Terms and Definitions

Incremental calculation: refers to the calculation process of calculating the synchronization objects to be included in a synchronization task by comparing the differences between the synchronization objects at the source end and the destination end.

Synchronization: Refers to the process of transferring documents from a source system to a target system over a network.

Synchronization record table: refers to the table in which the source system records the synchronization result of each document. After the document synchronization is completed, the document information will be updated or added to the table for incremental calculation in the next synchronization.

Example 1

This embodiment provides a method for incremental synchronization of documents between servers, which is applied to incremental synchronization from a source server (source system) to a target server (peer server, target system). The source server maintains A synchronization record table for recording each synchronized object, the synchronization method is specifically: taking a certain directory on the source server as the root node, recursively sub-nodes, comparing each node with the information in the synchronization record table, and obtaining Difference information, generate a synchronization task table, disassemble the synchronization task table into multiple subtasks that are performed concurrently, call the peer file system API, and transmit to the target server concurrently, after each subtask is completed, in the synchronization Update the execution result of each subtask in the record table. Through the cycle of the above process, the incremental synchronization of documents between the two systems is realized. This synchronization method is suitable for document synchronization between two systems. Without the need to send a network request to the target to obtain the document structure and document attributes, the source system calculates and updates the synchronized content to realize the increase of the document. volume synchronization.

1. Incremental calculation

The difference information is obtained through incremental calculation, and the document object attributes compared during the incremental calculation include document id and version identifier.

A synchronization task is the process of completely mapping a source directory to the target. After the synchronization task is completed, the directory structure and file content of the source and target should be exactly the same. In order to reduce network requests and improve synchronization efficiency, the incremental calculation of synchronization must be performed before synchronization to obtain the content that needs to be synchronized and the corresponding execution operations.

Incremental calculation is mainly obtained by comparing the source document object and the information in the synchronization record of the object. Incremental calculation compares the following three basic changes:

·Added at the source

·Source editing

·Delete at source

Other file changes can be represented by the equivalence or combination of the above three basic changes:

Rename the source: equal to delete the source + add the source.

·Move (move out of the source directory): equal to the source delete.

·Move (move into the source directory): equal to adding at the source.

·Copy: equal to adding at the source.

When performing incremental calculations, you need to compare document object properties such as document id and version ID.

The document id is the unique identification of the document object. The document id is generated when the document is created. If the document is a file, editing, renaming, and overwriting the file after it is created will not modify its document id; if the document is a directory, renaming the directory and modifying the contents of the directory will not modify its document. id. The document id needs to reflect its inclusion relationship. For example, if the document id of a directory is "id1", the document id of the sub-object period under the directory is "id1/id2". All child objects and deeper objects can be found according to the document id, or its parent or higher-level objects can be found according to the document id.

The version identifier is a unique identifier for the content of the document. If the document is a file, editing and overwriting the file will not modify its document id, but each operation will create a new version ID. If the document is a directory, modifying subdocuments in the directory creates a new version ID on the directory object.

The synchronization record table records all the files that have been synchronized. Similarly, each record needs to contain the following information:

(1) Document id. After the document synchronization service transmits the document, it records the source document id in the synchronization record.

(2) Version identification. After the document synchronization service transmits the document, it records the source version identifier in the synchronization record.

As shown in Figure 2, the specific process of incremental calculation includes:

1) Read the version ID of each document object in the source server, and judge whether the version ID is in the synchronization record table. If so, go to step 4), if not, consider that the document object has not been synchronized, go to step 2 );

2) Determine whether the document id associated with the version identifier is in the synchronization record table, if so, execute step 3), if not, write the corresponding document object into the synchronization task table, and mark it as uploading;

3) If the document object is a directory, it is considered that the content in this directory has changed, and the sub-objects of the source directory are recursively. When all objects at the same level as the document object are traversed, the synchronization record table Take out all records, find objects that do not exist in the source server according to the file name, write these objects into the synchronization task table, and mark them for deletion;

If the document object is a file, it is considered that the content of the file has changed, and the document object is directly written into the synchronization task table and marked as upload;

4) If the version identifier is found in the synchronization record, it is considered that this object has not been modified at the source end and does not need to be synchronized;

5) The update calculation starts from the root directory and recurses downwards until all files and directories are completed.

2. Synchronization

After calculating the changes to be synchronized, each record is disassembled into subtasks with smaller granularity according to the synchronization task table.

For subtasks, the following two basic operations suffice in all cases:

1. Upload

2. Delete

After the task is dismantled, synchronize to the target according to the operation type of each subtask. Whenever a subtask is completed synchronously, the synchronization module needs to write the execution result of the subtask into the synchronization record table. specifically:

For the synchronization task marked "Upload", call the interface for uploading files to the peer server, and pass the file information for uploading.

Call the synchronization task marked "delete" to delete the file to the peer server, and pass the file information to delete.

The specific interface design of "upload" and "delete" is not described in this document.

As shown in Figure 3, the execution result of updating each subtask in the synchronization record table is divided into two situations: file and directory:

If the object to be operated is a file, when the object is marked for upload, the document id and version identifier are recorded in the synchronization record table at the same time, and when the object is marked for deletion, the record corresponding to the document id is deleted from the synchronization record table;

If the object to be operated is a directory, when the object is marked for upload, record the document id and version ID of each file in the upload directory in the synchronization record table, and delete the corresponding document id from the synchronization record table when the object is marked for deletion record, and delete the record of the document id of the child object in the synchronization record table.

When recording the document id and version identifier in the synchronization record table, if the document id exists in the synchronization record table, the corresponding version identifier is overwritten; otherwise, a synchronization record containing the document id and version identifier is added.

As shown in FIG. 4 , in an embodiment, the source server transmits information to the source server through the switching device, and the file transmission is performed through the switching device.

Example 2

This embodiment provides a device for synchronizing document increments between servers, which is set on the source server and includes:

The incremental calculation module is used to maintain a synchronization record table that records each synchronized object. When a synchronization task is performed, a certain directory on the source server is used as the root node, recursive child nodes, and each node is compared and synchronized. Record the information in the table, obtain the difference information, and generate the synchronization task table;

The synchronization module is used to disassemble the synchronization task table into multiple subtasks that are performed concurrently, and transmit them to the target server concurrently. After each subtask is completed, update the execution result of each subtask in the synchronization record table. .

The incremental calculation module is used to calculate which objects need to be uploaded and deleted for a synchronization task, and the corresponding upload or delete operations are marked on the objects; the synchronization module is used to upload or delete the synchronization objects on the target server, and then Update the sync record after the operation is complete. The interaction between the incremental computing module and the synchronization module is shown in Figure 1.

Example 3

This embodiment provides a computer-readable storage medium on which a computer program is stored, and the computer program is executed by a processor to implement the method described in Embodiment 1.

The preferred embodiments of the present invention have been described above in detail. It should be understood that those skilled in the art can make many modifications and changes according to the concept of the present invention without creative efforts. Therefore, any technical solutions that can be obtained by those skilled in the art through logical analysis, reasoning or limited experiments on the basis of the prior art according to the concept of the present invention shall fall within the protection scope determined by the claims.

Claims (10)

1. A method for incremental synchronization of documents between service terminals is applied to incremental synchronization from a source terminal server to a target terminal server, and is characterized in that the source terminal server maintains a synchronous record table for recording each synchronized object, and the synchronization method specifically comprises the following steps:

and taking a certain directory on the source server as a root node, recursion sub-nodes, comparing information in each node and a synchronous record table to obtain difference information, generating a synchronous task table, disassembling the synchronous task table into a plurality of sub-tasks which are performed concurrently, transmitting the plurality of sub-tasks to the target server concurrently, and updating the execution result of each sub-task in the synchronous record table after each sub-task is completed.

2. The method for document incremental synchronization between servers according to claim 1, wherein the difference information is obtained by incremental computation, and the document object attributes compared in the incremental computation include a document id and a version id.

3. The method for document incremental synchronization between servers according to claim 2, wherein the specific process of incremental computation includes:

1) reading the version identification of each document object in the source server, judging whether the version identification is in the synchronous record table, if so, executing the step 4), and if not, executing the step 2);

2) judging whether the document id associated with the version identification is in a synchronous record table, if so, executing the step 3), otherwise, writing the corresponding document object into a synchronous task table and marking the document object as uploading;

3) if the document object is a directory, recursion is carried out on the sub-objects of the directory, when all the objects at the same level as the document object are traversed, all the records are taken out from the synchronous record table, the objects which do not exist in the source server are searched according to the file name, the objects are written into the synchronous task table and marked as deleted;

if the document object is a file, directly writing the document object into a synchronous task table, and marking the document object as uploading;

4) synchronization is not performed;

5) recursion down from the root node until all files and directories are complete.

4. The method for incrementally synchronizing documents between servers according to claim 1, wherein the splitting of the synchronization task table into a plurality of subtasks specifically comprises: and performing small-granularity disassembly on each record in the synchronous task table.

5. The method for incremental synchronization of documents between servers according to claim 4, wherein the small-granularity parsing specifically comprises:

and calling a corresponding processing interface communicated with the target end server according to the mark of each synchronous task to realize corresponding processing of the file.

6. The method for incremental synchronization of documents between servers according to claim 2, wherein updating the execution result of each subtask in the synchronization record table specifically comprises:

if the object to be operated is a file, simultaneously recording a document id and a version identifier in the synchronous record table when the object is marked to be uploaded, and deleting a record corresponding to the document id from the synchronous record table when the object is marked to be deleted;

if the object to be operated is a directory, when the object is marked as uploading, the document id and the version identification of each file in the uploading directory are recorded in the synchronous record table, when the object is marked as deleting, the record corresponding to the document id is deleted from the synchronous record table, and the record corresponding to the document id of the sub-object in the synchronous record table is deleted.

7. The method for incremental synchronization of documents between servers according to claim 6, wherein when a document id and a version identifier are recorded in the synchronization record table, if the document id exists in the synchronization record table, the corresponding version identifier is overwritten; otherwise, adding a new synchronous record containing the document id and the version identification.

8. The method of claim 1, wherein the source server transmits information to the source server through a switching device.

9. An inter-server document increment synchronization device, which is arranged in a source server, includes:

the incremental computation module is used for maintaining a synchronous record table for recording each synchronized object, and when a synchronous task is executed, a certain directory on the source server is taken as a root node, a recursion child node is used for comparing information in each node and the synchronous record table to obtain difference information and generate a synchronous task table;

and the synchronization module is used for disassembling the synchronization task table into a plurality of subtasks which are performed concurrently, transmitting the subtasks to the target end server concurrently, and updating the execution result of each subtask in the synchronization record table after each subtask is completed.

10. A computer-readable storage medium, having stored thereon a computer program for execution by a processor to perform the method of any one of claims 1 to 8.

Images