TWI442223B - The data recovery method of the data de-duplication - Google Patents

Description

Data recovery method for deduplication

A data maintenance method for data deduplication, in particular, a data recovery method for deduplication.

Data de-duplication is a data reduction technique commonly used in disk-based backup systems with the primary goal of reducing the storage capacity used in storage systems. It works by looking for duplicate variable-sized blocks of data in different locations in different files over a period of time. Duplicate data blocks are replaced with indicators. The use of "deduplication" technology can make more backup space, not only can save backup data on the storage system for a longer period of time, but also save a lot of bandwidth required for offline storage.

In the process of performing deduplication, the client 111 performs a process of dividing the input file 112. The input file 112 will generate a plurality of data blocks (defined herein as the split data block 113) after being subjected to the segmentation process. Please refer to "Figure 1", which is a schematic diagram of the segmentation data block after deduplication of the prior art. Subsequently, the client 111 performs hash processing on the segmentation data block 113, and generates fingerprint feature values of the respective slice data blocks 113 (that is, the fingerprint feature values of the segmentation data block 113). The client 111 compares the obtained fingerprint feature value with the fingerprint feature value stored in the storage server, and determines whether there is the same fingerprint feature value. If the same fingerprint feature value exists, it means that the data block has been stored in the storage server.

When the client 111 wants to perform data reply processing, the client 111 requests a file request from the storage server. The storage server will transmit all the segmentation data blocks 113 (that is, the entire input file 112) directly to the client 111 according to the file request. The client 111 overwrites the input file 112 with the received segmentation data block 113 to restore the input file 112. Although this method is fast, it has problems such as high load on the client 111 (and the storage server) and bandwidth occupation during transmission.

In view of the above problems, the present invention provides a data recovery method for data deduplication for restoring part of data of a client object file.

The data recovery method for data deduplication disclosed in the present invention includes the following steps: the client obtains the file attribute of the target file; the client queries the storage server for the file attribute of the original file corresponding to the target file; and the client compares the target file. Whether the file attribute is consistent with the file attribute of the original file; if the target file is inconsistent with the file attribute of the original file, the target file is segmented, and at least all the data blocks and corresponding fingerprint feature values are generated; The terminal obtains all the fingerprint feature values of the original file, and the client compares the fingerprint feature values of the original file with the target file; the client obtains the corresponding segmentation data block from the storage server according to the different fingerprint feature values. Overwrite the obtained segmentation data block to the corresponding location in the target file.

The invention provides a data recovery method for data deduplication, which is used for restoring part of data of a target file of a client. The client performs local data restoration on the target file through the fingerprint feature value stored in the storage server and the corresponding segmentation data block.

The features and implementations of the present invention are described in detail below with reference to the drawings.

Please refer to "Figure 2" for a schematic diagram of the architecture of the present invention. Please refer to "Figure 2" for a schematic diagram of the architecture of the present invention. The present invention includes a client 210 and a storage server 220. The client 210 can be connected to the storage server 220 through an Internet or an intranet, or the client 210 and the storage server 220 can be simultaneously run on the same computer device.

The storage server 220 further includes a fingerprint feature value index list 221, and the fingerprint feature value index table records a plurality of sets of fingerprint feature values 222. When the client 210 issues a query request for an input file to the storage server 220, the storage server 220 indexes the content of the list 221 based on the fingerprint feature value and performs an inquiry operation in the following manner. Please refer to "Figure 3", which is a schematic diagram of the process of deduplication of the present invention.

Step S310: The client loads the input file, and generates a data block of the corresponding input file and a fingerprint feature value of each corresponding data block;

Step S320: The client sends a query request to the storage server, and records the fingerprint feature value of the corresponding data block in the query request, so as to query the storage server for whether the same fingerprint feature value exists;

Step S330: When the fingerprint feature value is not stored in the fingerprint feature value index list of the storage server, the storage server sends a storage request to the client to transmit the data block corresponding to the fingerprint feature value to the storage server for storage. And the storage server adds the received fingerprint feature values to the fingerprint feature value index list in sequence;

Step S340: When the fingerprint feature value already exists in the fingerprint feature value index table of the storage server, the storage server returns to the client that the pen segmentation data block already exists.

The input file is loaded by the client 210, and the client 210 performs a segmentation process on the input file, and generates a data block of the corresponding input file and a fingerprint feature value 222 of each corresponding data block. The algorithm for calculating the fingerprint feature value 222 may be, but is not limited to, SHA-1 or MD5. The data block is based on a fixed-size partition or a content-defined chunking (CDC). The fixed length segmentation algorithm uses the predefined segmentation data block size to segment the input file. The advantage of the fixed length block algorithm is simplicity and high performance. The content definition segmentation algorithm is a variable length block algorithm that uses fingerprint data (such as Rabin fingerprint) to segment the file into chunking strategies of varying lengths. Unlike the fixed-length segmentation algorithm, the content definition segmentation algorithm is based on the input file content, so the segmentation data block size can be changed.

Then, the client 210 sends a query request to the storage server 220, and records the fingerprint feature value 222 of the corresponding data block in the query request to query the storage server 220 whether the same fingerprint feature value 222 exists. When the fingerprint feature value 222 is stored in the fingerprint feature value index list 221 of the storage server 220, the storage server 220 sends a storage request to the client 210 to transfer the data block corresponding to the fingerprint feature value 222 to the storage service. The terminal 220 stores the information, and the storage server 220 adds the received fingerprint feature values 222 to the fingerprint feature value index list 221 in sequence.

When the client 210 wants to perform file restoration processing, the client 210 issues a file restore request to the storage server 220. In order to clearly explain the file stored by the client 210 and the server, the file that the client 210 intends to perform file restoration processing defines it as an object file. The data files stored by the storage server 220 (that is, the sliced data blocks of each file) are defined as original files, so the number of original files is not only one. The storage server 220 performs corresponding file restoration processing according to the following steps. Please refer to "Fig. 4" and "Fig. 5", which are respectively schematic diagrams showing the difference between the operational flow diagram and the segmentation data block of the present invention. The system includes the following steps:

Step S410: The client obtains the file attribute of the target file.

Step S420: The client queries the storage server for the file attribute of the original file corresponding to the target file.

Step S430: Whether the file attribute of the target file is compared with the file attribute of the original file by the client;

Step S440: If the target file is consistent with the file attribute of the original file, the client does not perform file restoration processing;

Step S450: If the target file is inconsistent with the file attribute of the original file, the target file is segmented, and at least all the data blocks and the corresponding fingerprint feature values are generated;

Step S460: Obtain all fingerprint feature values of the original file from the storage server, and the client compares the difference between the original document and the fingerprint feature value of the target file;

Step S470: The client obtains the corresponding segmentation data block from the storage server according to the different fingerprint feature values, and overwrites the obtained segmentation data block to the corresponding location in the target file.

First, the client 210 obtains the file attribute of the target file, and the file attribute is a time stamp or an index number. In other words, the client 210 records the file attributes of the target file 520 before the client 210 performs the segmentation process on the target file. Next, the client 210 queries the storage server 220 for the file attributes of the original file 510 corresponding to the target file 520. The storage server 220 looks up whether the file attributes of the original file 510 corresponding to the target file 520 have been stored. If the client 210 has previously performed data backup on the target file 520, the storage server 220 stores the original file 510 of the corresponding target file 520 and related file attributes.

The client 210 compares the file attributes of the original file 510 transmitted from the storage server 220 with the file attributes of the target file 520. If the file attribute takes a timestamp as an example, the data files created at different times will be given different timestamps. Therefore, when the target file 520 does not match the file attribute of the original file 510, it means that the target file 520 has been modified.

If the target file 520 does not match the file attribute of the original file 510, the client 210 performs a segmentation process on the target file 520, and generates at least all the partial data blocks and the corresponding fingerprint feature values 222. The client 210 obtains all the fingerprint feature values 222 of the original file 510 from the storage server 220, and the client 210 compares the difference between the original file 510 and the fingerprint feature value 222 of the target file 520 (ie, in FIG. 5). Split the black block of the data block). The storage server 220 receives the fingerprint feature value 222 from the client 210, which may be a batch transmission, or may be transmitted to the client 210 in batches. Since the data amount of the fingerprint feature value 222 is much smaller than the segmentation data block, the use of the bandwidth is not seriously affected during the transmission. Finally, the client 210 obtains the corresponding segmentation data block from the storage server 220 according to the different fingerprint feature value 222, and overwrites the obtained segmentation data block to the corresponding location in the target file 520.

The present invention provides a data recovery method for data deduplication for restoring part of the data of the target file 520 of the client 210. The client 210 performs local data restoration on the target file 520 through the fingerprint feature value 222 stored in the storage server 220 and the corresponding segmentation data block. Moreover, the present invention does not require reading and writing of the object files 520 one by one as compared with the prior art, and only the processing of reading and calculation is required. Therefore, the writing time of the present invention can be effectively shortened compared to the prior art.

While the present invention has been described above in terms of the preferred embodiments thereof, it is not intended to limit the invention, and the invention may be modified and modified without departing from the spirit and scope of the invention. The patent protection scope of the invention is subject to the definition of the scope of the patent application attached to the specification.

111. . . Client

112. . . Input file

113. . . Segmentation data block

210. . . Client

220. . . Storage server

221. . . Fingerprint feature value index list

222. . . Fingerprint feature value

510. . . Original file

520. . . Target file

Figure 1 is a schematic diagram of a segmentation block after deduplication of the prior art.

Figure 2 is a schematic diagram of the architecture of the present invention.

Figure 3 is a schematic flow chart of the deduplication of the present invention.

Figure 4 is a schematic diagram of the operational flow of the present invention.

Figure 5 is a schematic diagram showing the difference of the segmentation data block of the present invention.

Claims (4)

A data recovery method for data deduplication, which is to provide a client to perform partial data recovery on an object file according to an original file stored in a storage server and subjected to deduplication processing, the data recovery method comprising: the client Obtaining a file attribute of the target file; the client queries the storage server for the file attribute of an original file corresponding to the target file; whether the file attribute of the target file is compared with the original The file attribute of the file is consistent; if the target file is inconsistent with the file attribute of the original file, then the target file is subjected to all sub-processing, and at least all the sub-blocks and corresponding fingerprint feature values are generated; to the storage The server obtains all the fingerprint feature values of the original file, the client is different from the fingerprint feature values of the original file and the target file; and the client according to the different fingerprint feature values Obtaining the corresponding segmentation data blocks from the storage server, and overwriting the obtained segmentation data blocks The target file corresponding position. The data restoration method of deduplication as described in claim 1, wherein the file attribute is a time stamp or an index number. The data restoration method of deduplication as described in claim 1, wherein the fingerprint feature value is transmitted through a hash algorithm or a one way algorithm. The data recovery method of the data deduplication method of claim 1, wherein the step of overwriting the obtained segment data block to the corresponding location in the object file further comprises: the client repeating the fingerprints different in comparison The feature value is obtained from the storage server to obtain the corresponding segmentation data block, and the target file is overwritten until all the target files are completed.