CN107391761B - A data management method and device based on deduplication technology - Google Patents

Abstract

The invention discloses a data management method and a device based on a repeated data deleting technology, wherein the method calculates a fingerprint value of target data through a HASH algorithm; determining a storage position corresponding to the fingerprint value of the target data through CRUSH mapping; then, the target data is used as data to be stored, and whether the storage position of the data to be stored is stored with data is judged; if yes, adding one to the reference count of the data to be stored; if the first metadata information of the data to be stored is not stored, the data to be stored is stored, the reference count of the data to be stored is set to be one, and finally the first metadata information of the data to be stored is stored. Therefore, in the process of data storage, the repeated storage of data is avoided, and the working efficiency is improved; meanwhile, based on the repeated data technology, the management of data is realized, the cost is saved, and the service life of the storage system is prolonged. The data management device based on the data de-duplication technology provided by the embodiment of the invention also has the technical effects.

Description

A data management method and device based on deduplication technology

technical field

The present invention relates to the technical field of cloud computing data centers, and more particularly, to a data management method and device based on duplicate data deletion technology.

Background technique

With the rapid development of computer technology and the Internet industry, data information is increasing day by day. In order to save storage space and realize resource sharing, distributed storage systems emerge as the times require. The distributed storage system stores data in multiple independent devices, adopts a scalable system structure, uses multiple storage servers to share the storage load, and uses the location server to locate and store information, which can improve the reliability, availability and management efficiency of the system , but also easy to extend.

However, since many terminals can access the storage server, there must be a large amount of duplicate data, which occupies storage space. At this time, the deduplication technology that optimizes storage capacity solves this problem. Data deduplication technology has been widely used in backup, long-term archiving and data disaster recovery by eliminating duplicate data in the storage system and reducing the data actually stored in the system or the data transmitted through the network. In the field of distributed storage, in order to reduce the cost of storage unit capacity, the processing of online duplicate data is imminent.

Therefore, how to implement the repeated data technology in the field of distributed storage, that is, to realize the storage, reading and deletion of data in the field of distributed storage by using the repeated data technology, is a problem to be solved by those skilled in the art.

SUMMARY OF THE INVENTION

The purpose of the present invention is to provide a data management method and device based on the repeated data deletion technology, so as to realize the storage, reading and deletion of data based on the repeated data technology in the field of distributed storage.

To achieve the above purpose, the embodiments of the present invention provide the following technical solutions:

A data management method based on data deduplication technology, comprising:

S11. Calculate the fingerprint value of the target data through the HASH algorithm;

S12, determine the storage location corresponding to the fingerprint value of the target data through CRUSH mapping; take the target data as the data to be stored, and execute S13;

S13, determine whether there is data in the storage location corresponding to the data to be stored; if so, execute S14; if not, execute S15;

S14, add one to the reference count corresponding to the data to be stored, and execute S16;

S15, store the data to be stored in the storage location corresponding to the data to be stored, and set the reference count corresponding to the data to be stored to one, and execute S16;

S16. Store first metadata information of the data to be stored, where the first metadata information includes: a fingerprint value of the data to be stored.

Wherein, before executing the S11, it also includes:

S21, determine whether there is second metadata information corresponding to the target data; if so, execute S22; if not, execute S11;

S22, acquiring the second metadata information;

S23, determine whether there is a fingerprint value in the second metadata information; if so, execute S24; if not, execute S11;

S24. Compare the length of the target data with the preset data length; if the length of the target data is equal to the preset data length, execute S11; if the length of the target data is less than the preset data length, execute S25;

S25, splicing the target data and the data corresponding to the second metadata information to obtain spliced data, and calculate the fingerprint value of the spliced data, and execute S26;

S26. Determine the storage location corresponding to the fingerprint value of the spliced data through CRUSH mapping; take the spliced data as the data to be stored, and execute S13.

Wherein, the if the length of the target data is equal to the preset data length, including:

If the length of the target data is equal to the preset data length, decrement the reference count of the data corresponding to the second metadata information by one;

judging whether the reference count of the data corresponding to the second metadata information is zero;

If so, delete the data corresponding to the second metadata information.

Wherein, splicing the target data and the data corresponding to the second metadata information to obtain spliced data, and calculating the fingerprint value of the spliced data, including:

acquiring data content corresponding to the second metadata information;

According to the preset data length and data offset, the target data and the data corresponding to the second metadata information are spliced to obtain spliced data;

Calculate the fingerprint value of the spliced data;

Decrement the reference count of the data corresponding to the second metadata information by one;

judging whether the reference count of the data corresponding to the second metadata information is zero;

If so, delete the data corresponding to the second metadata information.

Among them, it also includes:

Receive the deletion request sent by the client;

Determine the data to be deleted according to the deletion request, and obtain the third metadata information of the data to be deleted and the fingerprint value of the data to be deleted in the third metadata information;

Determine the storage location corresponding to the fingerprint value of the data to be deleted through CRUSH mapping, and decrement the reference count corresponding to the data to be deleted by one;

Determine whether the reference count corresponding to the data to be deleted is zero;

If so, delete the data to be deleted and the third metadata information.

A data management device based on deduplication technology, comprising:

The first calculation module is used to calculate the fingerprint value of the target data through the HASH algorithm;

The first determination module is used to determine the storage location corresponding to the fingerprint value of the target data through CRUSH mapping, and use the target data as the data to be stored;

a first judging module for judging whether there is data in the storage location corresponding to the data to be stored;

a first execution module, configured to add one to the reference count corresponding to the data to be stored when data is stored in the storage location corresponding to the data to be stored;

The first storage module is used to store the to-be-stored data in the storage location corresponding to the to-be-stored data when there is no data in the storage location corresponding to the to-be-stored data, and set the reference count corresponding to the to-be-stored data to one ;

The second storage module is configured to store first metadata information of the data to be stored, where the first metadata information includes: a fingerprint value of the data to be stored.

Among them, it also includes:

a second judgment module, configured to judge whether there is second metadata information corresponding to the target data; if not, trigger the first calculation module;

a first obtaining module, configured to obtain the second metadata information when there is second metadata information corresponding to the target data;

a third judging module for judging whether there is a fingerprint value in the second metadata information; if not, triggering the first computing module;

a comparison module, configured to compare the length of the target data with a preset data length when there is a fingerprint value in the second metadata information; if the length of the target data is equal to the preset data length, then triggering the first calculation module;

A splicing module, configured to splicing the target data and the data corresponding to the second metadata information when the length of the target data is less than the preset data length to obtain splicing data, and calculate the The fingerprint value of the concatenated data;

The second determining module is configured to determine the storage location corresponding to the fingerprint value of the spliced data through CRUSH mapping.

Wherein, the comparison module includes:

a first execution unit, configured to decrement the reference count of the data corresponding to the second metadata information by one when the length of the target data is equal to the preset data length;

a first judging unit, configured to judge whether the reference count of the data corresponding to the second metadata information is zero;

A first deletion unit, configured to delete the data corresponding to the second metadata information when the reference count of the data corresponding to the second metadata information is zero.

Wherein, the splicing module includes:

an acquiring unit, configured to acquire data content corresponding to the second metadata information;

a splicing unit, configured to splicing the target data and the data corresponding to the second metadata information according to the preset data length and data offset to obtain splicing data;

a computing unit for computing the fingerprint value of the spliced data;

a second execution unit, configured to decrement the reference count of the data corresponding to the second metadata information by one;

a second judging unit, configured to judge whether the reference count of the data corresponding to the second metadata information is zero;

A second deletion unit, configured to delete the data corresponding to the second metadata information when the reference count of the data corresponding to the second metadata information is zero.

Among them, it also includes:

The receiving module is used to receive the deletion request sent by the client;

a second acquiring module, configured to determine the data to be deleted according to the deletion request, and acquire third metadata information of the data to be deleted and the fingerprint value of the data to be deleted in the third metadata information;

A third determining module, configured to determine the storage location corresponding to the fingerprint value of the data to be deleted through CRUSH mapping, and decrement the reference count corresponding to the data to be deleted by one;

The fourth judgment module is used for judging whether the reference count corresponding to the data to be deleted is zero;

A deletion module, configured to delete the to-be-deleted data and the third metadata information when the reference count corresponding to the to-be-deleted data is zero.

As can be seen from the above solutions, a data management method based on the deduplication technology provided by the embodiment of the present invention includes:

S11. Calculate the fingerprint value of the target data through the HASH algorithm;

S12, determine the storage location corresponding to the fingerprint value of the target data through CRUSH mapping; take the target data as the data to be stored, and execute S13;

S13, determine whether there is data in the storage location corresponding to the data to be stored; if so, execute S14; if not, execute S15;

S14, add one to the reference count corresponding to the data to be stored, and execute S16;

S15, store the data to be stored in the storage location corresponding to the data to be stored, and set the reference count corresponding to the data to be stored to one, and execute S16;

S16. Store first metadata information of the data to be stored, where the first metadata information includes: a fingerprint value of the data to be stored.

It can be seen that the above method calculates the fingerprint value of the target data through the HASH algorithm; and determines the storage location corresponding to the fingerprint value of the target data through CRUSH mapping; wherein, the fingerprint value determines the uniqueness of the target data, and then determines the target data storage location The uniqueness of the location; then the target data is used as the data to be stored, and it is judged whether there is data in the storage location corresponding to the data to be stored; since the data to be stored has a unique storage location, if there is already data in the storage location , it indicates that the data to be stored has been stored, so the data to be stored is no longer stored, but the reference count corresponding to the data to be stored is incremented by one; if there is no data in the storage location, it indicates that the data to be stored is not stored, and the The data to be stored is stored in the storage location corresponding to the data to be stored, the reference count corresponding to the data to be stored is set to one, and finally the first metadata information of the data to be stored is stored, and the first metadata information includes: The fingerprint value of the stored data. In this way, through the above method, in the process of data storage, the multiple storage of data is avoided, the work efficiency is improved, and the storage space of the system is saved; at the same time, based on the repeated data technology, it is realized in the field of distributed storage. Data management saves costs and extends the life of storage systems.

Correspondingly, the data management apparatus based on the deduplication technology provided by the embodiment of the present invention also has the above technical effects.

Description of drawings

In order to explain the embodiments of the present invention or the technical solutions in the prior art more clearly, the following briefly introduces the accompanying drawings that need to be used in the description of the embodiments or the prior art. Obviously, the accompanying drawings in the following description are only These are some embodiments of the present invention. For those of ordinary skill in the art, other drawings can also be obtained according to these drawings without creative efforts.

1 is a flowchart of a data management method based on a deduplication technology disclosed in an embodiment of the present invention;

2 is a flowchart of another data management method based on a deduplication technology disclosed in an embodiment of the present invention;

3 is a flowchart of a data deletion method in a data management method based on a deduplication technology according to an embodiment of the present invention;

FIG. 4 is a schematic diagram of a data management apparatus based on a deduplication technology disclosed in an embodiment of the present invention.

Detailed ways

The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, but not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

The embodiment of the present invention discloses a data management method and device based on the duplicate data deletion technology, so as to realize the storage, reading and deletion of data based on the duplicate data technology in the field of distributed storage.

Referring to FIG. 1 , a data management method based on a deduplication technology provided by an embodiment of the present invention includes:

S11. Calculate the fingerprint value of the target data through the HASH algorithm;

Specifically, in this embodiment, the target data is the data to be stored in the current operation, and before calculating the fingerprint value of the target data, the target data needs to be divided into blocks first.

At present, in the field of distributed storage, in order to regularize the data in the underlying storage, the data to be stored is generally divided according to the size of the underlying storage object. For example: if the underlying storage object is divided according to the size of 4M, and the size of the target data is 10M, then the target data is divided into three blocks of 4M, 4M and 2M based on 4M. The data to be stored is cut into blocks less than or equal to 4M.

Specifically, when calculating the fingerprint value of the target data, the fingerprint value of the target data is calculated by the HASH algorithm according to the data content of the above-mentioned blocks. One correspondence, that is, the data content and the fingerprint value are matched in pairs, and the key-value pair matching information is formed. If the target data is divided into multiple blocks, each block has a fingerprint value, and subsequent operations are performed on the data corresponding to each fingerprint value; if the target data is divided into one block, the target data has a fingerprint value. , and perform subsequent operations on this fingerprint value. In this embodiment, the target data is divided into a block with a unique fingerprint value.

S12, determine the storage location corresponding to the fingerprint value of the target data through CRUSH mapping; take the target data as the data to be stored, and execute S13;

Specifically, according to the fingerprint value calculated by the above HASH algorithm, the target data is replaced by the fingerprint value of the target data through the CRUSH mapping process at the Rados layer, and is transmitted to the object storage device, and then the object storage device stores it in its own storage system. Find the storage location corresponding to the target data, and then determine the storage location.

S13, determine whether there is data in the storage location corresponding to the data to be stored; if so, execute S14; if not, execute S15;

Specifically, taking the target data as the data to be stored, after the object storage device determines the storage location of the data to be stored, it first determines whether there is data in the storage location. If there is data, it means that the data to be stored has been stored; If there is no data, it means that the data to be stored has not been stored.

S14, add one to the reference count corresponding to the data to be stored, and execute S16;

Specifically, if it is determined in step S13 that the data to be stored has been stored, the data to be stored is no longer stored, but the reference count corresponding to the data to be stored is incremented by one.

S15, store the data to be stored in the storage location corresponding to the data to be stored, and set the reference count corresponding to the data to be stored to one, and execute S16;

Specifically, it is determined in step S13 that the data to be stored has not been stored, the data to be stored is stored in a storage location corresponding to the data to be stored, and the reference count corresponding to the data to be stored is set to one.

S16. Store first metadata information of the data to be stored, where the first metadata information includes: a fingerprint value of the data to be stored.

Specifically, after the storage of the data to be stored is completed, the reference count corresponding to the data to be stored will also be stored in the dedicated storage location prepared by the object storage device; at the same time, the metadata information of the data to be stored is also stored, and the metadata information Including various attributes such as the fingerprint value of the data to be stored.

Specifically, when the metadata is stored in the fingerprint value, 8K metadata will be stored first, and then the fingerprint value corresponding to the file will be stored after the metadata. The metadata information stored in the metadata is based on files, and is stored in a cluster environment with 8K as an object. For a 4MB data block, there will be 4088KB of space to store fingerprint data, and SHA-1 is used as the fingerprint. HASH algorithm, the size of a fingerprint is 20 bytes, and 209305 fingerprint values will be stored at this time, corresponding to 817GB of data.

It can be seen that the present embodiment provides a data management method based on de-duplication technology, wherein the method calculates the fingerprint value of the target data through the HASH algorithm; and determines the storage location corresponding to the fingerprint value of the target data through CRUSH mapping; Among them, the fingerprint value determines the uniqueness of the target data, and then determines the uniqueness of the storage location of the target data; then the target data is used as the data to be stored, and it is judged whether there is data in the storage location corresponding to the data to be stored; The data to be stored has a unique storage location, so if there is already data in the storage location, it means that the data to be stored has been stored, and the data to be stored is no longer stored, but the reference count corresponding to the data to be stored is incremented by one; If there is no data in the storage location, it means that the data to be stored is not stored, then the data to be stored is stored in the storage location corresponding to the data to be stored, and the reference count corresponding to the data to be stored is set to one, and finally the data to be stored is stored. The first metadata information is stored, the first metadata information includes: the fingerprint value of the data to be stored. In this way, through the above method, in the process of data storage, the multiple storage of data is avoided, the work efficiency is improved, and the storage space of the system is saved; at the same time, based on the repeated data technology, it is realized in the field of distributed storage. Data management saves costs and extends the life of storage systems.

Referring to FIG. 2, another data management method based on the deduplication technology provided by an embodiment of the present invention includes:

S21, determine whether there is second metadata information corresponding to the target data; if so, execute S22; if not, execute S11;

Specifically, in this embodiment, before storing the target data, it will first determine whether the second metadata information of the target data exists, that is, whether the target data is stored for the first time or stored again, so as to determine whether the current operation is to create a write or to store it again. Modify write. If the second metadata information of the target data exists, it indicates that the target data is not stored for the first time, then it is determined that the current operation is a modification write, and step S22 is continued; if the second metadata information of the target data does not exist, it indicates that the target data is the first time If stored, step S11 is executed.

S22, acquiring the second metadata information;

Specifically, the specific process of obtaining the second metadata information is as follows: the file system client obtains the index information of the target number, and requests the metadata storage to obtain the second metadata information; the metadata storage obtains the index information according to the target data The second metadata information includes the fingerprint value of the target data, and the fingerprint value is stored in the form of a key-value pair.

It should be noted that only the second metadata information is obtained here. To obtain the data content corresponding to a certain metadata, that is, to read the data, the client needs to obtain the index information of the data to be read, and send the metadata to the metadata. Storage request metadata information; metadata storage obtains metadata information based on index information, including fingerprint values stored in the form of key-value pairs for each object that composes the file; Rados, according to data offset, data length and fingerprint value, to Read data directly from the object storage device. This completes the data reading process.

S23, determine whether there is a fingerprint value in the second metadata information; if so, execute S24; if not, execute S11;

Specifically, after obtaining the second metadata information of the target data through step S22, it is necessary to determine whether the second metadata information is complete, that is, to determine whether there is a fingerprint value in the second metadata information, and if there is a fingerprint value, continue to execute the step S24; if there is no fingerprint value, execute step S11.

S24. Compare the length of the target data with the preset data length; if the length of the target data is equal to the preset data length, execute S11; if the length of the target data is less than the preset data length, execute S25;

Specifically, after determining that the fingerprint value exists in the second metadata information through the above step S23, it is necessary to compare the length of the target data with the size of the preset data length. Before comparing the size of the data length, the size of the target data is generally divided, and the specific process of dividing the size is similar to the above-mentioned embodiment, so it is not repeated here.

Specifically, after the target data is processed into blocks, the length of the block is compared with the size of the preset data length. In this embodiment, assuming that the target data is divided into one data block, the length of this data block is equal to the target data length. The length of the data, then compare the length of the target data with the preset data length. The preset data length here is the default length in the system. Generally, the default length of the system is 4M. If the length of the target data is equal to the preset data length, execute S11; if the length of the target data is less than the preset data length, continue to execute S25;

S25, splicing the target data and the data corresponding to the second metadata information to obtain spliced data, and calculate the fingerprint value of the spliced data, and execute S26;

Specifically, if the length of the target data is smaller than the preset data length, the target data and the data corresponding to the second metadata information need to be spliced according to the data offset and the data length. In this embodiment, the preset data length is set as 4M. For example: the length of the data corresponding to the second metadata information is a data object of 0 to 4M, and the length of the target data is 1M. At this time, the position of 2 to 3M in 0 to 4M needs to be modified; All data 0-4M corresponding to the information are read and spliced with the target data 1M, that is, 0-4M is divided into 0-2M, 2-3M, 3-4M three segments, and the original 2-3M total of 1M content is replaced with 1M content of the target data, and then splicing the three segments of 0~2M, new 2~3M, 3~4M together to form a new 4M data, that is, the splicing data is obtained.

S26. Determine the storage location corresponding to the fingerprint value of the spliced data through CRUSH mapping; take the spliced data as the data to be stored, and execute S13.

Specifically, in this embodiment, the specific process of determining the storage location corresponding to the fingerprint value of the spliced data is similar to the above-mentioned embodiment, so it is not repeated here. After the storage location corresponding to the fingerprint value of the spliced data is determined, the spliced data needs to be used as the data to be stored, and step S13 is continued.

It can be seen that, in a data management method based on a deduplication technology provided by this embodiment, the method first determines whether there is second metadata information corresponding to the target data; when the target data has second metadata information, then acquire the second metadata information; when the target data does not have the second metadata information, execute S11; after acquiring the second metadata information, determine whether there is a fingerprint value in the second metadata information; If there is, compare the length of the target data with the preset data length; if not, execute S11; after comparing the length of the target data and the preset data length, if the length of the target data is equal to the The preset data length, then execute S11; if the length of the target data is less than the preset data length, then splicing the target data and the data corresponding to the second metadata information to obtain spliced data , and calculate the fingerprint value of the spliced data, and determine the storage location corresponding to the fingerprint value of the spliced data through CRUSH mapping; take the spliced data as the data to be stored, and then execute S13. Through the above method, in the process of data storage, the multiple storage of data is avoided, the work efficiency is improved, and the storage space of the system is saved; at the same time, based on the repeated data technology, the data is realized in the field of distributed storage. management, saving costs and extending the life of the storage system.

Based on any of the foregoing embodiments, it should be noted that the if the length of the target data is equal to the preset data length, including:

If the length of the target data is equal to the preset data length, decrement the reference count of the data corresponding to the second metadata information by one;

judging whether the reference count of the data corresponding to the second metadata information is zero;

If so, delete the data corresponding to the second metadata information.

Specifically, in the data modification and writing process, when the length of the target data is equal to the preset data length, the reference count of the data corresponding to the second metadata information is decremented by one. If there is no other reference to the data corresponding to the information, the reference count after decrement by one is zero, and at this time, the data corresponding to the second metadata information is deleted.

Based on any of the above embodiments, it should be noted that the splicing of the target data and the data corresponding to the second metadata information to obtain spliced data, and calculating the fingerprint value of the spliced data, includes:

acquiring data content corresponding to the second metadata information;

According to the preset data length and data offset, the target data and the data corresponding to the second metadata information are spliced to obtain spliced data;

Calculate the fingerprint value of the spliced data;

Decrement the reference count of the data corresponding to the second metadata information by one;

judging whether the reference count of the data corresponding to the second metadata information is zero;

If so, delete the data corresponding to the second metadata information.

Specifically, in the process of data modification and writing, the length of the target data is less than the preset data length, and the target data and the The data corresponding to the second metadata information is spliced to obtain spliced data, and the fingerprint value of the spliced data is calculated; and then it is judged whether there are other references to the data corresponding to the second metadata information. The specific process is as follows: The reference count of the data corresponding to the metadata information is decremented by one, and if the reference count is decremented by one and then zero, it indicates that there is no other reference to the data corresponding to the second metadata information, and the data corresponding to the second metadata information is deleted; If the reference count is not zero after being decremented by one, it indicates that there are other references to the data corresponding to the second metadata information, and the data corresponding to the second metadata information is retained.

Based on any of the above-mentioned embodiments, it should be noted that a data management method based on deduplication provided by the embodiment of the present invention also includes a data deletion method. Referring to FIG. 3 , the specific process is as follows:

S31. Receive the deletion request sent by the client;

S32, determining the data to be deleted according to the deletion request, and acquiring the third metadata information of the data to be deleted and the fingerprint value of the data to be deleted in the third metadata information;

S33, determine the storage location corresponding to the fingerprint value of the data to be deleted by CRUSH mapping, and decrement the reference count corresponding to the data to be deleted by one;

S34, determine whether the reference count corresponding to the data to be deleted is zero;

S35, if yes, then delete the data to be deleted and the third metadata information;

S36. If not, do not perform the deletion operation for the time being.

Specifically, when the data deletion method is executed, the data reading process is included, that is, the data to be deleted is determined according to the deletion request, and the third metadata information and the third metadata of the to-be-deleted data are obtained. The fingerprint value of the data to be deleted in the data information, here only the third metadata information of the data to be deleted and the fingerprint value therein are read, and the content of the data to be deleted is not read. Then, determine the storage location corresponding to the fingerprint value of the data to be deleted through CRUSH mapping, decrement the reference count corresponding to the data to be deleted and delete its metadata information, and inform the client that the data to be deleted is successfully deleted; If the reference count is decremented by one and then zero, it indicates that there is no other reference to the data corresponding to the second metadata information, and the to-be-deleted data is deleted.

A data management device based on the deduplication technology provided by the embodiments of the present invention will be introduced below. The data management device based on the deduplication technology described below and the data management device based on the deduplication technology described above Methods can refer to each other.

Referring to FIG. 4 , a data management device based on a deduplication technology provided by an embodiment of the present invention includes:

The first calculation module 401 is used to calculate the fingerprint value of the target data through the HASH algorithm;

The first determination module 402 is used to determine the storage location corresponding to the fingerprint value of the target data through CRUSH mapping, and use the target data as the data to be stored;

The first judgment module 403 is used for judging whether there is data in the storage location corresponding to the data to be stored;

The first execution module 404 is configured to add one to the reference count corresponding to the data to be stored when there is data in the storage location corresponding to the data to be stored;

The first storage module 405 is configured to store the data to be stored in the storage location corresponding to the data to be stored when there is no data in the storage location corresponding to the data to be stored, and set the reference count corresponding to the data to be stored. one;

The second storage module 406 is configured to store first metadata information of the data to be stored, where the first metadata information includes: a fingerprint value of the data to be stored.

Among them, it also includes:

a second judgment module, configured to judge whether there is second metadata information corresponding to the target data; if not, trigger the first calculation module;

Specifically, when the second judgment module judges that there is no second metadata information corresponding to the target data, the first calculation module is triggered, and the first calculation module calculates the fingerprint value of the target data through the HASH algorithm; The first determination module determines the storage location corresponding to the fingerprint value of the target data through CRUSH mapping, and uses the target data as the data to be stored; and then the first determination module determines whether there is data in the storage location corresponding to the data to be stored. When there is data in the storage location corresponding to the data to be stored, the first execution module adds one to the reference count corresponding to the data to be stored; when there is no data in the storage location corresponding to the data to be stored, the first execution module The storage module stores the to-be-stored data in a storage location corresponding to the to-be-stored data, and sets the reference count corresponding to the to-be-stored data to one; finally, the second storage module stores the first metadata information of the to-be-stored data, the first The metadata information includes: the fingerprint value of the data to be stored.

a first obtaining module, configured to obtain the second metadata information when there is second metadata information corresponding to the target data;

a third judging module for judging whether there is a fingerprint value in the second metadata information; if not, triggering the first computing module;

Specifically, when the third interpretation module judges that there is no fingerprint value in the second metadata information, the first calculation module is triggered; the first calculation module calculates the fingerprint value of the target data through the HASH algorithm; and the first determination module Determine the storage location corresponding to the fingerprint value of the target data through CRUSH mapping, and use the target data as the data to be stored; then the first judgment module determines whether there is data in the storage location corresponding to the data to be stored; When data is stored in the storage location corresponding to the stored data, the first execution module adds one to the reference count corresponding to the data to be stored; when there is no data stored in the storage location corresponding to the data to be stored, the first storage module will The stored data is stored in the storage location corresponding to the to-be-stored data, and the reference count corresponding to the to-be-stored data is set to one; finally, the second storage module stores the first metadata information of the to-be-stored data, and the first metadata information includes : The fingerprint value of the data to be stored.

a comparison module, configured to compare the length of the target data with a preset data length when there is a fingerprint value in the second metadata information; if the length of the target data is equal to the preset data length, then triggering the first calculation module;

Specifically, when there is a fingerprint value in the second metadata information, the length of the target data is compared with a preset data length; if the length of the target data is equal to the preset data length, trigger the first A calculation module; the fingerprint value of the target data is calculated by the first calculation module through the HASH algorithm; and the storage position corresponding to the fingerprint value of the target data is determined by the first determination module through CRUSH mapping, and the target data is used as the pending storage. Then the first judgment module judges whether there is data in the storage location corresponding to the data to be stored; when there is data in the storage location corresponding to the data to be stored, the first execution module counts the references corresponding to the data to be stored Add one; when there is no data in the storage location corresponding to the data to be stored, the first storage module stores the data to be stored in the storage location corresponding to the data to be stored, and sets the reference count corresponding to the data to be stored by one ; Finally, the second storage module stores the first metadata information of the data to be stored, and the first metadata information includes: the fingerprint value of the data to be stored.

A splicing module, configured to splicing the target data and the data corresponding to the second metadata information when the length of the target data is less than the preset data length to obtain splicing data, and calculate the The fingerprint value of the concatenated data;

The second determining module is configured to determine the storage location corresponding to the fingerprint value of the spliced data through CRUSH mapping.

Wherein, the comparison module includes:

a first execution unit, configured to decrement the reference count of the data corresponding to the second metadata information by one when the length of the target data is equal to the preset data length;

a first judging unit, configured to judge whether the reference count of the data corresponding to the second metadata information is zero;

A first deletion unit, configured to delete the data corresponding to the second metadata information when the reference count of the data corresponding to the second metadata information is zero.

Wherein, the splicing module includes:

an acquiring unit, configured to acquire data content corresponding to the second metadata information;

a splicing unit, configured to splicing the target data and the data corresponding to the second metadata information according to the preset data length and data offset to obtain splicing data;

a computing unit for computing the fingerprint value of the spliced data;

a second execution unit, configured to decrement the reference count of the data corresponding to the second metadata information by one;

a second judging unit, configured to judge whether the reference count of the data corresponding to the second metadata information is zero;

A second deletion unit, configured to delete the data corresponding to the second metadata information when the reference count of the data corresponding to the second metadata information is zero.

Among them, it also includes:

The receiving module is used to receive the deletion request sent by the client;

a second acquiring module, configured to determine the data to be deleted according to the deletion request, and acquire third metadata information of the data to be deleted and the fingerprint value of the data to be deleted in the third metadata information;

A third determining module, configured to determine the storage location corresponding to the fingerprint value of the data to be deleted through CRUSH mapping, and decrement the reference count corresponding to the data to be deleted by one;

The fourth judgment module is used for judging whether the reference count corresponding to the data to be deleted is zero;

A deletion module, configured to delete the to-be-deleted data and the third metadata information when the reference count corresponding to the to-be-deleted data is zero.

It can be seen that, in the data management device based on the deduplication technology provided by this embodiment, first the first calculation module calculates the fingerprint value of the target data through the HASH algorithm; the first determination module determines the fingerprint value of the target data through CRUSH mapping The corresponding storage location, the target data is taken as the data to be stored; the first judgment module judges whether there is data in the storage location corresponding to the data to be stored; when there is data in the storage location corresponding to the data to be stored, The first execution module adds one to the reference count corresponding to the data to be stored; when there is no data in the storage location corresponding to the data to be stored, the first storage module stores the data to be stored in the storage location corresponding to the data to be stored , and set the reference count corresponding to the data to be stored to one; finally, the second storage module stores the first metadata information of the data to be stored, and the first metadata information includes: the fingerprint value of the data to be stored. Thus, the storage of data and the storage of its metadata information are completed.

The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments, and the same and similar parts between the various embodiments can be referred to each other.

The above description of the disclosed embodiments enables any person skilled in the art to make or use the present invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the generic principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Thus, the present invention is not intended to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims (8)

1. a data management method based on deduplication technology, is characterized in that, comprises: S11. Calculate the fingerprint value of the target data through the HASH algorithm; S12, determine the storage location corresponding to the fingerprint value of the target data through CRUSH mapping; take the target data as the data to be stored, and execute S13; S13, determine whether there is data in the storage location corresponding to the data to be stored; if so, execute S14; if not, execute S15; S14, add one to the reference count corresponding to the data to be stored, and execute S16; S15, store the data to be stored in the storage location corresponding to the data to be stored, and set the reference count corresponding to the data to be stored to one, and execute S16; S16. Store first metadata information of the data to be stored, where the first metadata information includes: a fingerprint value of the data to be stored; Wherein, before executing the S11, it also includes: S21, determine whether there is second metadata information corresponding to the target data; if so, execute S22; if not, execute S11; S22, acquiring the second metadata information; S23, determine whether there is a fingerprint value in the second metadata information; if so, execute S24; if not, execute S11; S24. Compare the length of the target data with the preset data length; if the length of the target data is equal to the preset data length, execute S11; if the length of the target data is less than the preset data length, execute S25; S25, splicing the target data and the data corresponding to the second metadata information to obtain spliced data, and calculate the fingerprint value of the spliced data, and execute S26; S26. Determine the storage location corresponding to the fingerprint value of the spliced data through CRUSH mapping; take the spliced data as the data to be stored, and execute S13. 2. The data management method based on data deduplication technology according to claim 1, wherein, if the length of the target data is equal to the preset data length, comprising: If the length of the target data is equal to the preset data length, decrement the reference count of the data corresponding to the second metadata information by one; judging whether the reference count of the data corresponding to the second metadata information is zero; If so, delete the data corresponding to the second metadata information. 3. the data management method based on deduplication technology according to claim 1, is characterized in that, described target data and the data corresponding with described second metadata information are spliced, obtain splicing data, and Calculate the fingerprint value of the spliced data, including: acquiring data content corresponding to the second metadata information; According to the preset data length and data offset, the target data and the data corresponding to the second metadata information are spliced to obtain spliced data; Calculate the fingerprint value of the spliced data; Decrement the reference count of the data corresponding to the second metadata information by one; judging whether the reference count of the data corresponding to the second metadata information is zero; If so, delete the data corresponding to the second metadata information. 4. The data management method based on any one of claims 1-3, characterized in that, further comprising: Receive the deletion request sent by the client; Determine the data to be deleted according to the deletion request, and obtain the third metadata information of the data to be deleted and the fingerprint value of the data to be deleted in the third metadata information; Determine the storage location corresponding to the fingerprint value of the data to be deleted through CRUSH mapping, and decrement the reference count corresponding to the data to be deleted by one; Determine whether the reference count corresponding to the data to be deleted is zero; If so, delete the data to be deleted and the third metadata information. 5. A data management device based on deduplication technology, characterized in that, comprising: The first calculation module is used to calculate the fingerprint value of the target data through the HASH algorithm; The first determination module is used to determine the storage location corresponding to the fingerprint value of the target data through CRUSH mapping, and use the target data as the data to be stored; a first judging module for judging whether there is data in the storage location corresponding to the data to be stored; a first execution module, configured to add one to the reference count corresponding to the data to be stored when data is stored in the storage location corresponding to the data to be stored; The first storage module is used to store the to-be-stored data in the storage location corresponding to the to-be-stored data when there is no data in the storage location corresponding to the to-be-stored data, and set the reference count corresponding to the to-be-stored data to one ; a second storage module, configured to store first metadata information of the data to be stored, where the first metadata information includes: a fingerprint value of the data to be stored; Wherein, the data management device further includes: a second judgment module, configured to judge whether there is second metadata information corresponding to the target data; if not, trigger the first calculation module; a first obtaining module, configured to obtain the second metadata information when there is second metadata information corresponding to the target data; a third judging module for judging whether there is a fingerprint value in the second metadata information; if not, triggering the first computing module; a comparison module, configured to compare the length of the target data with a preset data length when there is a fingerprint value in the second metadata information; if the length of the target data is equal to the preset data length, then triggering the first calculation module; A splicing module, configured to splicing the target data and the data corresponding to the second metadata information when the length of the target data is less than the preset data length to obtain splicing data, and calculate the The fingerprint value of the concatenated data; The second determining module is configured to determine the storage location corresponding to the fingerprint value of the spliced data through CRUSH mapping. 6. The data management device based on deduplication technology according to claim 5, wherein the comparison module comprises: a first execution unit, configured to decrement the reference count of the data corresponding to the second metadata information by one when the length of the target data is equal to the preset data length; a first judging unit, configured to judge whether the reference count of the data corresponding to the second metadata information is zero; A first deletion unit, configured to delete the data corresponding to the second metadata information when the reference count of the data corresponding to the second metadata information is zero. 7. The data management device based on deduplication technology according to claim 5, wherein the splicing module comprises: an acquiring unit, configured to acquire data content corresponding to the second metadata information; a splicing unit, configured to splicing the target data and the data corresponding to the second metadata information according to the preset data length and data offset to obtain splicing data; a computing unit for computing the fingerprint value of the spliced data; a second execution unit, configured to decrement the reference count of the data corresponding to the second metadata information by one; a second judging unit, configured to judge whether the reference count of the data corresponding to the second metadata information is zero; A second deletion unit, configured to delete the data corresponding to the second metadata information when the reference count of the data corresponding to the second metadata information is zero. 8. The data management device based on any one of claims 5-7, characterized in that, further comprising: The receiving module is used to receive the deletion request sent by the client; a second acquiring module, configured to determine the data to be deleted according to the deletion request, and acquire third metadata information of the data to be deleted and the fingerprint value of the data to be deleted in the third metadata information; A third determining module, configured to determine the storage location corresponding to the fingerprint value of the data to be deleted through CRUSH mapping, and decrement the reference count corresponding to the data to be deleted by one; The fourth judgment module is used for judging whether the reference count corresponding to the data to be deleted is zero; A deletion module, configured to delete the to-be-deleted data and the third metadata information when the reference count corresponding to the to-be-deleted data is zero.

Images