CN115098481A - A metadata management method for deduplication - Google Patents

Abstract

The application relates to a metadata management method for deleting repeated data, which comprises the steps of receiving an I/O (input/output) request, cutting data into data blocks, and establishing a unique fingerprint by utilizing a Hash algorithm; in the fingerprint index structure, inquiring the fingerprint of the data block; creating a transaction entry in a bitmap form, scanning a bitmap by a metadata manager to obtain a transaction entry in a free state, and distributing a globally unique Trans-i d; if a request for hitting the fingerprint index is received, mapping the logic block number accessed by the upper layer into a physical block number in the fingerprint mapping, and adding 1 to the reference count; if the request of the fingerprint index is not hit, a new free physical block number is allocated, a logical block number mapping is established, a mapping table and a reference count are written into a metadata area of the PM, and the written data block is stored. The transaction entry combined with the log mechanism is introduced, atomicity in the transaction entry is guaranteed, and consistent metadata of repeated data deletion during crash is guaranteed.

Description

A metadata management method for deduplication

technical field

The present application relates to the field of computer storage, and in particular, to a metadata management method for deduplication of data.

Background technique

With the development of modern information technology and the Internet, a large amount of data is generated every day. The society is entering an era of big data, and the data that needs to be managed is becoming more and more complex. Therefore, the development of data deduplication technology solves some of the problems caused by the explosion of data.

Data deduplication is a popular data reduction technology that can effectively reduce the capacity of storage systems. It detects the same data object in the data stream based on the redundancy of the data itself, only transmits and stores a unique data object, and replaces other duplicate data copies with a pointer to the unique data object to reduce network transmission and storage. process of capacity requirements.

According to the granularity of data deduplication, it can be divided into file-level and block-level deduplication. File-level deduplication ensures that files are not duplicated, while block-level deduplication involves dividing files into data blocks for comparison. Since block-level deduplication technology is simple and efficient, it can achieve higher compression rate, so it has become the mainstream research object. As the amount of stored data increases, so does the difficulty of maintaining metadata to manage the data.

Additionally, appends, updates, and deletes to metadata need to be crash-consistent, otherwise unacceptable errors may occur in the event of a system crash or power outage. As shown in Figure 1, when processing the duplicate write request of file A data block (LBNx) and file B data block (LBNy), due to system crash or power failure, the reference count of its physical block (PBNa) may fail to increase, There are two duplicate data blocks in the system, and the reference count is 1. In subsequent operations, if file B is deleted and data is written to file C, the reference count of the data block PBNa will be set to zero, even if file A still needs to use this physical block. The system reclaims physical blocks with a reference count of zero. If this physical block is allocated to the LBNz of file C, this may cause the data block (LBNx) of the undeleted file A to be abnormally overwritten by the data block (LBNz) of file C. Finally, when the user accesses file A, the data of file C will be returned.

The above error is due to a partial update of the deduplication metadata without providing crash consistency guarantees in the event of an unexpected system power loss or system crash. Therefore, research on crash consistency of deduplication metadata will become very important.

It is worth noting that non-volatile memory has memory-level read and write latency and endurance, and this hardware feature is suitable for storing metadata information, which can effectively solve the conflict problem of deduplication and consistency.

SUMMARY OF THE INVENTION

The present application provides a metadata management method for deduplication to update metadata that needs to be deduplicated during an online process to ensure consistency in the event of a crash.

The technical solution of this application is as follows:

According to a first aspect of the embodiments of the present application, there is provided a metadata management method for deduplication, the method comprising:

Receive the I/O request to the data, cut the data into data blocks of a fixed size, and use a hash algorithm to perform fingerprint calculation to establish a unique fingerprint for the data block;

In the fingerprint index structure preloaded in the memory, query and compare the fingerprints of the data blocks to determine whether to hit the same data block;

Creating a transaction entry in the form of a bitmap, wherein one I/O request corresponds to one transaction entry, and the transaction entry has nothing to do with whether the fingerprint is hit;

Scan the bitmap through the metadata manager, obtain the transaction entry in the free state, and assign a globally unique Trans-id variable to the transaction entry;

If a request to hit the fingerprint index is received, the logical block number accessed by the upper layer is mapped to the physical block number in the fingerprint map, and the reference count of the physical block corresponding to the fingerprint is increased by 1;

If no request hits the fingerprint index, allocate a new free physical block number, build a logical block number map, forward the I/O request down, and write the map and reference count to the PM (persistent and storing the written data block, wherein the mapping also needs to be written into the corresponding transaction entry.

In some embodiments, the transaction entry includes a free state, a start state, and an end state.

In some embodiments, the transaction entry includes a Trans-id variable, a state variable, and at least one metadata update backup; wherein, the state variable is used to represent that the transaction entry is in the above-mentioned free state, starting One of state and end state; the Trans-id variable is used as the sequence number of transaction entries and the order in which they were checkpointed.

In some embodiments, the cutting the data into data blocks of a fixed size specifically includes:

Based on memory page and file system block size, set the deduplication data block size to 4KB.

In some embodiments, the memory layout related to deduplication metadata in the persistent memory is divided into a metadata area and a log area, wherein the metadata area stores the content of the data block mapping table and the physical block reference count, The log area is designed as a fixed-size area consisting of N transaction entries, and a bitmap is used to mark the usage of different entries.

According to a second aspect of the embodiments of the present application, there is provided another metadata management method for deduplication, the method comprising:

Receive the I/O request to the data, cut the data into data blocks of a fixed size, and use a hash algorithm to perform fingerprint calculation to establish a unique fingerprint for the data block;

In the fingerprint index structure preloaded in the memory, query and compare the fingerprints of the data blocks to determine whether to hit the same data block;

Creating a transaction entry in the form of a bitmap, wherein the one I/O request corresponds to a transaction entry, and the transaction entry has nothing to do with whether the fingerprint is hit;

Scan the bitmap through the metadata manager, obtain the transaction entry in the free state, and assign a globally unique Trans-id to the transaction entry;

If a request to hit the fingerprint index is received, the logical block number accessed by the upper layer is mapped to the target physical block number in the fingerprint map, the reference count of the corresponding physical block is increased by 1, and the reference count of the original mapped physical block number is decreased;

If the request for the fingerprint index is not hit, the I/O request is forwarded downward, and the deduplication metadata is written into the metadata area of the persistent memory, and the written data block is stored;

If the reference count of the originally mapped physical block number is decremented to 0, the deletion deletes the physical block number whose reference count is 0.

In some embodiments, a transaction entry includes a free state, a start state, and an end state.

In some embodiments, the transaction entry includes a Trans-id variable, a state variable, and at least one metadata update backup; wherein, the state variable is used to represent that the transaction entry is in the above-mentioned free state, starting One of state and end state; the Trans-id variable is used as the sequence number of transaction entries and the order in which they were checkpointed.

In some embodiments, the cutting the data into data blocks of a fixed size specifically includes:

Based on memory page and file system block size, set the deduplication data block size to 4KB.

According to a third aspect of the embodiments of the present application, a non-volatile storage device is provided, including: persistent memory storage; the persistent memory storage is used to implement the above-described metadata management method for deduplication, which is persistent High-performance memory storage typically employs memory with good low latency and byte addressability; capable of achieving near-memory-level speed and byte addressability. The metadata management method for deduplication involved in the present application can reduce a large amount of additional metadata I/O at the time of writing or at the time of updating, and introduce transaction entries combined with the log mechanism to ensure transaction entries. Atomicity within, guarantees crash-consistent deduplication metadata.

It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not limiting of the present application.

Description of drawings

The accompanying drawings are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present application, and together with the description, serve to explain the principles of the present application, and do not constitute an improper limitation of the present application.

1 is a schematic diagram illustrating a scenario of data corruption caused by a system crash or power failure when processing a duplicate write request of a file A data block (LBNx) and a file B data block (LBNy) according to an exemplary embodiment;

FIG. 2 is a schematic diagram of a system architecture based on a non-volatile memory file system according to an exemplary embodiment;

3 is a schematic flowchart of a metadata management method for deduplication according to an exemplary embodiment;

4 is a schematic diagram showing the composition of a transaction entry according to an exemplary embodiment;

5 is a schematic diagram of a transition flow of three states of a transaction entry according to an exemplary embodiment;

Fig. 6 is a schematic diagram of the execution flow of a metadata management method for deduplication according to an exemplary embodiment when writing new data;

FIG. 7 is a schematic flowchart of a method for managing metadata for deduplication according to an exemplary embodiment when new data is written.

Detailed ways

In order to make those skilled in the art better understand the technical solutions of the present application, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the accompanying drawings.

It should be noted that the terms "first", "second", etc. in the description and claims of the present application and the above drawings are used to distinguish similar objects, and are not necessarily used to describe a specific sequence or sequence. It is to be understood that data so used may be interchanged under appropriate circumstances so that the embodiments of the application described herein can be practiced in sequences other than those illustrated or described herein. The implementations described in the illustrative examples below are not intended to represent all implementations consistent with this application. Rather, they are merely examples of apparatus and methods consistent with some aspects of the present application as recited in the appended claims.

As shown in FIG. 2 , the present application exemplarily shows a schematic diagram of a system architecture based on a non-volatile memory file system.

The operating system generally divides the accessible memory space into two parts, one is the kernel space and the other is the user space. Kernel space is an area accessed by the operating system kernel, independent of ordinary applications, and is a protected memory space. User space is an area of memory accessible by normal applications.

In the kernel space, the deduplication system (hereinafter referred to as NC-DEDUP) consists of three parts: a metadata management module (marked as a metadata manager in the figure), a hash engine module and a block interface module. As shown in Figure 2, NC-DEDUP is located at the bottom layer of the file system and the upper layer of the block device driver. That is to say, the file system is the upper layer of NC-DEDUP, and the block device driver is the lower layer of NC-DEDUP. This structure ensures its good portability and provides good function expansibility. The specific functional modules are as follows:

Metadata management module: Metadata mainly includes the mapping table from fingerprint index to PBN (physical block number), the mapping table from LBN (logical block number) to PBN (physical block number), and the reference count of PBN. The metadata management module is responsible for the allocation and reclamation of metadata space, periodic persistence, fast query, and the implementation of the logging mechanism on PM (Persistent Memory) to ensure crash consistency.

Hash engine (Hash) module: read files sequentially for traversal search, which is very slow, so you need to build an index. The most commonly used index is a hash table. If we build an index on the data in the file, "Key" saves the value we want to query next time, and "Value" corresponds to which file location. A fingerprint is selected to identify each block based on its content, where the block fingerprint can uniquely identify the data block. Hash algorithm is used to identify whether data blocks are duplicated, and various hashing methods in data deduplication are adjusted according to different duplication rates to control computational overhead.

Block Interface Module: Based on the information in the block interface's deduplication metadata, the deduplicated data I/O (read/write) will be redirected or discarded.

In some embodiments, the metadata management of the present application is based on persistent memory (PM). The processing of any write request involves a series of modifications to the deduplication metadata. All possible metadata updates are divided into writing new blocks, repeating blocks, and updating blocks. Writing a new block means that the data block is not recorded, that is, the entry cannot be queried in the fingerprint index table, and the metadata information of this block needs to be recorded for subsequent repeated data operations, and a fingerprint index, an LBN map and a PBN reference count. Duplicate blocks mean that data blocks or logical block numbers are not recorded, but fingerprint information is recorded. For this type of data block, to query the PBN of the unique data block, it is necessary to update a LBN-to-unique PBN mapping and a PBN reference count. Update block means that the data block is recorded, and the mapping relationship between LBN and PBN has been saved. When writing data, an LBN mapping needs to be updated, and two PBN reference counts and one or zero indexes are updated.

In the metadata management method for deduplication provided by this application, the implementation of NC-DEDUP is based on the open source framework Dmdedup, which inherits the block I/O interface and the bio processing of the block device. Dmdedup is an open source block-level deduplication and deduplication module that uses a COW B-tree (copy on write) strategy to ensure consistency of deduplication and deduplication metadata.

As shown in Figure 3, the method performs the following steps:

S11: Receive an I/O request for data, cut the data into data blocks of a fixed size, and perform fingerprint calculation using a hash algorithm to establish a unique fingerprint for the data block.

In some embodiments, since the memory page and file system block sizes are usually 4KB, the deduplication data block size is set to 4KB, which can avoid block alignment problems and effectively reduce the overhead caused by block partitioning.

In some embodiments, the MD5 hash algorithm is used to calculate the data block fingerprint. The size of the fingerprint obtained by the traditional MD5 algorithm is 128 bits, which can effectively compress the fingerprint index space compared with other hash algorithms.

S12: In the fingerprint index structure preloaded in the memory, query and compare the fingerprints of the data blocks to determine whether to hit the same data block.

S13: Create a transaction entry in the form of a bitmap, wherein one I/O request corresponds to one transaction entry, and the transaction entry has nothing to do with whether the fingerprint is hit or not.

In some embodiments, in the PM-based metadata management, the memory layout related to deduplication metadata is mainly divided into a metadata area and a log area. The metadata area mainly stores the content of the data block mapping table and the reference count of the physical block. Accessing deduplication metadata will be significantly faster due to PM's low access latency and byte-addressable nature. The log area is designed as a fixed-size area consisting of N transaction entries, and a bitmap is used to mark the usage of different entries.

In some embodiments, the number of transaction entries in the log area on the PM is set to 512. More transaction entries can cache more metadata updates, but will affect the identification of subsequent duplicate write requests.

S14: Scan the bitmap through the metadata manager to obtain a transaction entry in a free state, and assign a globally unique Trans-id variable to the transaction entry.

In this embodiment of the present application, the transaction entry includes a free state, a start state, and an end state.

In some embodiments, as shown in FIG. 4, each transaction entry contains a Trans-id, a status variable (status), and at least one metadata update backup (ref-op). Wherein, the state variable is used to represent that the transaction entry is in one of the above three states; the Trans-id variable is used as the sequence number of the transaction entry and the order in which they are checked.

For each bio (block input output), blocking input and output) write operation issued by the upper layer, all operations on deduplication metadata will be packaged into a transaction entry, which will be modified and submitted uniformly. There is a checkpointed-id variable at the head of the PM space, which represents the current maximum Trans-id.

As shown in Figure 5, when the upper layer initiates an I/O write request, the metadata manager scans the bitmap, obtains the transaction entry in the free state (that is, the transaction entry marked as FREE), and stores the transaction entry The state transitions to BEGIN (that is, the start state). If no free entry is found in the bitmap, the operating system needs to reschedule the deduplication worker thread and block incoming read and write requests from that thread.

The transaction area is then populated with updated metadata operations and I/O requests are issued. When the lower block device completes the I/O request, the driver of the lower block device will issue an interrupt request, and the state of the transaction will be converted to END (that is, the end state) in the device's software interrupt routine.

After each I/O action on the data is completed, the metadata manager will start with the current checkpointed-id and check in id order to see if there are transaction entries that can be checkpointed into the metadata area on the PM. After writing all eligible transactions to the metadata area, set the transaction status to FREE and clear the flag in the bitmap.

The upper-level user program only needs to complete the writing of the log data area on the PM, and then it can return safely. Because of PM's low access latency, log update operations can complete quickly, so write completions can be quickly reported to users. When a system crash or power failure occurs, only entries in the log area that are in the end state need to be scanned and updated to the metadata area.

S15: If a request to hit the fingerprint index is received, map the logical block number accessed by the upper layer to the physical block number in the fingerprint map, and add 1 to the reference count of the physical block number corresponding to the fingerprint.

In some embodiments, when the metadata corresponding to the physical block number is updated, it is written into an allocated transaction.

In some embodiments, the hit fingerprint index is used to indicate that the data currently undergoing I/O and the existing data are determined to be the same data block after being queried by the fingerprint.

For write requests that hit the fingerprint index, NC-DEDUP does not forward the upper layer data to the I/O device, but records the updated deduplication metadata to the log area, and then informs the user that the I/O is completed.

S16: If the request for the fingerprint index is not hit, allocate a new free physical block number, establish a logical block number mapping, forward the I/O request downward, and write the mapping and reference count into the PM (persistent memory) metadata area, and store the written data block, wherein the mapping also needs to be written into the corresponding transaction entry.

As shown in Figure 6, when NC-DEDUP accepts a write request, it executes the above metadata management method for deduplication. For write requests, NC-DEDUP adds metadata updates to a transaction, ensuring atomicity within transaction entries. In addition, NC-DEDUP reduces a lot of additional metadata I/O on the basis of ensuring data consistency.

As shown in Figure 7, another metadata management method for deduplication provided by this application is mainly aimed at the situation when NC-DEDUP updates data, including:

S21: Receive an I/O request for data, cut the data into data blocks of a fixed size, and use a hash algorithm to perform fingerprint calculation to establish a unique fingerprint for the data block.

S22: In the fingerprint index structure preloaded in the memory, query and compare the fingerprints of the data blocks to determine whether to hit the same data block.

S23: Create a transaction entry in the form of a bitmap, wherein the one I/O request corresponds to one transaction entry, and the transaction entry has nothing to do with whether the fingerprint is hit or not.

In some embodiments, in the PM-based metadata management, the memory layout related to deduplication metadata is mainly divided into a metadata area and a log area. The metadata area mainly stores the content of the data block mapping table and the reference count of the physical block. Accessing deduplication metadata will be significantly faster due to PM's low access latency and byte-addressable nature. The log area is designed as a fixed-size area consisting of N transaction entries, and a bitmap is used to mark the usage of different entries.

S24: Scan the bitmap through the metadata manager to obtain a transaction entry in a free state, and assign a globally unique Trans-id variable to the transaction entry.

In fact, the above four steps are relatively the same as when NC-DEDUP writes new data.

S25: If a request to hit the fingerprint index is received, the logical block number accessed by the upper layer is mapped to the target physical block number in the fingerprint mapping, the reference count of the corresponding physical block is increased by 1, and the reference count of the original mapped physical block number is decreased. .

In some embodiments, when the metadata corresponding to the target physical block number is updated, it is written into an allocated transaction.

For write requests that hit the fingerprint index, NC-DEDUP does not forward the upper layer data to the I/O device, but records the updated deduplication metadata to the log area, and then informs the user that the I/O is completed.

S26: If the request for the fingerprint index is not hit, forward the I/O request downward, write the deduplication metadata into the metadata area of the PM, and store the written data block.

In some embodiments, deduplication metadata includes mapping tables and reference counts.

S27: If the reference count of the originally mapped physical block number is decremented to 0, delete the physical block number whose reference count is 0 to be deleted.

In some embodiments, if the original mapped PBN reference count is decremented to 0, it means that no other file shares its data block. The storage space for cleanup data blocks can be reclaimed through the garbage collection mechanism.

As shown in Figure 7, when NC-DEDUP updates data, NC-DEDUP adds metadata updates to a transaction to ensure atomicity within transaction entries. On the basis of ensuring data consistency, NC-DEDUP reduces a lot of additional metadata I/O. In addition, NC-DEDUP guarantees metadata checking and data I/O ordering, which ensures that deduplicated metadata points to the correct and safe data block.

Compared with the execution method of the existing deduplication technology when ensuring the consistency between data, the metadata management method for deduplication involved in the present application can reduce the amount of data in writing or updating. A large amount of additional metadata I/O, the introduction of transaction entries combined with the logging mechanism, to ensure atomicity within transaction entries, and to ensure consistent deduplication metadata in the event of a crash.

The application scenarios described in the embodiments of the present application are for the purpose of illustrating the technical solutions of the embodiments of the present application more clearly, and do not constitute a limitation on the technical solutions provided by the embodiments of the present application. It appears that the technical solutions provided by the embodiments of the present application are also applicable to similar technical problems.

As will be appreciated by one skilled in the art, various aspects of the present application may be implemented as a system, method or program product. Therefore, various aspects of the present application can be embodied in the following forms, namely: a complete hardware implementation, a complete software implementation (including firmware, microcode, etc.), or a combination of hardware and software aspects, which may be collectively referred to herein as implementations "circuit", "module" or "system".

In some possible implementations, an electronic device according to the present application may include at least one processor, and at least one memory. The memory stores program codes, which, when executed by the processor, cause the processor to execute the operation data management methods described above in this specification according to various exemplary embodiments of the present application. For example, the processor may perform steps as in an operational data management method.

In an exemplary embodiment, various aspects of a metadata management method for deduplication provided by the present application can also be implemented in the form of a program product, which includes program code, and when the program product runs on a computer device, The program code is used to cause the computer device to execute the steps in the metadata management method for deduplication according to various exemplary embodiments of the present application described above in this specification.

It should be noted that although several units or sub-units of the apparatus are mentioned in the above detailed description, this division is merely exemplary and not mandatory. Indeed, according to embodiments of the present application, the features and functions of two or more units described above may be embodied in one unit. Conversely, the features and functions of one unit described above may be further subdivided to be embodied by multiple units.

Furthermore, although the operations of the methods of the present application are depicted in the figures in a particular order, this does not require or imply that the operations must be performed in the particular order, or that all illustrated operations must be performed to achieve desirable results. Additionally or alternatively, certain steps may be omitted, multiple steps may be combined to be performed as one step, and/or one step may be decomposed into multiple steps to be performed.

As will be appreciated by those skilled in the art, the embodiments of the present application may be provided as a method, a system, or a computer program product. Accordingly, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present application may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) having computer-usable program code embodied therein.

The present application is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the present application. It will be understood that each flow and/or block in the flowchart illustrations and/or block diagrams, and combinations of flows and/or blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to the processor of a general purpose computer, special purpose computer, embedded processor or other programmable image scaling device to produce a machine such that the instructions executed by the processor of the computer or other programmable image scaling device produce Means for implementing the functions specified in a flow or flow of a flowchart and/or a block or blocks of a block diagram.

The computer program instructions may also be stored in a computer readable memory capable of directing a computer or other programmable image scaling device to function in a particular manner, such that the instructions stored in the computer readable memory result in an article of manufacture comprising instruction means, the instructions The apparatus implements the functions specified in the flow or flow of the flowcharts and/or the block or blocks of the block diagrams.

These computer program instructions can also be loaded on a computer or other programmable image scaling device to cause a series of operational steps to be performed on the computer or other programmable device to produce a computer-implemented process such that an image executed on the computer or other programmable device The instructions provide steps for implementing the functions specified in the flow or blocks of the flowcharts and/or the block or blocks of the block diagrams.

While the preferred embodiments of the present application have been described, additional changes and modifications to these embodiments may occur to those skilled in the art once the basic inventive concepts are known. Therefore, the appended claims are intended to be construed to include the preferred embodiment and all changes and modifications that fall within the scope of this application.

Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims (9)

1. A metadata management method for deduplication, wherein the method comprises: Receive the I/O request to the data, cut the data into data blocks of a fixed size, and use a hash algorithm to perform fingerprint calculation to establish a unique fingerprint for the data block; In the fingerprint index structure preloaded in the memory, query and compare the fingerprints of the data blocks to determine whether to hit the same data block; Creating a transaction entry in the form of a bitmap, wherein one I/O request corresponds to one transaction entry, and the transaction entry has nothing to do with whether the fingerprint is hit; Scan the bitmap through the metadata manager, obtain the transaction entry in the free state, and assign a globally unique Trans-id variable to the transaction entry; If a request to hit the fingerprint index is received, the logical block number accessed by the upper layer is mapped to the physical block number in the fingerprint map, and the reference count of the physical block number corresponding to the fingerprint is increased by 1; If no request hits the fingerprint index, allocate a new free physical block number, build a logical block number map, forward the I/O request down, and write the map and reference count to the PM (persistent and storing the written data block, wherein the mapping also needs to be written into the corresponding transaction entry. 2. The method of claim 1, wherein the transaction entry includes a free state, a start state, and an end state. 3. The method of claim 2, wherein the transaction entry comprises a Trans-id variable, a state variable, and at least one metadata update backup; wherein the state variable is used to characterize the transaction Entries are in one of the free, start, and end states described above; the Trans-id variable is used as the sequence number of transaction entries and the order in which they were checkpointed. 4. The method according to claim 3, wherein the cutting the data into data blocks of a fixed size specifically comprises: Based on memory page and file system block size, set the deduplication data block size to 4KB. 5. The method according to claim 4, wherein the memory layout related to deduplication metadata in the persistent memory is divided into a metadata area and a log area, wherein the metadata area stores data block mappings The contents of the table and the physical block reference count, the log area is designed as a fixed-size area consisting of N transaction entries, and a bitmap is used to mark the usage of different entries. 6. A metadata management method for deduplication, wherein the method comprises: Receive the I/O request to the data, cut the data into data blocks of a fixed size, and use a hash algorithm to perform fingerprint calculation to establish a unique fingerprint for the data block; In the fingerprint index structure preloaded in the memory, query and compare the fingerprints of the data blocks to determine whether to hit the same data block; Creating a transaction entry in the form of a bitmap, wherein the one I/O request corresponds to a transaction entry, and the transaction entry has nothing to do with whether the fingerprint is hit; Scan the bitmap through the metadata manager, obtain the transaction entry in the free state, and assign a globally unique Trans-id variable to the transaction entry; If a request to hit the fingerprint index is received, the logical block number accessed by the upper layer is mapped to the target physical block number in the fingerprint map, the reference count of the corresponding physical block number is increased by 1, and the reference count of the original mapped physical block number is decreased; If the request for the fingerprint index is not hit, the I/O request is forwarded downward, and the deduplication metadata is written into the metadata area of the persistent memory, and the written data block is stored; If the reference count of the originally mapped physical block number is decremented to 0, the deletion deletes the physical block number whose reference count is 0. 7. The method of claim 6, wherein the transaction entry includes a free state, a start state, and an end state. 8. The method of claim 7, wherein the transaction entry comprises a Trans-id variable, a state variable, and at least one metadata update backup; wherein the state variable is used to characterize the transaction Entries are in one of the free, start, and end states described above; the Trans-id variable is used as the sequence number of transaction entries and the order in which they were checkpointed. 9. The method according to claim 8, wherein the cutting the data into data blocks of a fixed size specifically comprises: Based on memory page and file system block size, set the deduplication data block size to 4KB.

Images