CN105068757B - A kind of redundant data De-weight method based on file semantics and system real-time status - Google Patents

Abstract

The invention discloses a redundant data deduplication method based on file semantics and system real-time status. The method is mainly realized by three functional modules: deduplication priority calculation module (MPD module) based on multi-semantic dimension division, layered Level data deduplication module (deduplication device) and deduplication control module (controller) based on the real-time status of the system. Based on multi-dimensional file semantics, the MPD module outputs file objects that are prioritized for deduplication operations, and the deduplication device executes hierarchical deduplication sequentially based on the above output, including deduplication at the global file level and local deduplication based on data block level deduplication At the same time, during the operation of the deduplication device, the controller will dynamically adjust the deduplication device according to the real-time status of the system, so as to save more storage while ensuring the read request response performance of the distributed primary storage system Space cost overhead.

Description

A redundant data deduplication method based on file semantics and real-time system status

technical field

The invention belongs to the technical field of computer information management, and in particular relates to a redundant data deduplication method based on file semantics and system real-time status.

Background technique

With the further popularization and in-depth application of cloud computing and mobile Internet, various online applications and services are playing a more important role in all walks of life. is growing at an explosive rate. The distributed storage system is the background support system of various cloud services. All service data are stored in the distributed storage system. The storage system provides a unified read-write interface for users or upper-level application services to access or modify and store in the disk. The data. Corresponding to different application scenarios and main purposes, distributed storage systems can be roughly divided into two categories: backup storage systems and primary storage systems. The backup storage system is mainly used for the backup of cold data, such as system logs, historical files, etc. These backup systems are generally built on relatively cheap storage hardware devices, or even the underlying storage devices that can be based on tape storage, because the backup storage system The access to the data is very low, and historical data is generally only read when there is a special need. Therefore, the backup storage system does not have high requirements for data read and write performance. In contrast, the primary storage system generally refers to a system that stores data directly accessed by upper-layer application services, and has higher requirements for data access performance, because the efficiency of reading and writing directly determines the user's The experience of upper-layer applications and services. The main storage system usually takes the data block as the basic unit, divides the file into data blocks and stores them in the underlying disk, and then maintains an index about all data blocks in the memory of the main storage system. The purpose of the index is It is to record the information of the file to which the data block belongs and the physical location of the data block on the disk. Since the main storage system needs to have high data access performance, in a distributed environment, system developers usually set up a set of redundant data storage and management mechanisms, that is, the same data, in the distributed More than one backup will be kept in the main storage system, and two files with very similar structures, such as two project folders containing a large number of identical files, will also keep two complete folders.

Before the arrival of the big data era, our storage devices needed to store and process a relatively small amount of data. Even if the hardware equipment needed to build a distributed primary storage system is expensive, the hardware overhead has not become a service provider. factors that need attention. In the era of big data, with the rapid increase of various Internet applications and services, the number of users is increasing every day, and the scale of services and applications is becoming larger and more complex. Therefore, the main storage system supporting various cloud services There has also been an explosive growth in the amount of data that needs to be stored. These data need to be directly accessed by upper-layer applications and services, so cheap backup storage systems cannot be used to assist storage. Although the cost of storage hardware devices continues to decrease with technological improvements, and devices with larger capacities can be purchased at lower prices, but because in the era of big data, the amount of data is growing at an exponential rate, so In practice, the growth rate of the amount of stored data has exceeded the rate of decline in the cost of hardware storage devices, and purchasing more storage devices cannot fundamentally cope with the surge in the amount of data. Moreover, in terms of economic benefits, this is a method that consumes a lot of input costs.

Driven by this background and challenges, redundant data deduplication technology has gradually attracted more and more attention from service providers, especially service providers who need to store large amounts of data in the primary storage system. From a general point of view, redundant data deduplication technology is to compare the "signatures" between data, if the data with the same signature is found, it will be judged as redundant data, and then the redundant data will be The data is deleted, and then the information of the deleted data is updated in the index, and its physical disk location points to the location of the retained redundant data. When the user or application accesses the deleted data next time, the system will direct the request to the location of the redundant data kept on the disk according to the information in the index, and perform the operation requested by the user on the data. From the granularity of data deduplication technology, it can generally be divided into two categories: deduplication at the file level and deduplication at the data block level. In a nutshell, data deduplication technology at the file level only deletes redundant identical two files, and its comparison unit is the entire file. The comparison and processing unit of the deduplication technology at the data block level is refined to the data block. As mentioned above, the underlying storage of the distributed primary storage system is actually based on the block. A file may It will be divided into multiple blocks and stored on the disk. In this case, each data block will be calculated to belong to the signature of this block. The deduplication technology is based on the comparison of signatures based on data blocks. remaining data blocks. According to the scope of data deduplication technology, global deduplication and local deduplication can be differentiated. The concept of scope works in a distributed storage system. In a nutshell, the global deduplication technology will detect redundant data on all servers in the system, even if the servers where these data are located are geographically separated. , while local deduplication technology only focuses on redundant data on the same server or on the same storage device. From the perspective of execution timing of data deduplication technology, it can be divided into two types: offline deduplication and online deduplication. Offline deduplication generally refers to the deduplication program running in the background of the system, which detects and deletes redundant data after the new file data is written to the disk, while online deduplication refers to the new data being written During the process of detection and redundancy deletion.

Combining the above classifications, the deduplication technology solutions commonly implemented and used in actual storage systems are a combination of the above classifications, and the most commonly used solutions are two types: global deduplication based on the file level (GlobalFile -level Deduplication, GFD) and local deduplication based on the data block level (Local Chunk-level Deduplication, LCD). Although these two solutions can meet certain deduplication requirements to a certain extent, their design does not consider the balance and trade-off between the application scenario of the primary storage system and other factors. First of all, the data deduplication technology will cause the loss of data reading performance. GFD may cause the data accessed by the user to be transmitted from the remote server, which will cause delays in network transmission, and LCD will cause local disks. Fragmentation, so that multiple disk seeks may be required when reading a certain data, resulting in a delay in the response to the read operation. The greater the deduplication, the more the overhead on the storage device can be saved, but the read performance will be affected more severely. Therefore, the balance and trade-off between deduplication and read performance is a primary consideration. Secondly, the primary storage is different from the backup storage system. The data of the latter is cold, infrequently accessed and usually stored as a historical record. Therefore, from the perspective of the system, these data can be uniformly viewed as is a binary byte stream. In the main storage system, due to the characteristics of directly supporting upper-layer services, the data stored in it is very diverse, and due to different types of services and different access characteristics of user groups, it has certain semantics. The semantics of these files should be used in the deduplication scheme design for the main storage system. Third, since the main storage system directly deals with users, the user's access characteristics will be different due to different time, different regions, or changes in services, and users will also vary from person to person, so for the main storage system The deduplication scheme needs to be dynamic and can be adjusted, so as to better make a better choice in terms of the balance between read performance and storage space efficiency.

To sum up, in the era of big data, cloud service providers have an urgent need to reduce the cost of storage space on the one hand, and on the other hand, hope that the deduplication of redundant data will not affect the performance of upper-layer applications and services. If it is too affected, the user experience can be guaranteed. How to design and implement an efficient redundant data deduplication solution to achieve system space efficiency by considering the use characteristics and data characteristics of the distributed primary storage system, and how to use the rich file semantics and changing system status different from the backup storage system The high-efficiency balance between data upload and data read performance has become an important problem that those skilled in the art urgently need to solve.

Contents of the invention

Aiming at the above-mentioned technical problems existing in the prior art, the present invention provides a redundant data deduplication method based on file semantics and real-time system status, which enables the distributed primary storage system to maintain high read request response performance while , reducing the overhead of storage space costs.

A redundant data deduplication method based on file semantics and real-time system status is as follows:

Periodically detect the read response delay and deduplication ratio of the distributed storage system; according to the read response delay and deduplication ratio of the system at the current moment, the following dynamic adjustment mechanism based on SLA (Service Level Agreement) is adopted to The system's deweighter adjusts:

According to the SLA currently referenced by the system, determine whether the read response delay of the system at the current moment is greater than 1.1 times the upper limit of the SLA read response delay interval:

If yes, make the deduplicator stop executing GFD and LCD for the system in the next cycle; if not, judge whether the read response delay of the system at the current moment is less than the upper limit of the read response delay interval of the SLA:

If not, make the deduplication device keep executing GFD on the system in the next cycle, and stop executing LCD; if so, determine whether the deduplication ratio of the system at the current moment is less than the lower limit of the deduplication ratio interval of the SLA:

If yes, make the de-weighter execute GFD and LCD normally on the system in the next cycle; if not, make the SLA currently referenced by the system raise a level, and re-judgment according to the above dynamic adjustment mechanism according to the new SLA.

The read response delay of the system at the current moment is the average response delay of all read requests from users to the system in the previous cycle; the deduplication ratio of the system at the current moment is the total volume of redundant data removed by the system from the initial moment to the current moment The ratio of the accumulated and stored data volume from the initial moment to the current moment when the system does not remove redundant data.

If the read response delay of the system at the current moment is greater than 1.1 times the upper limit of the SLA read response delay interval where the system is currently located, the de-duplicator will stop executing GFD and LCD for the system in the next cycle; if the de-duplicator is added in the next cycle If the system stops executing GFD and LCD for three consecutive cycles, the current SLA referenced by the system will be lowered by one level.

The level breakdown of said SLA is as follows:

For the first level of SLA, the read response delay interval is [0, 600ms], and the deduplication ratio interval is [0.25, 1);

For the second-level SLA, the read response delay interval is [0, 600ms], and the deduplication ratio interval is [0.1, 0.25];

For the third-level SLA, the read response delay interval is [600ms, 750ms], and the deduplication ratio interval is [0.25, 1);

For the fourth level of SLA, the read response delay interval is [600ms, 750ms], and the deduplication ratio interval is [0.1, 0.25];

For the fifth-level SLA, the read response delay interval is [750ms, 900ms], and the deduplication ratio interval is [0.25, 1);

For the sixth-level SLA, the read response delay interval is [750ms, 900ms], and the deduplication ratio interval is [0.1, 0.25];

For the seventh-level SLA, the read response delay interval is [900ms, 1200ms], and the deduplication ratio interval is [0.25, 1);

For the eighth level of SLA, the read response delay interval is [900ms, 1200ms], and the deduplication ratio interval is [0.1, 0.25].

The deduplication device uses a redundant data deduplication scheme based on priority levels to deduplicate the system, and the specific process is as follows:

(1) Based on the file semantics, calculate the deduplication priority of each file in the system;

(2) Extract the file location records of the n files with the largest deduplication priority from the global file index of the system, where n is a natural number greater than 1;

(3) According to the file location record, the redundant data of the above n files is deduplicated hierarchically:

First, perform GFD on the n files, and then update the global file index; initially, the file location records of the n files are marked as "dirty", and the file location records of the files after removing redundant data through GFD are marked as "dirty". for "clean";

Then, LCD is performed on the files whose file location records are still marked as "dirty" after GFD processing, and then the local data block index is updated;

(4) Repeat steps (1) to (3).

In the described step (1), the deduplication priority of each file is calculated by the following formula:

ρ＝α ₁ H+α ₂ G+α ₃ E

Among them: ρ is the deduplication priority of the file, H is the priority value of the latest access time of the file, G is the priority value of the file size, E is the priority value of the file type, α ₁ ~ α ₃ are the corresponding H, G and E respectively The weight coefficient of .

The criteria for determining the priority value H are as follows:

If the latest access time of the file is greater than 1 month, then H=7;

If the latest access time of the file is greater than 15 days and less than or equal to 1 month, then H=6;

If the latest access time of the file is greater than 7 days and less than or equal to 15 days, then H=5;

If the latest access time of the file is greater than 3 days and less than or equal to 7 days, then H=4;

If the latest access time of the file is greater than 1 day and less than or equal to 3 days, then H=3;

If the latest access time of the file is greater than 12 hours and less than or equal to 1 day, then H=2;

If the latest access time of the file is greater than 6 hours and less than or equal to 12 hours, then H=1;

If the latest access time of the file is less than or equal to 6 hours, then H=-1.

The criteria for determining the priority value G are as follows:

If the file size is greater than 1G, then G=7;

If the file size is greater than 512MB and less than or equal to 1G, then G=6;

If the file size is greater than 256MB and less than or equal to 512MB, then G=5;

If the file size is greater than 64MB and less than or equal to 256MB, then G=4;

If the file size is greater than 8MB and less than or equal to 64MB, then G=3;

If the file size is greater than 1MB and less than or equal to 8MB, then G=2;

If the file size is greater than 128KB and less than or equal to 1MB, then G=1;

If the file size is less than or equal to 128KB, then G=-1.

The criteria for determining the priority value E are as follows:

If the file type is a backup log, then E=7;

If the file type is a mirror image, then E=6;

If the file type is project, then E=5;

If the file type is video, then E=4;

If the file type is audio, then E=3;

If the file type is document, then E=2;

If the file type is a picture, then E=1;

If the file type is other, then E=-1.

The present invention has the following significant beneficial effects compared with the prior art due to the adoption of the above-mentioned technical scheme:

(1) Compared with the traditional redundant data deduplication scheme, the method of the present invention has an obvious advantage in that it makes full use of the rich file semantics contained in the main storage system, and obtains values for attributes of multi-dimensional file semantics Carry out quantitative division, give different files a specific deduplication priority value, and by assigning different weights between different dimensions, quantitatively calculate the deduplication priority of a certain file in numerical values, so that deduplication The server can treat different files differently, so as to save storage space and maintain an efficient response to the read request of the upper application.

(2) The method of the present invention introduces a dynamic adjustment mechanism for the deduplication program during operation and a mechanism for uniformly describing the deduplication scheme requirements. The unified requirements description mechanism can enable the functional modules in the redundant data deduplication solution to qualitatively obtain the specific needs of users, and can clearly arrange the multi-level trade-offs of users in terms of read performance and space efficiency according to priorities. The dynamic adjustment can make the deduplication of the present invention perceptible to the system state, and the working mode of the deduplication device can be adjusted to different degrees when the response performance of the read request is affected.

(3) The method of the present invention introduces a double-layer deduplication program mechanism combining GFD and LCD, and adopts different mechanisms at different times. Compared with the traditional single-mechanism redundant data de-duplication scheme, the double-layer hybrid de-duplication mechanism proposed by the present invention increases the flexibility in de-duplication granularity and de-duplication scope, allowing the controller to Real-time status of demand and system, by adjusting the running timing between two different schemes to achieve the effect of meeting higher priority SLA.

Description of drawings

FIG. 1 is a schematic diagram of the functional architecture of the redundant data deduplication method of the present invention.

FIG. 2 is a schematic flow chart of the redundant data deduplication method of the present invention.

detailed description

In order to describe the present invention more specifically, the technical solutions of the present invention will be described in detail below in conjunction with the accompanying drawings and specific embodiments.

As shown in Figure 1, in the actual operating application environment, the redundant data deduplication scheme based on file semantics and real-time system status of the present invention runs in a general-purpose distributed primary storage system, and the storage system mainly includes metadata servers and object storage server cluster; where:

The metadata server is responsible for receiving user requests and directing the requests to the corresponding object storage servers. It is also responsible for detecting the running status of the entire distributed primary storage system and maintaining a global index in units of file names. The index contains the "signature", location information and metadata information of each file. This signature is unique to each file and is calculated on the binary content of the file through the SHA-1 algorithm.

The object storage server cluster includes multiple independent object storage servers. Each server saves a certain amount of files. The local disk of the server supports the storage of file data in the form of data blocks. Each object storage server Responsible for maintaining the index information in units of data blocks of files stored locally. Likewise, the index contains the physical location of the data block on the local disk, as well as a signature calculated using the SHA-1 algorithm based on the binary content of the data block.

This embodiment mainly includes three functional modules: deduplication priority calculation module (MPD module) based on multi-semantic dimension division, hierarchical data deduplication module (deduplication device), and deduplication control based on real-time system status module (controller). These three functional modules all run on the metadata server or object storage server, and can interact and operate with the global file index of the metadata server and the data block index on each object storage server, among which:

The MPD module runs on the metadata server. It calculates the list of files with high deduplication priority by obtaining the files and file auxiliary information stored on the metadata server. The MPD module is based on multi-dimensional file semantics and periodicity. Output a list, which contains a fixed number of file location records, and these files are the objects that the de-duplicator should prioritize to implement de-duplication. The MPD module of this solution performs deduplication priority calculation based on the three-dimensional semantics of the file: the latest access timestamp of the file, the size of the file, and the type of the file. On this basis, the MPD module clearly assigns the range of specific values in each dimension, so that the attribute values of each file in these three dimensions correspond to a clear value, and this data marks a certain file. For the deduplication priority in this dimension, the larger the value, the more priority is given to the deduplication operation. Moreover, these three dimensions have different weight coefficients, and the final deduplication priority of the file is the weighting of the corresponding values in these three dimensions. In practical applications, these assignments can be customized according to specific service scenarios and requirements. The default division standards and corresponding assignments in this embodiment, as well as the weight assignments of different dimensions are shown in Table 1:

Table 1

According to the values in Table 1, the final deduplication priority value of each file=recent access time priority value*0.5+file size priority value*0.3+file type priority value*0.2. The MPD module periodically scans the file index in the metadata server and includes the information of the top 50 files with the highest deduplication priority into a list, which is called a deduplication candidate list. Each file information occupies one line of the table, and each line also has an attached flag bit, and each flag bit is "dirty" at initialization, indicating that the row of files has not been processed by the deduplicator; correspondingly, If the deduplicator has finished processing the file in this line, the flag will be changed to "clean".

The deduplicator runs on each object storage server. It receives the list output by the MPD module as input, and deduplicates the files in the list hierarchically. The deduplication device will perform double-layer deduplication operations, namely deduplication GFD at the global file level and deduplication LCD at the local data block level. The deduplication device periodically obtains the latest deduplication candidate list from the MPD module, and then starts deduplication operation from the file with the largest deduplication priority value. If the controller allows it, the deduplicator will first implement the GFD operation on the file. In order to find whether there are redundant files in the global scope, the local deduplicator will initiate a query request to the metadata server, because A global file index is saved in the metadata server, so it can be known whether the file has different redundant backups distributed on different object storage servers. If redundant file backups are found through query, the current deduplication device will send a deduplication request to the object server where the redundant files are located, and the other object server will notify its local deduplication device to remove the redundant files from the local disk delete it. After the deduplication device that initiates the deduplication request learns that the redundant file has been deleted, it will set the flag bit of the file record in the deduplication priority list to "clean", and at the same time notify the metadata server to update the file index and delete the newly deleted file. The location information of the file is directed to the object storage server where the deduplicator is located.

After the deduplication device completes traversing the deduplication priority list in the GFD mode, it will traverse the list again in the LCD mode from the beginning if the controller allows it. The deduplication device implements LCD by searching the index of the data block on the local object storage to determine whether there is a data block redundant with one or some blocks of the current file locally. If redundant data blocks exist, after the redundant blocks are deleted, the index of the local object storage server is updated accordingly, and the corresponding file entry flag bit in the deduplication candidate list is marked as "clean". After the LCD process has also traversed the deduplication candidate list, the local deduplication device notifies the metadata server that the deduplication priority list of this period has been processed.

During the operation of the de-heavy device, the controller will dynamically adjust the de-heavy strategy of the de-weight device according to the real-time status of the system. The controller is a distributed component that runs on the metadata server and each object storage server at the same time. During the running process, the controller will dynamically implement the deduplication strategy of the deduplication device according to the real-time status of the system. Adjustment. The part on the metadata server is responsible for monitoring the delay status of the response to the read request of the entire distributed primary storage system, and the part running on each object storage server is responsible for monitoring the storage space occupancy of the server. The controller dynamically adjusts the deduplication device according to a group of <read response delay range, deduplication ratio range> demand pairs that can be set in advance by the user and arranged according to priority. We call the collection of a hierarchical set of demand pairs with priorities above a service level agreement (SLA). This embodiment provides a fixed format of the SLA, so the SLA can be refined and set by the user according to this format, so as to be used as the input of the controller; the example format of the SLA is shown in Table 2:

Table 2

When a read request from a user arrives at the metadata server, the metadata server takes the timestamp at this time as the start response timestamp for the read request, inserts the timestamp into the request message, and then forwards the request to the corresponding on the object storage server. The object storage server that receives the read request completely reads out the object corresponding to the read request, and when the last data block starts to be sent to the client, the time stamp at this time is used as the end time stamp of the read request. The interval between the start timestamp and the end timestamp is the response delay of the request. The delay is captured by the controller subcomponent distributed on each object storage server, and sent to the controller component located on the metadata server, and the controller component located on the metadata server collects the control information from each object storage server The response delay sent by the device component relative to each read request, and then every fixed cycle T, calculate the average value of all read response delays in this cycle T, and this value is used as the read response in this cycle Delay reference value.

The deduplication ratio is the ratio of the volume of redundant data removed to the total volume of stored data before deduplication. After the deduplication device located on each object storage server completes the GFD or LCD deduplication operation on a certain file, it sets the flag bit in the deduplication priority list to "clean", and then records in the file item After adding the volume size of redundant data removed. When the metadata server reclaims each deduplication priority list that has been traversed, it counts the volume of the data removed from the list from the table, and then divides it by the entire distribution that can be directly obtained from the global file index According to the size of all files in the primary storage system, the real-time deduplication ratio is obtained.

The controller located on the metadata server will regularly detect and query the read response delay value and deduplication ratio of the entire distributed primary storage system in each cycle T, and then combine the specific demand range in the SLA to perform deduplication. Dynamic adjustment, as shown in Figure 2, the main operations include the following processes:

The controller starts from the SLA with the lowest priority level. If the current read response delay value meets or exceeds the demand range of the current SLA, and the immediate deduplication ratio meets or exceeds the upper limit of the deduplication ratio of the current level, deduplication is allowed. The server continues to work normally, and the current SLA priority level is raised by one level;

If the current read response latency value meets or exceeds the demand range of the current level, but the immediate deduplication ratio is lower than the lower limit of the deduplication ratio of the current level, the deduplication device is allowed to continue to work normally, and the current SLA is kept at current level;

If the current read response delay value does not meet the current demand range, there are two cases: a) If the current read response delay value is greater than 1.1 times the upper limit of the read delay demand range of the current level, the controller stops the deduplication For all operations, if the read response delay of the system has not fallen back to a range less than 1.1 times the upper limit of the read delay requirement range of the current level within three consecutive cycles, the current SLA level will be lowered by one level; b) if The current read response delay value is greater than the upper limit of the current demand range but less than 1.1 times the upper limit of the demand, the controller still stays at the demand level, and sends an instruction to the deduplication device to stop the LCD deduplication operation and only keep the GFD operation.

The above description of the embodiments is for those of ordinary skill in the art to understand and apply the present invention. It is obvious that those skilled in the art can easily make various modifications to the above-mentioned embodiments, and apply the general principles described here to other embodiments without creative efforts. Therefore, the present invention is not limited to the above-mentioned embodiments, and improvements and modifications made by those skilled in the art according to the disclosure of the present invention should fall within the protection scope of the present invention.

Claims (7)

1. a kind of redundant data De-weight method based on file semantics and system real-time status, as follows：

The periodically reading response delay and duplicate removal ratio of detection distributed memory system；Responded according to the reading of current time system Time delay and duplicate removal ratio, treasure is gone to be adjusted system using below based on SLA dynamic mechanisms：

According to the SLA of the current institute's reference of system, when judging whether the reading response delay of current time system is more than the SLA and reads response Prolong the section upper limit 1.1 times：

If so, treasure is then set to stop performing GFD and LCD to system within next cycle；If it is not, judge current time system Read the reading response delay section upper limit whether response delay is less than the SLA：

If it is not, then making treasure retain system within next cycle performs GFD, stop performing LCD；If so, when judging current Whether the duplicate removal ratio of etching system is less than the duplicate removal rate terms lower limit of the SLA：

If so, then make treasure execution GFD and LCD normal to system within next cycle；If it is not, then make the current institute's reference of system SLA lift a grade, and judgement is re-started according to above-mentioned dynamic mechanism according to new SLA；

Wherein：The GFD represents the global duplicate removal based on file level, and the LCD represents local based on data block aspect Duplicate removal, the grade of the SLA is detailed as follows：

The SLA of the first estate, it be [0,600ms] that it, which reads response delay section, duplicate removal rate terms be [0.25,1)；

The SLA of second grade, it is [0,600ms] that it, which reads response delay section, and duplicate removal rate terms are [0.1,0.25]；

The SLA of the tertiary gradient, it be [600ms, 750ms] that it, which reads response delay section, duplicate removal rate terms be [0.25,1)；

The SLA of the fourth estate, it is [600ms, 750ms] that it, which reads response delay section, and duplicate removal rate terms are [0.1,0.25]；

The SLA of 5th grade, it be [750ms, 900ms] that it, which reads response delay section, duplicate removal rate terms be [0.25,1)；

The SLA of 6th grade, it is [750ms, 900ms] that it, which reads response delay section, and duplicate removal rate terms are [0.1,0.25]；

The SLA of 7th grade, it be [900ms, 1200ms] that it, which reads response delay section, duplicate removal rate terms be [0.25,1)；

The SLA of 8th grade, it is [900ms, 1200ms] that it, which reads response delay section, and duplicate removal rate terms are [0.1,0.25].

2. redundant data De-weight method according to claim 1, it is characterised in that：If during the reading response of current time system Prolong 1.1 times for being presently in SLA more than system and reading the response delay section upper limit, then treasure is stopped within next cycle to system Only perform GFD and LCD；If removing treasure plus next cycle, continuous three cycles stop performing GFD and LCD to system, The SLA of the current institute's reference of system is set to decline a grade.

3. redundant data De-weight method according to claim 1, it is characterised in that：Described goes treasure to use based on preferential Spend hierarchical redundant data duplicate removal scheme and duplicate removal is carried out to system, detailed process is as follows：

(1) file semantics are based on, calculate the duplicate removal relative importance value of each file in system；

(2) the document location record of n maximum file of duplicate removal relative importance value is extracted from the global profile index of system, n is big In 1 natural number；

(3) recorded according to document location and hierarchical redundant data duplicate removal is carried out to above-mentioned n file：

First, GFD is carried out to this n file, and then updates global profile index；Initially, the document location record of this n file It is marked as " dirty ", and then " clean " is labeled as the document location record by file after GFD removal redundant datas；

Then, to after GFD is handled document location record still carry out LCD labeled as the file of " dirty ", and then update local number Indexed according to block；

(4) performed repeatedly to (3) according to step (1).

4. redundant data De-weight method according to claim 3, it is characterised in that：By following in described step (1) Formula calculates the duplicate removal relative importance value of each file：

ρ=α₁H+α₂G+α₃E

Wherein：ρ is the duplicate removal relative importance value of file, and H is the preferred value of file last access time, and G is the preferred value of file size, E be file type preferred value, α₁~α₃Respectively correspond to H, G and E weight coefficient.

5. redundant data De-weight method according to claim 4, it is characterised in that：The preferred value H calibrates standard such as really Under：

If file last access time is more than 1 month, H=7；

If file last access time was more than 15 days and less than or equal to 1 month, H=6；

If file last access time was more than 7 days and less than or equal to 15 day, H=5；

If file last access time was more than 3 days and less than or equal to 7 day, H=4；

If file last access time was more than 1 day and less than or equal to 3 day, H=3；

If file last access time was more than 12 hours and less than or equal to 1 day, H=2；

If file last access time was more than 6 hours and less than or equal to 12 hour, H=1；

If file last access time is less than or equal to 6 hours, H=-1.

6. redundant data De-weight method according to claim 4, it is characterised in that：The preferred value G calibrates standard such as really Under：

If file size is more than 1G, G=7；

If file size is more than 512MB and is less than or equal to 1G, G=6；

If file size is more than 256MB and is less than or equal to 512MB, G=5；

If file size is more than 64MB and is less than or equal to 256MB, G=4；

If file size is more than 8MB and is less than or equal to 64MB, G=3；

If file size is more than 1MB and is less than or equal to 8MB, G=2；

If file size is more than 128KB and is less than or equal to 1MB, G=1；

If file size is less than or equal to 128KB, G=-1.

7. redundant data De-weight method according to claim 4, it is characterised in that：The preferred value E calibrates standard such as really Under：

If file type is backup log, E=7；

If file type is mirror image, E=6；

If file type is project, E=5；

If file type is video, E=4；

If file type is audio, E=3；

If file type is document, E=2；

If file type is picture, E=1；

If file type is other, E=-1.