CN106528001B - A kind of caching system based on nonvolatile memory and software RAID - Google Patents

Abstract

The present invention proposes a cache system based on non-volatile memory and software RAID, which is characterized by a novel storage system architecture integrating non-volatile memory Apache Pass, software RAID and Device Mapper driver module Buaacache. The invention utilizes the Device Mapper mechanism of Linux to manage the software RAID, and uses the high-speed device Apache Pass as the Cache of the low-speed RAID, thereby constructing a two-level cache system. Persistent Memory has the characteristics of no loss of media data after power failure and excellent read and write performance. The present invention selects Persistent Memory as the high-speed cache of software RAID, and under the condition that the cost is far lower than that of hardware RAID, the reading and writing performance of software RAID approaches or even exceeds that of hardware RAID.

Description

A Cache System Based on Non-Volatile Memory and Software RAID

technical field

The invention relates to the technical field of computer data storage, in particular to a novel cache system, in particular to a novel power-down non-volatile memory Persistent Memory and software RAID.

Background technique

With the rise of SSDs, mainstream storage systems are gradually adopting a hierarchical storage method that mixes SSDs and mechanical hard disks. The current storage solutions such as dm-cache are designed and implemented based on the Linux Device Mapper framework, which is essentially a general-purpose block device cache module. Using SSDs with good random read and write performance, SSDs are used as disk caches. Compared with SSD, the new non-volatile memory (Persistent Memory, referred to as PMEM, also known as persistent memory) has high-speed I/O, large storage capacity, non-volatile power-off and byte-seeking Therefore, a new cache system based on Persistent Memory and software RAID is proposed.

Non-volatile memory: Persistent Memory, referred to as PMEM, is characterized by the fact that information will not be lost after a complete power failure. In addition to ensuring that the media data will not be lost after power failure, it also has the characteristics of byte addressable (Byte-addressable) and superior I/O performance. In the write back mode (write back), depending on the material, the write time and read time of this type of device are close to DRAM, so it is very attractive to use it as a software RAID cache. Existing PMEM research solutions are mostly focused on the hardware field, and no manufacturer or research institution has yet proposed a solution combining PMEM and software RAID.

AEP: Apache Pass, a new generation of Persistent Memory developed by Intel for the server platform "Purley" to be launched in 2017, is used as a software RAID cache in the cache system.

Software RAID: Software Redundant Arrays of Independent Disks, called software redundant array of independent disks, referred to as software array. Compared with hardware RAID, software RAID does not require an external RAID card to implement, but aggregates several underlying storage devices such as mechanical hard drives or solid-state drives into a large virtual RAID device in the form of pure software. With PMEM as a high-speed cache, software RAID can approach or even surpass the latter in terms of performance while the cost is much lower than that of hardware RAID. In the Linux system, software RAID is currently implemented in the form of MD (Multiple Devices) virtual block devices, using multiple underlying block devices to virtualize a new virtual block device, and using striping (stripping) technology to evenly distribute data blocks to Multiple disks to improve the read and write performance of the virtual device. Currently under Linux, software RAID supports RAID-0, RAID-1, RAID-4, RAID-5, RAID-6, and RAID-10.

Bottleneck of the traditional software RAID solution: It depends heavily on the CPU performance and current load of the server. In the write-back mode, a power failure will cause the data in the memory to be lost and cannot be recovered. Compared with hardware RAID, the advantage is not obvious. PMEM is a popular new storage device that has emerged in recent years. At present, no manufacturers and research institutions have released a solution that combines PersistentMemory and software RAID. Therefore, the present invention selects Persistent Memory with excellent performance in terms of non-loss of power-off medium data and I/O performance. As the high-speed cache of software RAID, a new high-efficiency cache system is proposed.

Contents of the invention

The purpose of the present invention is to address the shortcomings of traditional software RAID solutions in power-down recovery and read-write performance, and to propose a solution for the combination of non-volatile memory (Persistent Memory, PMEM) and software RAID. A caching system based on PMEM and software RAID.

The present invention is based on the Device Mapper disk mapping mechanism of Linux, develops the kernel driver module Buaacache, and uses the high-speed device PMEM as the high-speed cache of low-speed RAID, thereby constructing a two-level cache system.

The kernel driver module Buaacache of the present invention creates a virtual mixed block device with low-speed RAID and high-speed PEME as a whole, and users can use the mixed block device like reading and writing common block devices. The cache device PMEM and the cache object RAID in the virtual block device are invisible to the user and cannot be directly accessed by the user. When the user reads and writes the newly created virtual block device, the actual storage process is cached by Buaacache. If the read and write hits, the data in the cache will be returned directly (or directly written to the cache), otherwise the disk array will be read and written.

The data space of the cache device PMEM in the present invention is divided into a metadata space and a data space, and the data space is divided into cache blocks of a predefined size by using a group associative mapping method, and the cache blocks are further combined into a group according to a fixed number (Set). The metadata mainly records the one-to-one mapping relationship between a hard disk block and a certain cache block in a certain group, and reads and writes are realized through PMEM's unique byte addressable (Byte-addressable) method.

The invention makes up for the defect of power-off data loss in the traditional write-back mechanism by virtue of the characteristic that the medium data is not lost after the PMEM device is powered off, avoids multiple accesses to the disk array, and greatly shortens the reading and writing time. By adopting the method of writing back the memory metadata in time, the metadata can be backed up to the cache device in real time during the operation of the cache system, and the system operation site can be saved. When a power failure occurs, the host can restore the system operation site from the cache device after the host is powered on again, preventing the dirty data in the cache from being written back to the disk due to the loss of the corresponding relationship with the cached disk block, and realizing power-off recovery Function.

The present invention selects the open source program dm-cache for secondary development. Since dm-cache is a part of the Linux mature kernel, one or more fast devices can be used to play the role of cache for the back-end slow storage system. However, the latest version of dm-cache can only be compiled and run on the kernel version 3.10, and the kernel version span is too large to run normally on the new version of the stable kernel. In view of the fact that there is no new kernel porting version issued yet, the present invention has carried out targeted porting work.

Description of drawings

Figure 1 Linux I/O hierarchy diagram of the cache system;

Figure 2 Buaacache driver module diagram;

Figure 3 shows the overall architecture of the cache system;

Figure 4 cache device Apache Pass structure diagram;

Figure 5 is a schematic diagram of a cache system mapping method;

Figure 6 is a schematic diagram of a cache system read request flow;

Figure 7 is a schematic diagram of a cache system write request flow;

Fig. 8 schematic diagram of kernel transplantation work;

Figure 9 is a schematic diagram of bio kernel transplantation;

Figure 10 Schematic diagram of part of the kernel transplantation of Device Mapper;

Detailed ways

In order to make the object, technical solution and advantages of the present invention more clearly, the present invention will be further described in detail below in conjunction with the accompanying drawings and specific embodiments.

The purpose of the present invention is aimed at the deficiencies in the storage performance and easy loss of power-off data of the current disk array system based on software RAID, and proposes a cache system based on non-volatile memory (Persistent Memory, PMEM for short) and software RAID. The invention is based on the Device Mapper mechanism of Linux, develops the kernel driver module Buaacache, and adopts the PMEM device Apache Pass developed by Intel as the high-speed cache of low-speed RAID. The software RAID is created through the "-C" or "-create" parameter of the mdadm software under the Linux4.4 kernel, and is managed by mdadm.

The layers of the present invention in the Linux I/O system are shown in FIG. 1 . As shown in Figure 2, Buaacache runs under the Linux64-bit stable kernel version 4.4, works in the block device driver layer in the Linux block device I/O architecture, and is registered as the target driver (Target Driver) under the Linux kernel Device Mapper framework. It is dynamically loaded and used as a kernel driver. The driver module invokes the dmsetup program in Device Mapper to create the software disk array and ApachePass as a whole as a hybrid block device, and the user can operate the device like a common block device.

The system structure of the present invention is shown in Figure 3. When a user executes a read-write request, the request is first captured by the file system, and the file system sends the request to the general block device layer, specifically presented in the form of the kernel data structure bio, and then Continue to send it to the Device Mapper layer. After the Device Mapper layer splits and reorganizes the bio, it finally sends it to the Buaacache driver layer.

The cache structure of the present invention is as shown in Figure 4, and the first cache block of each nonvolatile memory Apache Pass can be used as a super block (Super Block) to store key information related to the entire cache, and can also be reserved as it uses. Next, the metadata space stores some metadata (Meta Data), and the metadata area stores metadata for managing data cache blocks (DataBlock). The data area is divided into several data cache blocks (Data Block) of a specified size. By default, every 8 sectors form a cache block, and every 512 cache blocks form a group (Set). At the same time, the hard disks in the software RAID are also divided according to this. Each cache block records the starting sector number of the corresponding hard disk block, which is recorded by the metadata structure, and each metadata structure maintains the corresponding state, necessary lock and tag information. Metadata and data blocks are stored linearly in sequential order.

The cache mapping method of the present invention is shown in FIG. 5 . The present invention adopts a 512-way group associative mapping mechanism to avoid aggregation of high-temperature data represented by metadata. Group associative mapping can be divided into two steps: direct mapping and linear hash detection: the first step: direct mapping, mapping disk blocks to a group in the cache device; second step: linear hash detection, once the target group is found , just search the corresponding data block linearly in the group; through the group associative mapping, the sequential data blocks in the local range on the disk can be cached into a target group.

The cache reading and writing flow charts of the present invention are shown in FIG. 6 and FIG. 7 respectively, and the cache recycling mechanism of the present invention only supports the write-back mode. When the user's read and write requests miss the cache, and all the cache blocks are dirty and there are no free cache blocks, the cache reclamation mechanism uses the Two-Handed clock algorithm to write back the dirty blocks that have not been accessed in the last 20 minutes to the disk array for maintenance. data consistency.

The power-off recovery mechanism of the present invention is shown in FIG. 8 . When the user's read and write requests miss, once there is metadata update, the present invention writes the cache device from the memory in a byte addressing manner in time. After an unexpected power outage, the metadata in the memory is backed up in the cache device in time. After the host is powered on again, the Buaacache driver module will reload the backed-up metadata in the cache device into the memory, and then the cache device The dirty data corresponding to the middle metadata (data not written back to the disk array in time from the cache due to power failure) is written back to RAID to realize power failure recovery.

The present invention carries out secondary development based on the open-source solution dm-cache, as described in the summary of the invention, transplants dm-cache from kernel 3.10 version to kernel 4.4 long-term support version. In the transplantation work, the kernel change tracking is performed on the block device read and write request field bio and some fields in DeviceMapper, and corresponding adaptations are made. bio is a core data structure of the general block layer in the linux kernel, which is responsible for contacting the memory buffer and the block device.

As shown in Figure 9, compared with the kernel version 3.10, the bio field in the kernel version 4.4 has a new bvec_iter structure, which encapsulates the four key fields of bi_sector, bi_size, bi_idx and bi_bvec_done in the original bio.

As shown in Figure 10, the Device Mapper framework provides synchronous or asynchronous IO services. In kernel 4.4, dm-io has four dm_io_mem_type types. Compared with kernel version 3.10, in kernel version 4.4, dm_io_memory uses struct bio*bio instead of struct bio_vec, and dm_io_memtype uses DM_IO_BIO instead of DM_IO_BVEC.

Claims (2)

1. A cache system based on nonvolatile memory and software RAID, characterized in that: a fusion of nonvolatile memory Apache Pass, software RAID and Device Mapper driver module Buaacache's novel cache system architecture; The non-volatile memory refers to a new type of storage device with the characteristics of no loss of information after complete power failure, byte addressing and I/O performance, and is used as a cache of software RAID in a cache system; Described software RAID refers to in Linux operating system, uses "-C" or "-create" parameter of mdadm software to finish creating, is managed by mdadm; Described RAID is redundant RAID or non-redundant RAID; The Device Mapper driver module Buaacache is based on the kernel driver module Buaacache under the Linux Device Mapper framework, works in the block device driver layer in the Linux block device I/O architecture, logically uses the non-volatile memory Apache Pass as a low-speed software RAID cache (Cache), physically creates software RAID and non-volatile memory as a whole as a hybrid block device; Buaacache driver module includes cache data organization module, cache mapping module, cache recycling module, metadata management module and Power failure recovery module; The cache data organization module refers to dividing Apache Pass into a reserved area, a metadata area and a data cache block area, the reserved area is responsible for storing the file system super block or is reserved for other purposes, and the metadata area is responsible for storing the main record cache block and hard disk The metadata of the block correspondence, the data cache block area stores data in the form of cache block organization; The cache mapping module refers to dividing the disks in the Apache Pass and the software RAID into several groups (Set) according to a fixed size, adopting a 512-way group associative mapping mechanism to avoid aggregation of high-temperature data represented by metadata ;Group associative mapping is divided into two steps: direct mapping and linear hash detection; Described cache reclamation module refers to adopting Two-Handed clock algorithm, writes back the dirty piece that Apache Pass has not visited in the last 20 minutes to disk array; The metadata management module refers to writing metadata from memory to Apache Pass or loading metadata from Apache Pass to memory, and the addressing mode is byte addressing; when writing cache from RAID or writing back dirty from Apache Pass When the block is transferred to RAID, the metadata update operation is performed; The power-down recovery module refers to once there is metadata update, write back to Apache Pass in time with byte addressing mode; After power-on, the Buaacache driver module reloads the backed-up metadata in the cache device into the memory, and then writes the corresponding dirty data in the cache device back to the disk array according to the metadata to realize power-off recovery. 2. the caching system based on nonvolatile memory and software RAID according to claim 1, is characterized in that: described Device Mapper driver module Buaacache carries out secondary development based on open source scheme dm-cache, dm-cache is from kernel The 3.10 version is ported to the kernel 4.4 long-term support version. The specific porting steps include steps 1.1 and 1.2: Step 1.1: Add a bvec_iter structure in the bio field of the block device read and write request field, and encapsulate the four key fields of bi_sector, bi_size, bi_idx and bi_bvec_done in bio; Step 1.2: In the Device Mapper fields dm_io_memory and dm_io_memtype, replace struct bio_vec with structbio*bio, and replace DM_IO_BVEC with DM_IO_BIO, respectively.