CN100347655C - Data storage system and data storage control device - Google Patents

Abstract

Data storage system and data storage control device. Storage systems having multiple control modules for controlling multiple storage devices make installation easier while maintaining low latency response even as the number of control modules increases. Multiple storage devices are connected to the second interfaces of each control module by using a back-end router, thereby maintaining redundancy for all control modules to access all storage devices. Moreover, the control module and the first switching unit are connected through a serial bus, which has a small number of signals, and an interface is formed by using a backplane. Thus, mounting on a printed circuit board becomes possible.

Description

Data storage system and data storage control device

technical field

The present invention relates to a configuration of a data storage system and a data storage control device, which are used in an external storage device of a computer, and more particularly, to a data storage system and a data storage control device, having the ability to Combination and connection of units constructing a data storage system connecting multiple disk devices with high performance and high flexibility.

Background technique

Recently, as various data are computerized and processed on computers, data storage devices (external storage devices) capable of efficiently storing a large amount of data for processing with high reliability independently of a host computer performing data processing ) becomes more and more important.

Such a data storage device uses a disk array device having many disk devices such as magnetic disks and optical disks and a disk controller for controlling the many disk devices. Such a disk array device can simultaneously receive disk access requests from a plurality of host computers, and can control many disks.

Recently, a disk array device capable of controlling a group of disk devices having several thousand or more disk devices each having hundreds of terabytes in itself has been proposed.

Such a disk array device encapsulates a memory that functions as a part of the disk buffer. Accordingly, it is possible to reduce the data access time when receiving a read request or a write request from the host computer, and achieve higher performance.

Generally speaking, a disk array device is composed of the following main units, namely: a channel adapter, which is the connection part with the host computer; a disk adapter, which is the connection part with the disk drive; a buffer memory; a cache control unit, which manages the buffer memory; and many disk drives.

Figure 11 is a schematic diagram showing the first prior art. The disk array device 102 shown in FIG.

These two cache managers 10 are directly connected via a bus 10c, thereby enabling communication. The two cache managers 10 and 10, the cache manager 10 and the channel adapter 11, and the cache manager 10 and the disk adapter 13 are respectively connected via a PCI bus because low latency is required.

The channel adapter 11 is connected with a host computer (not shown) by such as Fiber Channel (Fibre Channel) or ^Ethernet (Ethernet ⁾ ), and the disk adapter 13 is connected with each drive of the disk package group 12 by cables such as Fiber Channel. connect.

The disk packaging group 12 has two ports (eg, Fiber Channel ports), and the two ports are connected to different disk adapters 13 . This provides redundancy which increases failure resistance.

FIG. 12 is a block diagram showing a disk array device 100 according to a second prior art. As shown in accompanying drawing 12, traditional disk array device 100 has: cache manager (represented by CM in the figure) 10, and this cache manager 10 is made up of cache memory and the cache control unit as main unit; Denoted by CA) 11, which is an interface with a host computer (not shown); a disk package group 12, which is composed of a plurality of disk drives; and a disk adapter (denoted by DA in the figure) 13, which is an interface with this disk drive 12 .

The disk array device also has: a router (represented by RT in the figure) 14 for interconnecting the buffer memories 10; a channel adapter 11 and a disk adapter 13 for data transfer and communication between these main units.

This disk array device 100 includes four cache managers 10 and four routers 14 corresponding to these cache managers 10 . These cache managers 10 and routers 14 are connected to each other one-to-one, so the connection between a plurality of cache managers 10 is redundant, and accessibility is improved (for example, Japanese Patent Application Laid-Open No. 2001- 256003).

In other words, even if one router 14 fails, the connection between the plurality of cache managers 10 can be ensured by means of another router 14, and even in this case, the disk array device 100 can continue normal operation .

In this disk array device 100 , each router 14 is connected to two channel adapters 11 and two disk adapters 13 , and the disk array device 100 includes a total of eight channel adapters and a total of eight disk adapters 13 .

These channel adapters 11 and disk adapters 13 are able to communicate with all cache managers 10 through the interconnection of cache managers 10 and routers 14 .

The channel adapter 11 is connected with the host computer processing data by means of Fiber Channel (Fibre Channel) or ^Ethernet ( ^Ethernet ), and the disk adapter 13 is connected with the disk packaging group 12 (specifically disk drive) through (for example) Fiber Channel connected.

In addition, between the channel adapter 11 and the cache manager 10 and between the disk adapter 13 and the cache manager 10, not only user data from the host computer but also internal operations for maintaining the disk array device 100 (such as , various information about the consistency of data mirroring between multiple buffer memories).

Cache manager 10, channel adapter 11, and disk adapter 13 are connected to router 14 through interfaces, which can achieve lower latency than communication between disk array device 100 and host computer or disk array device 100 and disk drives. For example, cache manager 10, channel adapter 11, and disk adapter 13 are connected to router 14 through a bus designed to connect LSI (Large Scale Integration) and printed circuit boards, such as a PCI (Peripheral Component Interconnect) bus.

A disk package group 12 for housing disk drives has two fiber channel ports, and these two ports are respectively connected to disk adapters 13 belonging to different routers 14 . Thus, even when a failure occurs in the disk adapter 13 or the router 14 , it is possible to prevent disconnection of the connection with the cache manager 10 .

Due to recent advances in computerization, there is a need for greater capacity and faster data storage systems. In the case of the disk array device of the first kind of prior art mentioned above, if the buffer manager 10, the channel adapter 11 and the disk adapter 13 are expanded to increase capacity and speed, the number of disk packaging groups 12 must be increased. The number of ports and the number of connecting cables between the disk adapter 13 and the disk packaging group 12 must be increased.

Increasing the number of ports of the disc pack 12 increases the number of cables corresponding to the number of disc adapters to be connected to one disc pack, which increases the installation space. This means that the size of the device will increase. It is also not a good idea to increase the number of ports since there is only two path systems for a disk package group to achieve sufficient redundancy. Moreover, the number of disk adapters to be connected is not constant, but will change according to user needs, so if a lot of ports are expanded and a small number of disk adapters are used, it will cause waste, but if a small number of ports are expanded, it will be wasteful. Cannot support many disk adapters. In other words, flexibility is lost.

On the other hand, in the case of the disk array device of the second prior art, it is feasible to expand the cache manager 10, the channel adapter 11, and the disk adapter 13, but all communication is performed through the router 14, so the communication Data is all funneled into router 14, which becomes a traffic bottleneck, so high traffic cannot be expected. And in the case of the disk array device 100, if a large-scale disk array device with many main units is constructed, the number of connections between the cache manager 10 and the router 14 will increase sharply, which will cause complicated connection relationships and make it difficult to install Physically it can become difficult.

For example, in the case of the structure shown in FIG. 12 , four (four slices) cache managers 10 and four routers 14 are connected through a backplane 15 , as shown in FIG. 13 . In this case, as shown in FIG. 12, the number of signals is (4×4×(number of signal lines per path)). For example, if one path is connected by means of 64-bit PCI (parallel path), the number of signal lines on the backplane 15 is 100×16=1600 including control lines. To form these signal lines, the printed circuit board on the backplane 15 requires six signal layers.

In the case of a large-scale configuration, such as a configuration where eight (four-chip) cache managers 10 and eight (four-chip) routers 14 are connected through the backplane 15, the required number of signal lines is about 100×8 ×8=6400. Therefore, the printed circuit board of the backplane 15 needs 24 layers, which is four times that of the above case, and it is very difficult to realize such a structure.

If four PCI-Express buses with fewer signal lines than the 64-bit PCI bus are used for connection, the number of signal lines is 16×8×8=1024. However, the PCI bus operates at a frequency of 66MHz, while the PCI-Express bus is a 2.5Gbps high-speed bus. In order to maintain the signal quality of the high-speed bus, expensive substrate materials must be used.

If a low-speed bus is used, the wiring layer can be replaced by using vias, but in the case of a high-speed bus, vias should be avoided because they degrade the signal quality. Therefore in the case of a high-speed bus, it needs to be laid out so that all signal lines do not cross, so about twice as many signal layers are required as compared to a low-speed bus with the same number of signal lines. For example, circuit boards require 12 signal layers, and these layers must be constructed using expensive materials, so this is also difficult to achieve.

And in the case of the disk array device 100 of the second prior art, if one of the routers 14 breaks down, the channel adapter 11 and the disk adapter 13 connected with this router 14 also cannot be used when this router 14 breaks down. of.

Contents of the invention

In view of the aforementioned problems, an object of the present invention is to provide a data storage system and a data storage control device, which are used to realize high-flow data transfer between units, and can easily realize small-scale to large-scale structures, while Does not cause installation problems.

Another object of the present invention is to provide a data storage system and a data storage control device which have the flexibility to easily realize a small-scale to large-scale structure in combination of the same units while maintaining operational redundancy.

Still another object of the present invention is to provide a data storage system and a data storage control device for easily implementing small-scale to large-scale structures without causing installation problems while maintaining high throughput and redundancy.

In order to achieve these objects, the data storage system of the present invention has: a plurality of storage devices with one or more access ports for storing data; and a plurality of control modules for performing Access control of the storage device. And the control module further includes: a buffer memory for storing a part of data stored in the storage device; a buffer control unit for controlling the buffer memory; a first interface unit for communicating with the first interface unit The cache control unit is connected to control the connection with the host; the second interface unit is connected to the cache control unit and is used to control the connection with the plurality of storage devices; and a first switching unit, configured to connect between the plurality of control modules and the plurality of storage devices, and used to selectively switch the second interface unit of each control module and the plurality of storage devices Each of the control modules is connected to the plurality of first switching units, and each of the storage devices is connected to one or more different first switching units. And each of the second interface units of the plurality of control modules is connected to the plurality of first switching units through a serial bus using a backplane.

The data storage control device of the present invention has: a buffer memory for storing a part of data stored in the storage device; a buffer control unit for controlling the buffer memory; a plurality of control modules with a first interface unit, the The first interface unit is connected to the cache control unit for controlling the connection with the host, and the second interface unit is connected to the cache control unit for controlling the connection with the multiple the connection of a plurality of storage devices; and a plurality of first switching units, which are arranged between the plurality of control modules and the plurality of storage devices, and are used to selectively switch the second interface unit of each control module and the plurality of storage devices. The plurality of storage devices, each of the control modules is connected to the plurality of first switching units, and each of the storage devices is connected to one or more different first switching units. And each of the second interface units of the plurality of control modules is connected to the plurality of first switching units through a serial bus using a backplane.

In the present invention, preferably, the cache control unit and the second interface unit are connected through a high-speed serial bus with low latency, and the second interface unit and the plurality of first switching units use The backplanes are connected through a serial bus.

In the present invention, preferably, the control module further includes a communication unit for communicating with another control module, and also includes a second switching unit for selectively connecting one of the control modules communication unit.

In the present invention, preferably, the communication unit and the second switching unit of each control module are connected by using a backplane.

In the present invention, preferably, the first switching unit and the plurality of storage devices are connected through cables.

In the present invention, preferably, the storage device further includes a plurality of access ports, and a plurality of different first switching units are connected to the plurality of access ports.

In the present invention, preferably, the cache control unit and the second interface unit are connected through multiple high-speed serial buses, and the second interface unit and the plurality of first switching units use The backplanes are connected via a serial bus.

In the present invention, preferably, the high-speed serial bus is a PCI-Express bus.

In the present invention, preferably, the serial bus is a fiber channel.

In the present invention, preferably, the cache control unit and the first interface unit are connected through a high-speed serial bus with low latency.

In the present invention, the second interface of each control module is connected to a plurality of first switching units, so all control modules can maintain the redundancy of accessing all storage devices, and even if the number of control modules increases, The control module and the first switching unit are also connected via a serial bus (with a small number of signals constituting the interface) using the backplane, so mounting on a printed circuit board is feasible.

Description of drawings

Accompanying drawing 1 is a block diagram showing a data storage system according to an embodiment of the present invention;

Accompanying drawing 2 is a block diagram showing the control module in accompanying drawing 1;

Accompanying drawing 3 is a block diagram showing the backend router and disk encapsulation group in accompanying drawing 1 and accompanying drawing 2;

Accompanying drawing 4 is the block diagram showing the disc packing group in accompanying drawing 1 and accompanying drawing 3;

Accompanying drawing 5 is the schematic diagram showing the reading processing of the structure in accompanying drawing 1 and accompanying drawing 2;

Accompanying drawing 6 is the schematic diagram showing the writing process of the structure in accompanying drawing 1 and accompanying drawing 2;

Accompanying drawing 7 is the schematic diagram showing the installation structure of the control module according to the embodiment of the present invention;

Accompanying drawing 8 is the schematic diagram showing the installation structure example of the data storage system according to the embodiment of the present invention;

Accompanying drawing 9 is a block diagram showing a large-scale storage system according to an embodiment of the present invention;

Accompanying drawing 10 is a block diagram showing a medium-scale storage system according to another embodiment of the present invention;

Accompanying drawing 11 is a block diagram showing a storage system according to the first prior art;

Accompanying drawing 12 is a block diagram showing a storage system according to a second prior art;

Fig. 13 is a schematic diagram showing the installation structure of the storage system according to the second prior art in Fig. 12 .

Detailed ways

Embodiments of the present invention will now be described in the order of data storage system, read/write processing, mounting structure and other embodiments.

data storage system

Accompanying drawing 1 is a block diagram showing a data storage system according to an embodiment of the present invention, accompanying drawing 2 is a block diagram showing a control module in accompanying drawing 1, and accompanying drawing 3 is showing a rear view in accompanying drawing 1 Figure 4 is a block diagram showing the disk encapsulation group in Figures 1 and 3.

Figure 1 shows a mass storage system with eight control modules as an example. As shown in the accompanying drawing 1, the storage system 1 has: a plurality of disk package groups 2-0 to 2-25 for storing data; data processing unit) and the control modules 4-0 to 4-7 between the plurality of disk packaging groups 2-0 to 2-25; To 4-7 and a plurality of disk encapsulation groups 2-0 to 2-25 back-end router (the first switching unit: marked as BRT in the figure, referred to as BRT hereinafter) 5-0 to 5-7 and many There are (two in this case) front-end routers (second switching unit: marked as FRT in the figure, hereinafter referred to as FRT) 6-0 to 6-1.

Each of the control modules 4-0 to 4-7 has a cache manager 40, channel adapters (first interface unit: labeled CA in the figure) 41a to 41d, disk adapters (second interface unit: labeled DA in the figure ) 42a and 42b and a DMA (Direct Memory Access) engine (communication unit: marked as DMA in the figure) 43.

In the accompanying drawing 1, in order to simplify the drawing, only the control module 4-0 is marked with reference numerals buffer manager "40", channel adapters "41a", "41b", "41c" and "41d", disk "42a" and "42b" of the adapter and "43" of the DMA, and these reference numerals of the constituent elements in the other control modules 4-1 to 4-7 are omitted.

The control modules 4-0 to 4-7 will now be described with reference to FIG. 2 . The cache manager 40 performs read/write processing according to a processing request (read request or write request) from the host computer, and the cache controller 40 has a cache memory 40b and a cache control unit 40a.

The buffer memory 40b holds a part of the data stored in the plurality of disks of the disk package groups 2-0 to 2-25, that is, it functions as a buffer for the plurality of disks.

The buffer control unit 40a controls the buffer memory 40b, the channel adapter 41, the device adapter 42, and the DMA 43. To this end, the cache control unit 40a has one or more (two in FIG. 2 ) CPUs 400 and 410 and a memory controller 420. The memory controller 420 controls read/write of each memory, and switches paths.

The memory controller 420 that is connected with the buffer memory 40b through the memory bus 434 is connected with the CPUs 400 and 410 through the CPU buses 430 and 432, and also through the four-way high-speed serial bus (for example, PCI-Express) that will be mentioned later. 440 and 442 are connected to disk adapters 42a and 42b. Likewise, the memory controller 420 is connected to the channel adapters 41a, 41b, 41c and 41d via four high-speed serial buses (for example, PCI-Express) 443, 444, 445 and 446, and is connected to the channel adapters 41a, 41b, 41c and 41d via four high-speed serial buses (for example, , PCI-Express) 447 and 448 are connected with DMA 43-a to 43-b.

It will be introduced later that this high-speed bus (such as PCI-Express) communicates in groups, and by setting multiple serial buses, it becomes possible to communicate with a very fast response speed, and the communication carried out like this has a small Latency, i.e. having very low latency, even with reduced number of signal lines.

The channel adapters 41a to 41d are interfaces for host computers, and the channel adapters 41a to 41d are respectively connected to different host computers. Preferably, the channel adapters 41a to 41d are respectively connected to the interface units of the corresponding host computers by means of buses such as Fiber Channel and Ethernet( ^R) (Ethernet ^(R )), and in this case, optical fiber or coaxial cables are used as bus.

Each of these channel adapters 41a to 41d constitutes a part of the respective control modules 4-0 to 4-7, but must support multiple kind of agreement. Since the protocol to be installed is different depending on the corresponding host computer, the buffer manager 40, which is the main unit of the control modules 4-0 to 4-7, is installed on different printed circuit boards, as described later in FIG. 7 In that way, the respective channel adapters 41a to 41d can be easily replaced when necessary.

Examples of protocols with the host computer that the channel adapters 41a to 41d should support are iSCSI (Internet Small Computer System Interface) corresponding to Fiber Channel and Ethernet ^(R ) mentioned above. The respective channel adapters 41 a to 41 d are directly connected to the cache manager 40 via a bus designed to connect LSI (Large Scale Integration) with a printed circuit board (such as the aforementioned PCI-Express bus). Thereby, the high throughput required between the respective channel adapters 41a to 41d and the cache manager 40 can be achieved.

The disc adapters 42a and 42b are the interfaces of the disc encapsulation groups 2-0 to 2-25 to disc drives, and are connected to the BRTs 5-0 to 5-7 connected to the disc encapsulation groups 2-0 to 2-25, for For this connection, four FC (Fibre Channel)) ports are used. The respective disk adapters 42a and 42b are directly connected to the cache manager 40 through the above-mentioned bus (such as PCI-Express) designed to connect LSI (Large Scale Integration) with a printed circuit board. Thus, the high traffic required between the respective disk adapters 42a and 42b and the cache manager 40 can be achieved.

As shown in Figures 1 and 3, the BRTs 5-0 through 5-7 are multi-port switches that selectively switch and communicatively connect the disk adapters 42a and 42b of the respective control modules 4-0 through 4-7 Package groups 2-0 to 2-25 with individual discs.

As shown in FIG. 3, a plurality (two in this case) of BRTs 5-0 to 5-1 are connected to respective disk package groups 2-0 to 2-7. As shown in FIG. 4 , each disk package group 2-0 has a plurality of disk drives 200 each having two ports, and this disk package group 2-0 also has unit disk package groups 20-0 to 23 -0, these unit disk package groups have four connection ports 210 , 212 , 214 and 216 . These unit disk package groups are connected in series in order to achieve an increase in capacity.

In the disk package groups 20 - 0 to 23 - 0 , each port of each disk drive 200 is connected to two ports 210 and 212 via a pair of FC cables drawn from the two ports 210 and 212 . These two ports 210 and 212 are connected with different BRTs 5-0 and 5-1, as shown in FIG. 3 .

As shown in FIG. 1, the disk adapters 42a and 42b of the respective control modules 4-0 to 4-7 are connected to all the disk package groups 2-0 to 2-25, respectively. In other words, the disk adapters 42a of the respective control modules 4-0 to 4-7 are respectively connected to the BRT 5-0 (see accompanying drawing 3) on the disk packaging groups 2-0 to 2-7, connected to the disk packaging groups BRT 5-2 on 2-8, 2-9, ——, BRT 5-4 connected to disk package group 2-16, 2-17, —— and connected to disk package group 2-24, 2- 25. Connect with BRT 5-6 on ——.

In the same way, the disc adapters 42b of each control module 4-0 to 4-7 are respectively connected to the BRT 5-1 (see accompanying drawing 3) on the disc package group 2-0 to 2-7, connected to the disc package BRT 5-3 on groups 2-8, 2-9, -, connected to BRT 5-5 on reel package groups 2-16, 2-17, - and connected to reel package groups 2-24, 2 -25, - the BRT 5-7 on - is connected.

Thus, multiple (in this case two) BRTs are connected to each disk package group 2-0 to 2-31, and different disk adapters 42a and 42b in the same control module 4-0 to 4-7 Two BRTs connected to the same tray package group 2-0 to 2-31 are respectively connected.

With this structure, each control module 4-0 to 4-7 can access all the disk package groups (disk drives) 2-0 to 2-31 via the disk adapter 42a or 42b.

Each of these disk adapters 42a and 42b constituting a part of the control modules 4-0 to 4-7 is mounted on a circuit board of the cache manager 40, which is a part of the control modules 4-0 to 4-7. The main unit of each disk adapter 42a and 42b is directly connected with the cache manager 40 by means of, for example, a PCI (Peripheral Component Interconnect)-Express bus, and thus, the requirements between the respective disk adapters 42a and 42b and the cache manager 40 can be realized. of high traffic.

Also as shown in Figure 2, each disk adapter 42a and 42b is connected to a corresponding BRT 5-0 to 5-7 by means of a bus (such as Fiber Channel and Ethernet ^(R )). In this case, the bus is mounted on the printed circuit board of the backplane via electronic wiring, described later.

The disk adapters 42 and 42b of each control module 4-0 to 4-7 and the BRT 5-0 to 5-7 are connected in a one-to-one network, thereby being connected to all disk packaging groups. As previously mentioned, as long as the control As the number of modules 4-0 to 4-7 (in other words, the number of disk adapters 42a and 42b) increases, the number of connection lines increases and the connection relationship becomes more complicated, which makes physical installation difficult. But when using Fiber Channel (few number of signals, small interface formed) for connection between disk adapters 42a and 42b and BRT 5-0 to 5-7, mounting on printed circuit board became feasible.

When the respective disk adapters 42a and 42b are connected to the corresponding BRTs 5-0 to 5-7 through Fiber Channel, the BRTs 5-0 to 5-7 become Fiber Channel switches. The individual BRTs 5-0 to 5-7 are also connected to the corresponding reel packaging groups 2-0 to 2-31, for example by fiber-optic channels, and in this case, due to the different modules, fiber optic cables 500 and 510 are used for the connection. As shown in FIG. 1, the DMA engine 43 communicates with other control modules 4-0 to 4-7, and is responsible for communication and data transfer processing with other control modules 4-0 to 4-7. The respective DMA engines 43 of the respective control modules 4-0 to 4-7 constitute part of the control modules 4-0 to 4-7, and are mounted on the circuit board of the buffer manager 40, which is the control module 4 -Major units from 0 to 4-7. And the DMA engine 43 is directly connected with the cache manager 40 by means of the aforementioned high-speed serial bus, and communicates with the DMA engines 43 of other control modules 4-0 to 4-7 via the FRTs 6-0 and 6-1.

The FRTs 6-0 and 6-1 are connected to the DMA engines 43 of a plurality (specifically three or more, in this case eight) of the control modules 4-0 to 4-7, and selectively These control modules 4-0 to 4-7 are switched and communicatively connected.

With this structure, each DMA engine 43 of each control module 4-0 to 4-7 executes communication and data transfer processing (for example, mirroring processing) in connection with this control module in accordance with an access request from a host computer. Between the buffer manager 40 on the module and the buffer manager 40 of other control modules 4-0 to 4-7, it takes place via FRT 6-0 and 6-1.

As shown in accompanying drawing 2, the DMA engine 43 of each control module 4-0 to 4-7 is made up of multiple (in this case, two) DMA engines 43-a and 43-b, and these two DMA engines Each of 43-a and 43-b uses these two FRTs 6-0 and 6-1.

The DMA engines 43-a and 43-b are connected to the cache manager 40 through, for example, the above-mentioned PCI-Express bus in order to achieve low latency.

In the case of communication and data transfer processing between the respective control modules 4-0 to 4-7 (in other words, between the buffer managers 40 of the respective control modules 4-0 to 4-7), the data transfer amount High, preferably to reduce the time required for communication, and require high throughput and low latency (fast response speed). Therefore, as shown in accompanying drawing 1 and accompanying drawing 2, the DMA engine 43 of each control module 4-0 to 4-7 and FRT 6-0 and 6-1 are by using high-speed serial transmission (PCI-Express or Rapid- IO) bus, which is designed to meet the requirements of high traffic and low latency at the same time.

PCI-Express and Rapid-IO use 2.5Gbps high-speed serial transmission, and use a small amplitude differential interface called LVDS (Low Voltage Differential Signaling) as its bus interface.

read/write processing

The read processing of the data storage system in FIGS. 1 to 4 will now be described. FIG. 5 is a schematic diagram showing a read operation of the structure in FIG. 1 and FIG. 2 .

When the buffer manager 40 receives a read request from a host computer via the corresponding channel adapters 41a to 41d, if the buffer memory 40b stores the target data of this read request, the cache manager 40 will save the target data in the buffer memory 40b. This object data in 40b is sent to the host computer via channel adapters 41a to 41d.

If this data is not stored in the buffer memory 40b, then the cache control unit 40a reads the target data from the disc drive 200 that has saved this data into the buffer memory 40b, and then sends the target data to the main computer.

This reading process for the disk drive will be described with reference to FIG. 5 .

(1) The control unit 40a (CPU) of the cache manager 40 creates an FC header and a descriptor in the descriptor area of the cache memory 40b. The descriptor is an instruction to request data (DMA) transfer to the data transfer circuit (DMA circuit), including the address of the FC header on the buffer memory, the address of the data transferred to the buffer memory, the number of data bytes and the disk for data transfer logical address.

(2) The data transfer circuit of the disk adapter 42 is started.

(3) The activated data transfer circuit of the disk adapter 42 reads the descriptor from the buffer memory 40b.

(4) The activated data transfer circuit of the disk adapter 42 reads the FC header from the buffer memory 40b.

(5) The started data delivery circuit of the disk adapter 42 analyzes the descriptor, and obtains data about the requested disk, the first address and the number of bytes, and delivers the FC header to the target disk via the Fiber Channel 500 (510) Drive 200. Disk drive 200 reads the requested target data and sends it to the data transfer circuitry of disk adapter 42 via Fiber Channel 500 (510).

(6) The disk drive 200 reads the requested target data, and when the transfer is completed, sends a completion notification to the data transfer circuit of the disk adapter 42 via the fiber channel 500 (510).

(7) When the completion notification is received, the activated data transfer circuit of the disk adapter 42 reads the read data from the memory of the disk adapter 42 and stores it into the buffer memory 40b.

(8) When the read transfer is completed, the activated data transfer circuit of the disk adapter 42 sends a completion notification to the cache manager 40 through an interrupt.

(9) When receiving an interrupt parameter from the disk adapter 42, the control unit 40a of the cache manager 40 confirms the read delivery.

(10) The control unit 42a of the cache manager 40 checks the end pointer of the disk adapter 42, and confirms that the read transfer is completed.

All connection lines must have high traffic to achieve sufficiently high performance, and since especially the handshaking between the cache control unit 40a and the disk adapter 42 is very frequent (seven times in FIG. 5 ), a particularly low Waiting time for the bus.

In this example, PCI-Express (quad) and Fiber Channel (4G) are used as high-traffic connections, but PCI-Express is a low-latency connection, while Fiber Channel connections have relatively high latency (data transfer takes time).

In the case of the second prior art, fiber channel with high latency cannot be used for RT 14 between CM 10 and DA 13 or CA 11 (see accompanying drawing 12), but in the present invention, with accompanying drawing 1 In the structure, Fiber Channel can be used for BRT 5-0 to 5-7.

To achieve low latency, the number of signals on the bus cannot be reduced to less than a certain amount, but according to the present invention, Fiber Channel using a small number of signal lines can be used for the connection between the disk adapter 42 and the BRT5-0, so this reduces the overhead. The number of signal lines on the board, which is very efficient for the installation.

Write operations will now be covered. When a write request from one of the host computers is received via the corresponding channel adapters 41a to 41d, the channel adapters 41a to 41d that have received the write request command and the write data ask the cache manager 40 where the write data should be written. The address of the buffer memory 40b.

When receiving a response from the cache manager 40, the channel adapters 41a to 41d write the write data into the buffer memory 40b of the cache manager 40, and also write the write data into a buffer memory other than this cache manager. In at least one cache manager 40 of 40 (in other words, the cache managers 40 in the different control modules 4-0 to 4-7) in the buffer memory 40b. To this end, the channel adapters 41a to 41d start the DMA engine 43 and write the write data to the buffer in the buffer manager 40 in another control module 4-0 to 4-7 via the FRTs 6-0 and 6-1. in memory 40b.

Here the write data is written in the buffer memory 40b of at least two different control modules 4-0 to 4-7, this is because the data will be copied (mirrored), so that even if the control modules 4-0 to 4-7 7 or an unexpected hardware failure of the cache manager 40 can also prevent data loss.

When the writing of the write data to the plurality of buffer memories 40b ends normally, the channel adapters 41a to 41d issue a completion notification to the host computers 3-0 to 3-31, and the processing ends.

This written data must also be written back into the target disk drive (writeback technique, writeback). The cache control unit 40a reverse-writes the written data of the buffer memory 40b to the disk drive 200 storing this target data according to an internal schedule. The writing process performed by this pair of disk drives will be described with reference to FIG. 6. FIG.

(1) The control unit 40a (CPU) of the cache manager 40 creates an FC header and a descriptor in the descriptor area of the cache memory 40b. This descriptor is an instruction to request data transfer (DMA) to the data transfer (DMA) circuit, and includes the address of the FC header on the buffer memory, the address of the data to be transferred to the buffer memory and the number of data bytes thereof, and the data transfer The logical address of the disk.

(2) The data transfer circuit of the disk adapter 42 is started.

(3) The activated data transfer circuit of the disk adapter 42 reads the descriptor from the buffer memory 40b.

(4) The activated data transfer circuit of the disk adapter 42 reads the FC header from the buffer memory 40b.

(5) The activated data transfer circuit of the disk adapter 42 analyzes the descriptor, and obtains data on the requested disk, head address, and byte count, and reads the data from the buffer memory 40b.

(6) After the reading is completed, the data transfer circuit of the disk adapter 42 transfers the FC header and data to the target disk drive 200 via the fiber channel 500 (510). The disk drive 200 writes the transferred data on the internal disk.

(7) When writing of data is completed, the disk drive 200 sends a completion notification to the data transfer circuit of the disk adapter 42 via the fiber channel 500 (510).

(8) When receiving the completion notification, the activated data transfer circuit of the disk adapter 42 sends a completion notification to the cache manager 40 by means of an interrupt.

(9) When receiving an interrupt parameter from the disk adapter 42, the control unit 40a of the cache manager 40 confirms the write operation.

(10) The control unit 42a of the cache manager 40 checks the end pointer of the disk adapter 42, and confirms that the write operation is completed.

In Fig. 6 and Fig. 5, the arrow marks represent the transfer of packet information (such as data), and the U-shaped arrow marks represent the data reading that sends data back to the data requesting end. Since it is necessary to confirm the start and end states of the control circuit in the DA, data should be exchanged seven times between the CM 40 and the DA 42 once the data is transferred. Data is exchanged between DA 42 and disc 200 twice.

From this, it can be understood that the connection line between the cache control unit 40 and the disk adapter 42 requires low latency, and on the other hand, an interface with a small number of signal lines can be used for the disk adapter 42 and the disk device 200 .

installation structure

Accompanying drawing 7 is a schematic diagram showing an example of an installation structure of a control module according to the present invention, and accompanying drawing 8 is a schematic diagram showing an example of an installation structure including a control module and a disc package group in accompanying drawing 7, and accompanying drawing 9 and Figure 10 is a block diagram showing a data storage system having these mounting structures.

As shown in FIG. 8, four disk package groups 2-0, 2-1, 2-8, and 2-9 are mounted on the upper side of the memory device main body. The control circuit is installed in the lower half of the storage device. The lower half is divided into front and rear parts by the back plate 7, as shown in Figure 7 . Slots are created on the front and rear sides of the backplane 7, respectively. In the case of the storage system having the large-scale structure in accompanying drawing 9, eight (eight pieces) of CM 4-0 to 4-7 are provided on the front side, and two (two pieces) of FRT 6 are provided on the rear side -0 and 6-1, eight (eight slices) BRT 5-0 to 5-7 and the service processor SVC responsible for power control (not shown in accompanying drawing 1 and accompanying drawing 9).

In accompanying drawing 7, eight slices of CM 4-0 to 4-7 and two slices of FRT 6-0 and 6-1 are connected by four-way PCI-Express through backplane 7. PCI-Express has four (differential and bidirectional) signal wires, so 16 signal wires are used for four channels, which means that the number of signal wires is 16×16=256. These eight pieces of CM 4-0 to 4-7 and eight pieces of BRT 5-0 to 5-7 are connected by optical fiber channels through the backplane 7. Fiber Channel (with differential and bidirectional signal lines) has 1 x 2 x 2 = 4 signal lines, so the number of signal lines is 8 x 8 x 4 = 256.

By using the bus differently depending on the connection position as described above, eight pieces of CM 4-0 to 4-7, two pieces of FRT 6-0 and 6-1 and eight pieces of BRT 5-0 can be realized by means of 512 signal lines to 5-7, even in the case where the storage system has a large-scale structure as shown in FIG. 9 . The number of this signal line is the number of signal lines that can be installed on the backplane substrate 7 in sufficient quantities, and the number of signal layers of the circuit board is six, which is sufficient and can be realized from the perspective of cost. within the range.

In accompanying drawing 8, four disc packing groups 2-0, 2-1, 2-8 and 2-9 (see accompanying drawing 9) are installed, and other disc packing groups 2-3 to 2-7 and 2 -10 to 2-31 mounted on different bodies.

The medium-scale storage system in FIG. 10 can also be implemented with a similar structure. In other words, the same architecture can be used to implement four-unit CM 4-0 to 4-3, four-unit BRT 5-0 to 5-3, two-unit FRT 6-0 to 6-1 and 16-module disk package groups 2-0 to 2-15 configurations.

Disk adapters 42a and 42b of each control module 4-0 to 4-7 are connected with all disk drives 200 by means of BRT, so that each control module 4-0 to 4-7 can connect all disk drives via disk adapter 42a or 42b to access.

These disk adapters 42a and 42b are respectively mounted on the circuit board of the cache manager 40, which is the main unit of the control modules 4-0 to 4-7, and the respective disk adapters 42a and 42b can be connected by means such as PCI-Express A low-latency bus such as the Cache Manager 40 is directly connected to the cache manager 40, so high traffic can be achieved.

The disk adapters 42a and 42b of each control module 4-0 to 4-7 are connected to the BRT 5-0 to 5-7 in a one-to-one mesh connection, so even if the number of control modules 4-0 to 4-7 of the system (change In other words, the number of disk adapters 42a and 42b) increases, and fiber channel (with a small number of signals constituting the interface) can also be used for connection between the disk adapters 42a and 42b and BRT 5-0 to 5-7, which solves the problem of installation question.

In the case of communication and data transfer processing between the respective control modules 4-0 to 4-7 (in other words, between the buffer managers 40 of the respective control modules 4-0 to 4-7), the data transfer amount It is very high, and it is best to reduce the time required for connection, and require high flow and low waiting time (very fast response speed), so as shown in Figure 2, each control module 4-0 to 4-7 The DMA engine 43 and FRT 6-0 and 6-1 are connected through PCI-Express, this bus is a bus using high-speed serial transmission, and is designed to meet the requirements of high traffic and low waiting time at the same time.

other embodiments

According to the above embodiments, the signal lines in the control module are introduced using PCI-Express, but other high-speed serial buses, such as Rapid-IO, can also be used. The number of channel adapters and disk adapters in the control module can be increased or decreased as required.

As the disk drive, storage devices such as hard disk drives, optical disk drives, and magneto-optical disk drives can be used.

The present invention is described using specific embodiments, but the present invention can also be embodied in various ways within the scope of the essential characteristics of the present invention, and these ways should not be excluded from the scope of the present invention outside.

Since the second interface of each control module is connected to a plurality of first switching units, all control modules can maintain the redundancy of accessing all storage devices, and even if the number of control modules increases, they can also use the backplane A row bus (with a small number of signals constituting the interface) connects the control module with the first switching unit, thus enabling mounting on a printed circuit board while maintaining low-latency communication within the control module. Therefore, the present invention is very effective for unifying the architecture from large scale to small scale, and can contribute to reducing the cost of the device.

Cross References to Related Applications

This application is based on and claims the priority of the prior Japanese patent application No. 2004-347411 filed on November 30, 2004 and the prior Japanese patent application No. 2005-022121 filed on January 28, 2005, The entire contents of these prior applications are incorporated herein by reference.

Claims (20)

1. data-storage system, this data-storage system comprises:

A plurality of memory storages with one or more access port are used to store data; With

A plurality of control modules are used for carrying out the access control of described memory storage according to the access instruction that comes from main frame,

Wherein said control module further comprises:

Memory buffer is used for storing the data that a part is stored in described memory storage;

Caching control unit is used to control described memory buffer;

First interface unit, described first interface unit is connected with described caching control unit, is used to control the interface with described main frame;

Second interface unit, described second interface unit is connected with described caching control unit, is used to control the interface with described a plurality of memory storages;

And wherein said data-storage system also comprises:

A plurality of first switch units, be used between described a plurality of control modules and described a plurality of memory storages, being connected, and be used for optionally described second interface unit and described a plurality of memory storage of each control module of switch, each described control module links to each other with described a plurality of first switch units, and each described memory storage described first switch unit different with one or more is connected; With

Backboard is used for being connected with described a plurality of described first switch units by universal serial bus each described second interface unit with described a plurality of control modules.

2. data-storage system according to claim 1, wherein said caching control unit is connected by the high-speed serial bus with low latency with described second interface unit, and described second interface unit uses described backboard to be connected by universal serial bus with described a plurality of first switch units.

3. data-storage system according to claim 1, wherein said control module also comprise be used for the communication unit that communicates with the described control module of another one and

Described system also comprises second switch unit, is used for optionally connecting the communication unit of each described control module.

4. data-storage system according to claim 3, wherein the communication unit of each control module is to use described backboard to be connected with second switch unit.

5. data-storage system according to claim 1, wherein said first switch unit is connected by cable with described a plurality of memory storages.

6. data-storage system according to claim 1, wherein said memory storage comprises a plurality of access ports,

And wherein said a plurality of first different switch unit is connected with described a plurality of access ports.

7. data-storage system according to claim 2, wherein said caching control unit is connected by the multipath high-speed universal serial bus with described second interface unit,

And described second interface unit is to use described backboard to be connected by universal serial bus with described a plurality of first switch units.

8. data-storage system according to claim 2, wherein said high-speed serial bus are the PCI-Express buses.

9. data-storage system according to claim 2, wherein said universal serial bus is an optical-fibre channel.

10. data-storage system according to claim 2, wherein said control module connects described caching control unit and described first interface unit by the high-speed serial bus with low latency.

11. a data recording control apparatus is used for being used to store according to the access instruction that comes from main frame the access control of a plurality of memory storages of data, this data recording control apparatus comprises:

A plurality of control modules comprise:

Memory buffer is used for storing the data that a part is stored in described memory storage;

Caching control unit is used to control described memory buffer;

First interface unit, described first interface unit is connected with described caching control unit, is used to control and being connected of described main frame;

Second interface unit, described second interface unit is connected with described caching control unit, is used to control and being connected of described a plurality of memory storages,

A plurality of first switch units, be used between described a plurality of control modules and described a plurality of memory storages, being connected, and be used for optionally described second interface unit and described a plurality of memory storage of each control module of switch, each described control module links to each other with described a plurality of first switch units, and each described memory storage described first switch unit different with one or more is connected; With

Backboard is used for being connected with described a plurality of first switch units by universal serial bus each described second interface unit with described a plurality of control modules.

12. data recording control apparatus according to claim 11, wherein said caching control unit is connected by the high-speed serial bus with low latency with described second interface unit,

And described second interface unit uses described backboard to be connected by universal serial bus with described a plurality of first switch units.

13. also comprising, data recording control apparatus according to claim 11, wherein said control module be used for the communication unit that communicates with the described control module of another one,

And described device also comprises second switch unit, is used for optionally connecting the communication unit of each described control module.

14. data recording control apparatus according to claim 13, wherein the communication unit of each control module is to use described backboard to be connected with second switch unit.

15. data recording control apparatus according to claim 11, wherein said first switch unit is connected by cable with described a plurality of memory storages.

16. data recording control apparatus according to claim 11, wherein said a plurality of first different switch units are connected with each the described memory storage with a plurality of access ports respectively.

17. data recording control apparatus according to claim 12, wherein said caching control unit is connected by the multipath high-speed universal serial bus with described second interface unit,

And described second interface unit is to use described backboard to be connected by universal serial bus with described a plurality of first switch units.

18. data recording control apparatus according to claim 12, wherein said high-speed serial bus are the PCI-Express buses.

19. data recording control apparatus according to claim 12, wherein said universal serial bus is an optical-fibre channel.

20. data recording control apparatus according to claim 12, wherein said caching control unit is to be connected by the high-speed serial bus with low latency with described first interface unit.

Images