[go: up one dir, main page]

CN107247564B - Method and system for data processing - Google Patents

Method and system for data processing Download PDF

Info

Publication number
CN107247564B
CN107247564B CN201710581341.6A CN201710581341A CN107247564B CN 107247564 B CN107247564 B CN 107247564B CN 201710581341 A CN201710581341 A CN 201710581341A CN 107247564 B CN107247564 B CN 107247564B
Authority
CN
China
Prior art keywords
data
node
abnormal
raid
forwarding
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
CN201710581341.6A
Other languages
Chinese (zh)
Other versions
CN107247564A (en
Inventor
赵鹏
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Suzhou Metabrain Intelligent Technology Co Ltd
Original Assignee
Suzhou Inspur Intelligent Technology Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Suzhou Inspur Intelligent Technology Co Ltd filed Critical Suzhou Inspur Intelligent Technology Co Ltd
Priority to CN201710581341.6A priority Critical patent/CN107247564B/en
Publication of CN107247564A publication Critical patent/CN107247564A/en
Application granted granted Critical
Publication of CN107247564B publication Critical patent/CN107247564B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • GPHYSICS
    • G06COMPUTING OR CALCULATING; COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G06F3/0601Interfaces specially adapted for storage systems
    • G06F3/0602Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
    • G06F3/062Securing storage systems
    • G06F3/0622Securing storage systems in relation to access
    • GPHYSICS
    • G06COMPUTING OR CALCULATING; COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F11/00Error detection; Error correction; Monitoring
    • G06F11/07Responding to the occurrence of a fault, e.g. fault tolerance
    • G06F11/14Error detection or correction of the data by redundancy in operation
    • G06F11/1402Saving, restoring, recovering or retrying
    • G06F11/1446Point-in-time backing up or restoration of persistent data
    • G06F11/1448Management of the data involved in backup or backup restore
    • GPHYSICS
    • G06COMPUTING OR CALCULATING; COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G06F3/0601Interfaces specially adapted for storage systems
    • G06F3/0628Interfaces specially adapted for storage systems making use of a particular technique
    • G06F3/0629Configuration or reconfiguration of storage systems
    • G06F3/0635Configuration or reconfiguration of storage systems by changing the path, e.g. traffic rerouting, path reconfiguration
    • GPHYSICS
    • G06COMPUTING OR CALCULATING; COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G06F3/0601Interfaces specially adapted for storage systems
    • G06F3/0668Interfaces specially adapted for storage systems adopting a particular infrastructure
    • G06F3/0671In-line storage system
    • G06F3/0683Plurality of storage devices
    • G06F3/0689Disk arrays, e.g. RAID, JBOD

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Human Computer Interaction (AREA)
  • Quality & Reliability (AREA)
  • Computer Networks & Wireless Communication (AREA)
  • Hardware Redundancy (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)

Abstract

本申请公开了一种数据处理的方法,包括:从IO数据的下发指令中提取归属信息,得到优先节点;判断优先节点是否处于高负载状态,若优先节点处于高负载状态,则按第一预设方式对IO数据进行转发;判断系统后端是否存在异常,若系统后端存在异常,则按第二预设方式对IO数据进行转发;判断Raid是否存在异常,若Raid存在异常,则按第三预设方式对IO数据进行转发;其中,Raid是由多个物理磁盘构成;判断物理磁盘是否存在异常,若物理磁盘存在异常,则按第四预设方式对IO数据进行转发。该方法分析更细致、能在多个层级进行数据转发,存储系统更高效稳定且保护能力更强。本申请还公开了一种数据处理的系统,具有上述有益效果。

Figure 201710581341

The present application discloses a data processing method, comprising: extracting attribution information from an instruction issued by IO data to obtain a priority node; judging whether the priority node is in a high load state, if the priority node is in a high load state, press the first The IO data is forwarded in the preset mode; it is judged whether there is an abnormality in the back end of the system. If there is an abnormality in the back end of the system, the IO data is forwarded according to the second preset mode; it is judged whether there is an abnormality in the Raid, and if there is an abnormality in the Raid, press The third preset mode forwards the IO data; wherein, the Raid is composed of multiple physical disks; it is determined whether the physical disk is abnormal, and if the physical disk is abnormal, the IO data is forwarded according to the fourth preset mode. This method has more detailed analysis, can forward data at multiple levels, and the storage system is more efficient and stable, and has stronger protection capabilities. The present application also discloses a data processing system, which has the above beneficial effects.

Figure 201710581341

Description

Data processing method and system
Technical Field
The present application relates to the field of data processing technologies, and in particular, to a method and a system for data processing.
Background
With the development of society, the importance of the data is obvious while the data volume is more and more. Therefore, a storage system which is efficient and stable in operation and has redundancy protection features is very important.
In the existing storage product, the problem of possible abnormality is solved only by a plurality of paths which are generally and simply issued and a backup is made, and for the whole huge storage system, the structure is gradually constructed from the bottom layer upwards, from small to large and from bottom to top, and the problem that the increasingly diversified storage system abnormality is solved only by the solution in the prior art.
Therefore, a problem to be solved by those skilled in the art is how to provide a data processing mechanism that has more detailed hierarchy and performs data forwarding at multiple levels, so that the storage system operates more efficiently and more stably, and has stronger data redundancy protection capability under the increasingly diversified abnormal conditions of the storage system.
Disclosure of Invention
The application aims to provide a data processing method and a data processing system, which can perform data forwarding at multiple levels in a more detailed hierarchy under the condition that increasingly diversified exceptions occur in a storage system, so that the storage system is more efficient and stable in operation and has stronger data redundancy protection capability.
In order to solve the above technical problem, the present application provides a data processing method, including:
extracting attribution information from an instruction for issuing IO data to obtain a priority node;
judging whether the priority node is in a high load state, if so, forwarding the IO data according to a first preset mode;
judging whether the back end of the system is abnormal or not, and if the back end of the system is abnormal, forwarding the IO data according to a second preset mode;
judging whether the Raid is abnormal or not, and if the Raid is abnormal, forwarding the IO data according to a third preset mode; wherein the Raid is composed of a plurality of physical disks;
and judging whether the physical disk is abnormal or not, and if the physical disk is abnormal, forwarding the IO data according to a fourth preset mode.
Optionally, determining whether the priority node is in a high load state, and if the priority node is in the high load state, forwarding the IO data according to a first preset manner, including:
acquiring the load state of the priority node to obtain a load state parameter;
judging whether the load state parameter exceeds a high load threshold value;
if the load state parameter exceeds the high load threshold, forwarding the IO data to an opposite end node; and the opposite end node and the priority node form an IO Group.
Optionally, determining whether a system back end is abnormal, if the system back end is abnormal, forwarding the IO data according to a second preset mode, including:
judging whether the system back end corresponding to the preferred node has back end abnormity;
and if the back-end exception exists, forwarding the IO data to a node which is located in the same cluster and does not have the back-end exception.
Optionally, determining whether the Raid is abnormal, and if the Raid is abnormal, forwarding the IO data according to a third preset mode, including:
judging whether the Raid attributed to the system back end is abnormal or not;
and if the Raid abnormity exists, forwarding the IO data to a node in the cluster without the Raid abnormity.
Optionally, determining whether the physical disk is abnormal, if the physical disk is abnormal, forwarding the IO data according to a fourth preset mode, where the method includes:
judging whether the physical disk forming the Raid has disk abnormity or not;
and if the disk exception exists, forwarding the IO data to a node in the cluster without the disk exception.
The present application also provides a system for data processing, the system comprising:
the extraction unit is used for extracting attribution information from an instruction for issuing the IO data to obtain a priority node;
the first judging unit is used for judging whether the priority node is in a high-load state or not, and if the priority node is in the high-load state, forwarding the IO data according to a first preset mode;
the second judging unit is used for judging whether the back end of the system is abnormal or not, and if the back end of the system is abnormal, forwarding the IO data according to a second preset mode;
the third judging unit is used for judging whether the Raid is abnormal or not, and if the Raid is abnormal, forwarding the IO data according to a third preset mode; wherein the Raid is composed of a plurality of physical disks;
and the fourth judging unit is used for judging whether the physical disk is abnormal or not, and if the physical disk is abnormal, forwarding the IO data according to a fourth preset mode.
Optionally, the first determining unit includes:
the acquisition subunit is used for acquiring the load state of the priority node to obtain a load state parameter;
the load judging subunit is used for judging whether the load state parameter exceeds a high load threshold value;
a first forwarding subunit, configured to forward the IO data to an opposite end node if the load state parameter exceeds the high load threshold; and the opposite end node and the priority node form an IO Group.
Optionally, the second judging unit includes:
a back-end abnormality judging subunit, configured to judge whether a back-end abnormality exists at the system back end corresponding to the preferred node;
and the second processing subunit is configured to, if the backend exception exists, forward the IO data to a node in the same cluster where the backend exception does not exist.
Optionally, the third determining unit includes:
a Raid abnormity judging subunit, configured to judge whether a Raid abnormity exists in the Raid attributed to the system back end;
and the third processing subunit is configured to forward the IO data to a node in the cluster where the Raid exception does not exist, if the Raid exception exists.
Optionally, the fourth determining unit includes:
a disk abnormality determining subunit, configured to determine whether a disk abnormality exists in the physical disk constituting the Raid;
and the fourth processing subunit is configured to forward the IO data to a node in the cluster where the disk exception does not exist, if the disk exception exists.
According to the data processing method, the priority node is obtained by extracting the attribution information from the instruction of issuing the IO data; judging whether the priority node is in a high load state, if so, forwarding the IO data according to a first preset mode; judging whether the back end of the system is abnormal or not, and if the back end of the system is abnormal, forwarding the IO data according to a second preset mode; judging whether the Raid is abnormal or not, and if the Raid is abnormal, forwarding the IO data according to a third preset mode; wherein the Raid is composed of a plurality of physical disks; and judging whether the physical disk is abnormal or not, and if the physical disk is abnormal, forwarding the IO data according to a fourth preset mode.
Obviously, according to the technical scheme provided by the application, whether the abnormality occurs in the four layers from top to bottom is judged according to different judgment standards, so that the specific layer is obtained, and corresponding solutions are correspondingly provided according to the problems occurring in different levels. Under the increasingly diversified abnormal condition of the storage system, the data processing method which is more detailed in layering and performs data forwarding at multiple levels can be provided, so that the storage system is more efficient and stable in operation and has stronger data redundancy protection capability. The application also provides a data processing system, which has the beneficial effects and is not repeated herein.
Drawings
In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings needed to be used in the description of the embodiments or the prior art will be briefly introduced below, it is obvious that the drawings in the following description are only embodiments of the present application, and for those skilled in the art, other drawings can be obtained according to the provided drawings without creative efforts.
Fig. 1 is a flowchart of a data processing method according to an embodiment of the present application;
FIG. 2 is a flow chart of another method of data processing provided by an embodiment of the present application;
FIG. 3 is a flow chart of yet another method for data processing provided by an embodiment of the present application;
fig. 4 is a block diagram of a data processing system according to an embodiment of the present application.
Detailed Description
The core of the application is to provide a data processing method and system, which can provide a data processing method with more detailed hierarchy and data forwarding at multiple levels under the condition that increasingly diversified exceptions occur in a storage system, so that the storage system is more efficient and stable in operation and has stronger data redundancy protection capability.
In order to make the objects, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application, and it is obvious that the described embodiments are some embodiments of the present application, but not all embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present application.
Referring to fig. 1, fig. 1 is a flowchart of a data processing method according to an embodiment of the present disclosure.
The method specifically comprises the following steps:
s101: extracting attribution information from an instruction for issuing IO data to obtain a priority node;
the method comprises the steps of extracting attribution information carried in an IO data issuing instruction to obtain a priority node. The embodiment of the application is established in an IO cluster, and the following is better explained through a practical example:
an IO cluster is composed of 4 IO groups, an IO Group is composed of two APs (nodes), each node is provided with a controller for controlling sending and forwarding, that is, when the top end sends an IO data, the IO data is sent according to a certain sending rule, if the data needs to be sent from a third IO Group in the IO cluster according to the sending rule, the 1# node is usually defaulted as an advantageous node, that is, when the data needs to be sent from the third IO Group, the data is sent from the 1# node by default, and only when a special condition occurs, the data is sent from an opposite end node of the 1# node, namely the 2# node. And the nodes except the corresponding nodes of the same IO group cannot be forwarded to the nodes for issuing.
Of course, the manner how to obtain the attribution information from the issued instruction or in which IO group the attribution information should be issued is various, and the attribution information may be determined according to the initial number of the IO group or the initial number of each node from the issued instruction, and the attribution information may be determined according to the retrieval result, so as to determine the advantageous node.
S102: judging whether the priority node is in a high load state or not;
on the basis of S101, this step is intended to determine whether the priority node that should issue the IO data is in a special state, i.e., a high load state. In the high load state, the node has previously received an instruction to issue additional IO data, and this issuing step is still performed until now, and the issuing of this IO data cannot be completed in a short time. In other words, the priority node is already used, and the issued IO data is large or large in quantity, which causes the load of the node to be too high, and is not suitable for issuing new IO data again in a short time.
S103: forwarding the IO data according to a first preset mode;
in this step, when the determination result in S102 indicates that the priority node is in the high load state, the IO data is forwarded in a first preset manner.
S104: judging whether the back end of the system is abnormal or not;
s105: forwarding the IO data according to a second preset mode;
s104 and S105 firstly judge whether the system back end corresponding to the issuing node is abnormal or not, and forward the IO data according to a second preset mode under the condition that the abnormality occurs.
The system back end may be back end software or a logical volume corresponding to the issuing node, in other words, the logical volume may be equivalent to one partition that is divided from the entire hard disk storage space on our PC, that is, a C disk, a D disk, and an E disk, and the back end software controls how to reach the corresponding logical volume from the issuing node.
If it is detected that the back-end software or the logical volume has a fault and an abnormality occurs, which inevitably results in that subsequent write-in operation cannot be continued, the IO data needs to be forwarded according to the second preset mode mentioned in this step, so that the IO data is successfully issued.
S106: judging whether the Raid is abnormal or not;
s107: forwarding the IO data according to a third preset mode;
s106 and S107 firstly judge whether the Raid at the back end of the system is abnormal or not, and forward the IO data according to a third preset mode when the Raid is abnormal.
The Raid (Redundant Arrays of Independent Disks, chinese name: disk array) has the meaning of "array with redundancy capability composed of Independent Disks", the disk array is a disk group with huge capacity composed of many inexpensive Disks, and the performance of the entire disk system is enhanced by the addition effect generated by providing data by individual Disks. With this technique, data is divided into a plurality of sectors, each of which is stored on a respective hard disk.
In other words, the Raid can be understood as a disk array formed by a plurality of physical disks, just as if we installed two or more hard disks in their own PC, but in the non-partitioned area, we only see a large storage space formed by the respective storage spaces of the two hard disks in the PC. However, due to the large scale in an IO cluster, there may be multiple raids for data to be issued and stored, that is, there may be a logical volume separated from a certain Raid at the system back end of the issuing node, and if the raids bearing all the logical volumes on the issuing node are abnormal, it means that all the logical volumes on the issuing node cannot be used normally, at this time, data needs to be forwarded according to the third preset mode provided by the present application, so that the IO data is issued normally.
S108: judging whether the physical disk is abnormal or not;
s109: and forwarding the IO data according to a fourth preset mode.
S108 and S109 first determine whether the plurality of physical disks forming the Raid are abnormal, and forward the IO data in a fourth preset manner when the abnormality occurs.
As mentioned above, a Raid is composed of a plurality of physical disks, that is, even if a logical volume is divided from the Raid, the logical volume may occupy two physically divided different disks at the same time, and in this case, if an abnormality occurs in a certain physical disk, it is necessary to perform corresponding forwarding, so that IO data is normally issued and finally reaches the physical disks.
For example, each level creates a respective abnormal log recording function, and the abnormal or fault occurring in different levels is recorded in different abnormal logs respectively, and then only the log file needs to be combined to determine whether an abnormal occurs. Furthermore, in combination with the above example that the IO cluster is composed of a plurality of IO groups, the exception logs can be generated according to different IO groups, so that it can be known more clearly that the system backend, Raid, and physical disk corresponding to which node of which IO group is abnormal, so as to perform corresponding forwarding work.
Based on the above technical solution, the method for processing data provided in the embodiment of the present application obtains which layer has a problem by determining whether the four layers from top to bottom are abnormal according to different determination criteria, and correspondingly provides a corresponding solution according to the problems at different levels. Under the increasingly diversified abnormal condition of the storage system, the data processing method which is more detailed in layering and performs data forwarding at multiple levels can be provided, so that the storage system is more efficient and stable in operation and has stronger data redundancy protection capability.
Referring to fig. 2, fig. 2 is a flowchart of another data processing method provided in the embodiment of the present application.
The present embodiment is a specific limitation on how the first preset manner in S103 is forwarded in the previous embodiment, other steps are substantially the same as those in the previous embodiment, and the same parts may refer to relevant parts of the previous embodiment, and are not described herein again.
The method specifically comprises the following steps:
s201: acquiring the load state of the priority node to obtain a load state parameter;
s202: whether the load status parameter exceeds a high load threshold;
s203: and forwarding the IO data to the corresponding node.
In this step, when the priority node is judged to be in a high load state, the IO data to be sent down at the priority node is forwarded to the opposite node of the opposite end of the priority node in the same IO group for sending, that is, when the priority node 1# in the same IO group is in a high load state, the IO data is forwarded to the priority node 2# for sending.
It should be noted that the IO data cannot be forwarded to the nodes in other IO groups for delivery, because the requirements made inside the whole cluster in order to enable the corresponding logical volume to be delivered to can be made on the whole large system setting.
Referring to fig. 3, fig. 3 is a flowchart of another data processing method according to an embodiment of the present disclosure.
The present embodiment is a limitation on how to forward the second preset manner in the embodiment S105, other steps are substantially the same as those in other embodiments, and the same parts may refer to related parts in other embodiments, which are not described herein again.
The method specifically comprises the following steps:
s301: acquiring the load state of the priority node to obtain a load state parameter;
s302: whether the system back end corresponding to the prior node has back end abnormity or not is judged;
s303: and forwarding the IO data to a node which is located in the same cluster and has no backend exception.
The method comprises the following steps of forwarding IO data to a node which does not have backend abnormality in the same cluster when the backend abnormality exists in a system backend corresponding to the preferred node.
In combination with the contents described in the above other embodiments, when the system backend corresponding to the priority node is abnormal or the backend software or the logical volume has a fault in a specific environment, the node without the backend abnormality is searched in the IO cluster to forward, so as to successfully implement the issuing of the IO data.
Of course, if there is no backend exception in the peer node of the IO group where the priority node is located, the priority node may be preferentially forwarded to the peer node for delivery, because the priority node is located in the same IO group after all, the transmission distance is also relatively short, which is more convenient. If the system back end corresponding to the end node and the system back end corresponding to the priority node are overlapped because of being in the same IO group, the system back end cannot be forwarded to the opposite end node for issuing, and the system back end needs to be forwarded to the nodes in other IO groups for issuing.
Further, even if the data is forwarded to other IO groups for distribution, the default priority node may be taken first. The present invention is not limited to this, and various factors such as the device model, system configuration, and management habit selected by the IO cluster in actual situations should be considered and selected accordingly.
Accordingly, the third preset manner corresponding to S107 and the fourth preset manner corresponding to S109 can be selected for forwarding correspondence substantially according to the idea of the present embodiment, and of course, in the case of different hierarchies, many other problems may also exist, and it is reasonable to modify the forwarding mechanism of each layer within the allowable range. And is not particularly limited herein.
Based on the above technical solution, in the data processing method provided in the embodiment of the present application, whether an abnormality occurs in any of the four layers from top to bottom is determined according to different determination criteria, so as to obtain which layer has a problem, and when a failure causes that IO data cannot be continuously issued due to an abnormality of each layer occurring in different layers, a corresponding forwarding manner is selected according to the difference of the layers, so that the IO data can be normally issued. Under the increasingly diversified abnormal condition of the storage system, the data processing method which is more detailed in layering and performs data forwarding at multiple levels can be provided, so that the storage system is more efficient and stable in operation and has stronger data redundancy protection capability.
Because the situation is complicated and cannot be illustrated by a list, those skilled in the art can realize that many examples exist based on the principle of the basic method provided by the present application in combination with the actual situation, and the method is within the scope of the present application without sufficient inventive effort.
Referring to fig. 4, fig. 4 is a block diagram of a data processing system according to an embodiment of the present disclosure.
The system may include:
an extracting unit 100, configured to extract attribution information from an instruction for issuing IO data to obtain a priority node;
a first determining unit 200, configured to determine whether the priority node is in a high load state, and if the priority node is in the high load state, forward the IO data according to a first preset manner;
a second determining unit 300, configured to determine whether the back end of the system is abnormal, and if the back end of the system is abnormal, forward the IO data according to a second preset manner;
a third determining unit 400, configured to determine whether the Raid is abnormal, and if the Raid is abnormal, forward the IO data according to a third preset manner; wherein, Raid is composed of a plurality of physical disks;
a fourth determining unit 500, configured to determine whether the physical disk is abnormal, and if the physical disk is abnormal, forward the IO data according to a fourth preset manner.
The IO groups can be interconnected, and the objects of interconnection include: control flow and IO data flow, but only nodes within one IO group in the cluster have access to the backend disk. This is why the first predetermined manner cannot be forwarded to other IO groups.
Optionally, the first determining unit includes:
the acquisition subunit is used for acquiring the load state of the priority node to obtain a load state parameter;
the load judging subunit is used for judging whether the load state parameter exceeds a high load threshold value;
the first forwarding subunit is configured to forward the IO data to the peer node if the load state parameter exceeds the high load threshold; wherein, the opposite end node and the priority node form an IO Group.
This layer forwarding, also called upper layer forwarding (UFW), can be considered as one of multipath IO forwarding: a multi-path strategy is configured between the front-end host and the storage, and IO of the host is issued to a priority node in the cluster preferentially; if the IO load of the priority node is too high, the IO can be issued from other nodes, if the IO is not issued to the priority node to which the volume belongs, the IO is issued to another node in an IO group where the priority node belongs, namely the opposite node, the IO is issued from the opposite node; if the IO is issued to other IO groups in the cluster, the IO is forwarded to the priority node through the UFW module for issuing, and the process can ensure that the volume can be accessed in the whole cluster.
Therefore, all nodes in the cluster can be accessed, which means that when a logical volume is created in the IO group, the logical volume needs to be mapped to the front-end host, and if the mapping relationship occurs in the IO group, the logical volume can directly receive the IO of the front-end host; as the service is expanded, a cluster is created at this time, a volume in the IO group needs to be mapped to a front-end host through other nodes inside the cluster, the IO data sent by the front-end host is sent to another IO group in the cluster, the IO group receiving the IO data at this time, that is, a certain controller node in the IO group needs to forward the IO data to a priority node to which the volume in the IO group belongs, and then the IO data is issued from the IO group, where the reason is that a back-end disk can only access in one IO group; if the front-end host is configured with the multipath technology, the volume can receive IO data from different IO groups, and the front-end IO load is balanced.
Optionally, the second judging unit includes:
a back-end abnormality judging subunit, configured to judge whether a back-end abnormality exists at a system back end corresponding to the preferred node;
and the second processing subunit is configured to, if a backend exception exists, forward the IO data to a node in the same cluster where the backend exception does not exist.
This layer of forwarding is also called duplicate forwarding or mirror forwarding (CFW): copy forwarding is a redundant protection technology, and when a mirror module finds that a downstream module, such as a lower layer logical volume or backend software, has a fault or Bug, IO is transferred to a peer node or other IO groups to issue IO data.
The copy forwarding is different from a caching mirroring technology, the copy forwarding does not occur every time, the copy forwarding is enabled only when a module downstream of the IO stack fails or when copy generation is started, data in the copy is stored in a disk instead of a cache, and the copy forwarding is a protection measure in the IO stack.
Optionally, the third determining unit includes:
a Raid abnormity judging subunit, configured to judge whether a Raid abnormity exists in a Raid attributed to the system back end;
and the third processing subunit is configured to forward the IO data to a node in the cluster where the Raid exception does not exist, if the Raid exception exists.
This layer of forwarding is also called managed disk forwarding (MFW): MFW forwarding refers to IO forwarding in a storage pool inside a cluster, and one scenario is that when an MFW module detects that a downstream module Raid fails, IO is transferred to a peer controller and is issued from the peer controller; in another scenario, the managed disk is from another IO group, and the IO needs to be transferred to the IO group where the managed disk is located through the MFW module and then issued.
In other words, MFW forwarding means that when a storage pool is created in the whole cluster, data delivered from the present IO group needs to be forwarded to mdisks in other IO groups in the present cluster for delivery; or when the RAID detects that the module and the downstream module are in fault, the MFW forwards the IO data to the other nodes of the IO group and then sends the data down.
Optionally, the fourth determining unit includes:
the disk abnormity judging subunit is used for judging whether the physical disk forming the Raid has disk abnormity;
and the fourth processing subunit is configured to forward the IO data to a node in the cluster where the disk exception does not exist if the disk exception exists.
This layer of forwarding is also called Disk Forwarding (DFW): the method refers to that when a controller detects that a link of the controller accessing a disk fails or a RAID created by using disks of other IO groups is used, IO data are forwarded to the disk. The DFW forwarding means that the disks forming the Raid are from the same IO group, and at this time, the DFW module is required to forward the IO to the IO group where the disks are located.
The embodiments are described in a progressive manner in the specification, each embodiment focuses on differences from other embodiments, and the same and similar parts among the embodiments are referred to each other. The device disclosed by the embodiment corresponds to the method disclosed by the embodiment, so that the description is simple, and the relevant points can be referred to the method part for description.
Those of skill would further appreciate that the various illustrative elements and algorithm steps described in connection with the embodiments disclosed herein may be implemented as electronic hardware, computer software, or combinations of both, and that the various illustrative components and steps have been described above generally in terms of their functionality in order to clearly illustrate this interchangeability of hardware and software. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the implementation. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present application.
The principles and embodiments of the present application are explained herein using specific examples, which are provided only to help understand the method and the core idea of the present application. It should be noted that, for those skilled in the art, it is possible to make several improvements and modifications to the present application without departing from the principle of the present application, and such improvements and modifications also fall within the scope of the claims of the present application.
It is further noted that, in the present specification, relational terms such as first and second, and the like are used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Also, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other identical elements in a process, method, article, or apparatus that comprises the element.

Claims (10)

1. A method of data processing, comprising:
extracting attribution information from an instruction for issuing IO data to obtain a priority node;
judging whether the priority node is in a high load state, if so, forwarding the IO data according to a first preset mode;
judging whether the system back end of the priority node is abnormal or not, and if the system back end is abnormal, forwarding the IO data according to a second preset mode;
judging whether the Raid at the rear end of the system is abnormal or not, and if the Raid is abnormal, forwarding the IO data according to a third preset mode; wherein the Raid is composed of a plurality of physical disks;
and judging whether the physical disk is abnormal or not, and if the physical disk is abnormal, forwarding the IO data according to a fourth preset mode.
2. The method according to claim 1, wherein determining whether the priority node is in a high load state, and if the priority node is in the high load state, forwarding the IO data according to a first preset manner includes:
acquiring the load state of the priority node to obtain a load state parameter;
judging whether the load state parameter exceeds a high load threshold value;
if the load state parameter exceeds the high load threshold, forwarding the IO data to an opposite end node; and the opposite end node and the priority node form an IO Group.
3. The method according to claim 2, wherein determining whether a system back end of the priority node is abnormal, and if the system back end is abnormal, forwarding the IO data according to a second preset mode includes:
judging whether the system back end corresponding to the preferred node has back end abnormity;
and if the back-end exception exists, forwarding the IO data to a node which is located in the same cluster and does not have the back-end exception.
4. The method according to claim 3, wherein the step of judging whether the Raid at the back end of the system is abnormal or not, and if the Raid is abnormal, forwarding the IO data in a third preset manner includes:
judging whether the Raid attributed to the system back end is abnormal or not;
and if the Raid abnormity exists, forwarding the IO data to a node in the cluster without the Raid abnormity.
5. The method according to claim 4, wherein determining whether the physical disk is abnormal, and if the physical disk is abnormal, forwarding the IO data according to a fourth preset manner includes:
judging whether the physical disk forming the Raid has disk abnormity or not;
and if the disk exception exists, forwarding the IO data to a node in the cluster without the disk exception.
6. A system for data processing, comprising:
the extraction unit is used for extracting attribution information from an instruction for issuing the IO data to obtain a priority node;
the first judging unit is used for judging whether the priority node is in a high-load state or not, and if the priority node is in the high-load state, forwarding the IO data according to a first preset mode;
the second judging unit is used for judging whether the system back end of the priority node is abnormal or not, and if the system back end is abnormal, forwarding the IO data according to a second preset mode;
a third judging unit, configured to judge whether a Raid at the rear end of the system is abnormal, and if the Raid is abnormal, forward the IO data in a third preset manner; wherein the Raid is composed of a plurality of physical disks;
and the fourth judging unit is used for judging whether the physical disk is abnormal or not, and if the physical disk is abnormal, forwarding the IO data according to a fourth preset mode.
7. The system according to claim 6, wherein the first judging unit includes:
the acquisition subunit is used for acquiring the load state of the priority node to obtain a load state parameter;
the load judging subunit is used for judging whether the load state parameter exceeds a high load threshold value;
a first forwarding subunit, configured to forward the IO data to an opposite end node if the load state parameter exceeds the high load threshold; and the opposite end node and the priority node form an IO Group.
8. The system according to claim 7, wherein the second determination unit comprises:
a back-end abnormality judging subunit, configured to judge whether a back-end abnormality exists at the system back end corresponding to the preferred node;
and the second processing subunit is configured to, if the backend exception exists, forward the IO data to a node in the same cluster where the backend exception does not exist.
9. The system according to claim 8, wherein the third judging unit includes:
a Raid abnormity judging subunit, configured to judge whether a Raid abnormity exists in the Raid attributed to the system back end;
and the third processing subunit is configured to forward the IO data to a node in the cluster where the Raid exception does not exist, if the Raid exception exists.
10. The system according to claim 9, wherein the third judging unit includes:
a disk abnormality determining subunit, configured to determine whether a disk abnormality exists in the physical disk constituting the Raid;
and the fourth processing subunit is configured to forward the IO data to a node in the cluster where the disk exception does not exist, if the disk exception exists.
CN201710581341.6A 2017-07-17 2017-07-17 Method and system for data processing Active CN107247564B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN201710581341.6A CN107247564B (en) 2017-07-17 2017-07-17 Method and system for data processing

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN201710581341.6A CN107247564B (en) 2017-07-17 2017-07-17 Method and system for data processing

Publications (2)

Publication Number Publication Date
CN107247564A CN107247564A (en) 2017-10-13
CN107247564B true CN107247564B (en) 2021-02-02

Family

ID=60014666

Family Applications (1)

Application Number Title Priority Date Filing Date
CN201710581341.6A Active CN107247564B (en) 2017-07-17 2017-07-17 Method and system for data processing

Country Status (1)

Country Link
CN (1) CN107247564B (en)

Families Citing this family (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113849129B (en) * 2021-09-18 2023-08-25 苏州浪潮智能科技有限公司 IO request forwarding method, device and equipment between storage controllers
CN116668335A (en) * 2023-05-19 2023-08-29 超聚变数字技术有限公司 A cluster service processing method, server and system

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN101887386A (en) * 2010-06-02 2010-11-17 深圳市迪菲特科技股份有限公司 Method and system for processing failure of redundant array of independent disk controller
CN104461752A (en) * 2014-11-21 2015-03-25 浙江宇视科技有限公司 Two-level fault-tolerant multimedia distributed task processing method
CN105187517A (en) * 2015-08-20 2015-12-23 浪潮(北京)电子信息产业有限公司 Metadata clustering method and metadata cluster
CN105897933A (en) * 2016-06-15 2016-08-24 乐视控股(北京)有限公司 Service request processing method and device
CN106874138A (en) * 2015-12-11 2017-06-20 成都华为技术有限公司 The distribution method and device of a kind of service node

Family Cites Families (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
KR101502896B1 (en) * 2011-02-14 2015-03-24 주식회사 케이티 Distributed memory cluster control apparatus and method using map reduce
JP6007522B2 (en) * 2012-03-09 2016-10-12 日本電気株式会社 Cluster system
CN105515812A (en) * 2014-10-15 2016-04-20 中兴通讯股份有限公司 Fault processing method of resources and device

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN101887386A (en) * 2010-06-02 2010-11-17 深圳市迪菲特科技股份有限公司 Method and system for processing failure of redundant array of independent disk controller
CN104461752A (en) * 2014-11-21 2015-03-25 浙江宇视科技有限公司 Two-level fault-tolerant multimedia distributed task processing method
CN105187517A (en) * 2015-08-20 2015-12-23 浪潮(北京)电子信息产业有限公司 Metadata clustering method and metadata cluster
CN106874138A (en) * 2015-12-11 2017-06-20 成都华为技术有限公司 The distribution method and device of a kind of service node
CN105897933A (en) * 2016-06-15 2016-08-24 乐视控股(北京)有限公司 Service request processing method and device

Also Published As

Publication number Publication date
CN107247564A (en) 2017-10-13

Similar Documents

Publication Publication Date Title
US11740826B2 (en) Policy-based hierarchical data protection in distributed storage
CN103152395B (en) A kind of storage means of distributed file system and device
CN111857592B (en) Data storage method and device based on object storage system and electronic equipment
US10642704B2 (en) Storage controller failover system
CN103763383B (en) Integrated cloud storage system and its storage method
US7657796B1 (en) System and method for distributed storage verification
EP4139802B1 (en) Methods for managing input-ouput operations in zone translation layer architecture and devices thereof
US7689764B1 (en) Network routing of data based on content thereof
US10852966B1 (en) System and method for creating mapped RAID group during expansion of extent pool
US20140075111A1 (en) Block Level Management with Service Level Agreement
CN107463342B (en) CDN edge node file storage method and device
CN103649901A (en) Data transmission method, data receiving method and sotring equipment
TW200901025A (en) Method and system for a self managing and scalable grid storage
CN1823324A (en) System and method of relational configuration mirroring
US20120159100A1 (en) Storage device migration and redirection
EP3745269B1 (en) Hierarchical fault tolerance in system storage
US9348714B2 (en) Survival site load balancing
US11442827B2 (en) Policy-based hierarchical data protection in distributed storage
CN102999587A (en) Arrangement for mirror database across different servers used for failover
CN111488127B (en) Data parallel storage method, device and data reading method based on disk cluster
WO2014043456A1 (en) Storage block metadata tagger
US12423015B2 (en) Methods for handling storage devices with different zone sizes and devices thereof
CN110839023B (en) A multi-channel customer service system for power marketing
CN107247564B (en) Method and system for data processing
CN113805811A (en) Method, system, equipment and storage medium for optimizing read-write access file

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
TA01 Transfer of patent application right
TA01 Transfer of patent application right

Effective date of registration: 20210106

Address after: Building 9, No.1, guanpu Road, Guoxiang street, Wuzhong Economic Development Zone, Wuzhong District, Suzhou City, Jiangsu Province

Applicant after: SUZHOU LANGCHAO INTELLIGENT TECHNOLOGY Co.,Ltd.

Address before: Room 1601, floor 16, 278 Xinyi Road, Zhengdong New District, Zhengzhou City, Henan Province

Applicant before: ZHENGZHOU YUNHAI INFORMATION TECHNOLOGY Co.,Ltd.

GR01 Patent grant
GR01 Patent grant
CP03 Change of name, title or address

Address after: Building 9, No.1, guanpu Road, Guoxiang street, Wuzhong Economic Development Zone, Wuzhong District, Suzhou City, Jiangsu Province

Patentee after: Suzhou Yuannao Intelligent Technology Co.,Ltd.

Country or region after: China

Address before: Building 9, No.1, guanpu Road, Guoxiang street, Wuzhong Economic Development Zone, Wuzhong District, Suzhou City, Jiangsu Province

Patentee before: SUZHOU LANGCHAO INTELLIGENT TECHNOLOGY Co.,Ltd.

Country or region before: China