Disclosure of Invention
The application aims to provide a data processing method and a data processing system, which can perform data forwarding at multiple levels in a more detailed hierarchy under the condition that increasingly diversified exceptions occur in a storage system, so that the storage system is more efficient and stable in operation and has stronger data redundancy protection capability.
In order to solve the above technical problem, the present application provides a data processing method, including:
extracting attribution information from an instruction for issuing IO data to obtain a priority node;
judging whether the priority node is in a high load state, if so, forwarding the IO data according to a first preset mode;
judging whether the back end of the system is abnormal or not, and if the back end of the system is abnormal, forwarding the IO data according to a second preset mode;
judging whether the Raid is abnormal or not, and if the Raid is abnormal, forwarding the IO data according to a third preset mode; wherein the Raid is composed of a plurality of physical disks;
and judging whether the physical disk is abnormal or not, and if the physical disk is abnormal, forwarding the IO data according to a fourth preset mode.
Optionally, determining whether the priority node is in a high load state, and if the priority node is in the high load state, forwarding the IO data according to a first preset manner, including:
acquiring the load state of the priority node to obtain a load state parameter;
judging whether the load state parameter exceeds a high load threshold value;
if the load state parameter exceeds the high load threshold, forwarding the IO data to an opposite end node; and the opposite end node and the priority node form an IO Group.
Optionally, determining whether a system back end is abnormal, if the system back end is abnormal, forwarding the IO data according to a second preset mode, including:
judging whether the system back end corresponding to the preferred node has back end abnormity;
and if the back-end exception exists, forwarding the IO data to a node which is located in the same cluster and does not have the back-end exception.
Optionally, determining whether the Raid is abnormal, and if the Raid is abnormal, forwarding the IO data according to a third preset mode, including:
judging whether the Raid attributed to the system back end is abnormal or not;
and if the Raid abnormity exists, forwarding the IO data to a node in the cluster without the Raid abnormity.
Optionally, determining whether the physical disk is abnormal, if the physical disk is abnormal, forwarding the IO data according to a fourth preset mode, where the method includes:
judging whether the physical disk forming the Raid has disk abnormity or not;
and if the disk exception exists, forwarding the IO data to a node in the cluster without the disk exception.
The present application also provides a system for data processing, the system comprising:
the extraction unit is used for extracting attribution information from an instruction for issuing the IO data to obtain a priority node;
the first judging unit is used for judging whether the priority node is in a high-load state or not, and if the priority node is in the high-load state, forwarding the IO data according to a first preset mode;
the second judging unit is used for judging whether the back end of the system is abnormal or not, and if the back end of the system is abnormal, forwarding the IO data according to a second preset mode;
the third judging unit is used for judging whether the Raid is abnormal or not, and if the Raid is abnormal, forwarding the IO data according to a third preset mode; wherein the Raid is composed of a plurality of physical disks;
and the fourth judging unit is used for judging whether the physical disk is abnormal or not, and if the physical disk is abnormal, forwarding the IO data according to a fourth preset mode.
Optionally, the first determining unit includes:
the acquisition subunit is used for acquiring the load state of the priority node to obtain a load state parameter;
the load judging subunit is used for judging whether the load state parameter exceeds a high load threshold value;
a first forwarding subunit, configured to forward the IO data to an opposite end node if the load state parameter exceeds the high load threshold; and the opposite end node and the priority node form an IO Group.
Optionally, the second judging unit includes:
a back-end abnormality judging subunit, configured to judge whether a back-end abnormality exists at the system back end corresponding to the preferred node;
and the second processing subunit is configured to, if the backend exception exists, forward the IO data to a node in the same cluster where the backend exception does not exist.
Optionally, the third determining unit includes:
a Raid abnormity judging subunit, configured to judge whether a Raid abnormity exists in the Raid attributed to the system back end;
and the third processing subunit is configured to forward the IO data to a node in the cluster where the Raid exception does not exist, if the Raid exception exists.
Optionally, the fourth determining unit includes:
a disk abnormality determining subunit, configured to determine whether a disk abnormality exists in the physical disk constituting the Raid;
and the fourth processing subunit is configured to forward the IO data to a node in the cluster where the disk exception does not exist, if the disk exception exists.
According to the data processing method, the priority node is obtained by extracting the attribution information from the instruction of issuing the IO data; judging whether the priority node is in a high load state, if so, forwarding the IO data according to a first preset mode; judging whether the back end of the system is abnormal or not, and if the back end of the system is abnormal, forwarding the IO data according to a second preset mode; judging whether the Raid is abnormal or not, and if the Raid is abnormal, forwarding the IO data according to a third preset mode; wherein the Raid is composed of a plurality of physical disks; and judging whether the physical disk is abnormal or not, and if the physical disk is abnormal, forwarding the IO data according to a fourth preset mode.
Obviously, according to the technical scheme provided by the application, whether the abnormality occurs in the four layers from top to bottom is judged according to different judgment standards, so that the specific layer is obtained, and corresponding solutions are correspondingly provided according to the problems occurring in different levels. Under the increasingly diversified abnormal condition of the storage system, the data processing method which is more detailed in layering and performs data forwarding at multiple levels can be provided, so that the storage system is more efficient and stable in operation and has stronger data redundancy protection capability. The application also provides a data processing system, which has the beneficial effects and is not repeated herein.
Detailed Description
The core of the application is to provide a data processing method and system, which can provide a data processing method with more detailed hierarchy and data forwarding at multiple levels under the condition that increasingly diversified exceptions occur in a storage system, so that the storage system is more efficient and stable in operation and has stronger data redundancy protection capability.
In order to make the objects, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application, and it is obvious that the described embodiments are some embodiments of the present application, but not all embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present application.
Referring to fig. 1, fig. 1 is a flowchart of a data processing method according to an embodiment of the present disclosure.
The method specifically comprises the following steps:
s101: extracting attribution information from an instruction for issuing IO data to obtain a priority node;
the method comprises the steps of extracting attribution information carried in an IO data issuing instruction to obtain a priority node. The embodiment of the application is established in an IO cluster, and the following is better explained through a practical example:
an IO cluster is composed of 4 IO groups, an IO Group is composed of two APs (nodes), each node is provided with a controller for controlling sending and forwarding, that is, when the top end sends an IO data, the IO data is sent according to a certain sending rule, if the data needs to be sent from a third IO Group in the IO cluster according to the sending rule, the 1# node is usually defaulted as an advantageous node, that is, when the data needs to be sent from the third IO Group, the data is sent from the 1# node by default, and only when a special condition occurs, the data is sent from an opposite end node of the 1# node, namely the 2# node. And the nodes except the corresponding nodes of the same IO group cannot be forwarded to the nodes for issuing.
Of course, the manner how to obtain the attribution information from the issued instruction or in which IO group the attribution information should be issued is various, and the attribution information may be determined according to the initial number of the IO group or the initial number of each node from the issued instruction, and the attribution information may be determined according to the retrieval result, so as to determine the advantageous node.
S102: judging whether the priority node is in a high load state or not;
on the basis of S101, this step is intended to determine whether the priority node that should issue the IO data is in a special state, i.e., a high load state. In the high load state, the node has previously received an instruction to issue additional IO data, and this issuing step is still performed until now, and the issuing of this IO data cannot be completed in a short time. In other words, the priority node is already used, and the issued IO data is large or large in quantity, which causes the load of the node to be too high, and is not suitable for issuing new IO data again in a short time.
S103: forwarding the IO data according to a first preset mode;
in this step, when the determination result in S102 indicates that the priority node is in the high load state, the IO data is forwarded in a first preset manner.
S104: judging whether the back end of the system is abnormal or not;
s105: forwarding the IO data according to a second preset mode;
s104 and S105 firstly judge whether the system back end corresponding to the issuing node is abnormal or not, and forward the IO data according to a second preset mode under the condition that the abnormality occurs.
The system back end may be back end software or a logical volume corresponding to the issuing node, in other words, the logical volume may be equivalent to one partition that is divided from the entire hard disk storage space on our PC, that is, a C disk, a D disk, and an E disk, and the back end software controls how to reach the corresponding logical volume from the issuing node.
If it is detected that the back-end software or the logical volume has a fault and an abnormality occurs, which inevitably results in that subsequent write-in operation cannot be continued, the IO data needs to be forwarded according to the second preset mode mentioned in this step, so that the IO data is successfully issued.
S106: judging whether the Raid is abnormal or not;
s107: forwarding the IO data according to a third preset mode;
s106 and S107 firstly judge whether the Raid at the back end of the system is abnormal or not, and forward the IO data according to a third preset mode when the Raid is abnormal.
The Raid (Redundant Arrays of Independent Disks, chinese name: disk array) has the meaning of "array with redundancy capability composed of Independent Disks", the disk array is a disk group with huge capacity composed of many inexpensive Disks, and the performance of the entire disk system is enhanced by the addition effect generated by providing data by individual Disks. With this technique, data is divided into a plurality of sectors, each of which is stored on a respective hard disk.
In other words, the Raid can be understood as a disk array formed by a plurality of physical disks, just as if we installed two or more hard disks in their own PC, but in the non-partitioned area, we only see a large storage space formed by the respective storage spaces of the two hard disks in the PC. However, due to the large scale in an IO cluster, there may be multiple raids for data to be issued and stored, that is, there may be a logical volume separated from a certain Raid at the system back end of the issuing node, and if the raids bearing all the logical volumes on the issuing node are abnormal, it means that all the logical volumes on the issuing node cannot be used normally, at this time, data needs to be forwarded according to the third preset mode provided by the present application, so that the IO data is issued normally.
S108: judging whether the physical disk is abnormal or not;
s109: and forwarding the IO data according to a fourth preset mode.
S108 and S109 first determine whether the plurality of physical disks forming the Raid are abnormal, and forward the IO data in a fourth preset manner when the abnormality occurs.
As mentioned above, a Raid is composed of a plurality of physical disks, that is, even if a logical volume is divided from the Raid, the logical volume may occupy two physically divided different disks at the same time, and in this case, if an abnormality occurs in a certain physical disk, it is necessary to perform corresponding forwarding, so that IO data is normally issued and finally reaches the physical disks.
For example, each level creates a respective abnormal log recording function, and the abnormal or fault occurring in different levels is recorded in different abnormal logs respectively, and then only the log file needs to be combined to determine whether an abnormal occurs. Furthermore, in combination with the above example that the IO cluster is composed of a plurality of IO groups, the exception logs can be generated according to different IO groups, so that it can be known more clearly that the system backend, Raid, and physical disk corresponding to which node of which IO group is abnormal, so as to perform corresponding forwarding work.
Based on the above technical solution, the method for processing data provided in the embodiment of the present application obtains which layer has a problem by determining whether the four layers from top to bottom are abnormal according to different determination criteria, and correspondingly provides a corresponding solution according to the problems at different levels. Under the increasingly diversified abnormal condition of the storage system, the data processing method which is more detailed in layering and performs data forwarding at multiple levels can be provided, so that the storage system is more efficient and stable in operation and has stronger data redundancy protection capability.
Referring to fig. 2, fig. 2 is a flowchart of another data processing method provided in the embodiment of the present application.
The present embodiment is a specific limitation on how the first preset manner in S103 is forwarded in the previous embodiment, other steps are substantially the same as those in the previous embodiment, and the same parts may refer to relevant parts of the previous embodiment, and are not described herein again.
The method specifically comprises the following steps:
s201: acquiring the load state of the priority node to obtain a load state parameter;
s202: whether the load status parameter exceeds a high load threshold;
s203: and forwarding the IO data to the corresponding node.
In this step, when the priority node is judged to be in a high load state, the IO data to be sent down at the priority node is forwarded to the opposite node of the opposite end of the priority node in the same IO group for sending, that is, when the priority node 1# in the same IO group is in a high load state, the IO data is forwarded to the priority node 2# for sending.
It should be noted that the IO data cannot be forwarded to the nodes in other IO groups for delivery, because the requirements made inside the whole cluster in order to enable the corresponding logical volume to be delivered to can be made on the whole large system setting.
Referring to fig. 3, fig. 3 is a flowchart of another data processing method according to an embodiment of the present disclosure.
The present embodiment is a limitation on how to forward the second preset manner in the embodiment S105, other steps are substantially the same as those in other embodiments, and the same parts may refer to related parts in other embodiments, which are not described herein again.
The method specifically comprises the following steps:
s301: acquiring the load state of the priority node to obtain a load state parameter;
s302: whether the system back end corresponding to the prior node has back end abnormity or not is judged;
s303: and forwarding the IO data to a node which is located in the same cluster and has no backend exception.
The method comprises the following steps of forwarding IO data to a node which does not have backend abnormality in the same cluster when the backend abnormality exists in a system backend corresponding to the preferred node.
In combination with the contents described in the above other embodiments, when the system backend corresponding to the priority node is abnormal or the backend software or the logical volume has a fault in a specific environment, the node without the backend abnormality is searched in the IO cluster to forward, so as to successfully implement the issuing of the IO data.
Of course, if there is no backend exception in the peer node of the IO group where the priority node is located, the priority node may be preferentially forwarded to the peer node for delivery, because the priority node is located in the same IO group after all, the transmission distance is also relatively short, which is more convenient. If the system back end corresponding to the end node and the system back end corresponding to the priority node are overlapped because of being in the same IO group, the system back end cannot be forwarded to the opposite end node for issuing, and the system back end needs to be forwarded to the nodes in other IO groups for issuing.
Further, even if the data is forwarded to other IO groups for distribution, the default priority node may be taken first. The present invention is not limited to this, and various factors such as the device model, system configuration, and management habit selected by the IO cluster in actual situations should be considered and selected accordingly.
Accordingly, the third preset manner corresponding to S107 and the fourth preset manner corresponding to S109 can be selected for forwarding correspondence substantially according to the idea of the present embodiment, and of course, in the case of different hierarchies, many other problems may also exist, and it is reasonable to modify the forwarding mechanism of each layer within the allowable range. And is not particularly limited herein.
Based on the above technical solution, in the data processing method provided in the embodiment of the present application, whether an abnormality occurs in any of the four layers from top to bottom is determined according to different determination criteria, so as to obtain which layer has a problem, and when a failure causes that IO data cannot be continuously issued due to an abnormality of each layer occurring in different layers, a corresponding forwarding manner is selected according to the difference of the layers, so that the IO data can be normally issued. Under the increasingly diversified abnormal condition of the storage system, the data processing method which is more detailed in layering and performs data forwarding at multiple levels can be provided, so that the storage system is more efficient and stable in operation and has stronger data redundancy protection capability.
Because the situation is complicated and cannot be illustrated by a list, those skilled in the art can realize that many examples exist based on the principle of the basic method provided by the present application in combination with the actual situation, and the method is within the scope of the present application without sufficient inventive effort.
Referring to fig. 4, fig. 4 is a block diagram of a data processing system according to an embodiment of the present disclosure.
The system may include:
an extracting unit 100, configured to extract attribution information from an instruction for issuing IO data to obtain a priority node;
a first determining unit 200, configured to determine whether the priority node is in a high load state, and if the priority node is in the high load state, forward the IO data according to a first preset manner;
a second determining unit 300, configured to determine whether the back end of the system is abnormal, and if the back end of the system is abnormal, forward the IO data according to a second preset manner;
a third determining unit 400, configured to determine whether the Raid is abnormal, and if the Raid is abnormal, forward the IO data according to a third preset manner; wherein, Raid is composed of a plurality of physical disks;
a fourth determining unit 500, configured to determine whether the physical disk is abnormal, and if the physical disk is abnormal, forward the IO data according to a fourth preset manner.
The IO groups can be interconnected, and the objects of interconnection include: control flow and IO data flow, but only nodes within one IO group in the cluster have access to the backend disk. This is why the first predetermined manner cannot be forwarded to other IO groups.
Optionally, the first determining unit includes:
the acquisition subunit is used for acquiring the load state of the priority node to obtain a load state parameter;
the load judging subunit is used for judging whether the load state parameter exceeds a high load threshold value;
the first forwarding subunit is configured to forward the IO data to the peer node if the load state parameter exceeds the high load threshold; wherein, the opposite end node and the priority node form an IO Group.
This layer forwarding, also called upper layer forwarding (UFW), can be considered as one of multipath IO forwarding: a multi-path strategy is configured between the front-end host and the storage, and IO of the host is issued to a priority node in the cluster preferentially; if the IO load of the priority node is too high, the IO can be issued from other nodes, if the IO is not issued to the priority node to which the volume belongs, the IO is issued to another node in an IO group where the priority node belongs, namely the opposite node, the IO is issued from the opposite node; if the IO is issued to other IO groups in the cluster, the IO is forwarded to the priority node through the UFW module for issuing, and the process can ensure that the volume can be accessed in the whole cluster.
Therefore, all nodes in the cluster can be accessed, which means that when a logical volume is created in the IO group, the logical volume needs to be mapped to the front-end host, and if the mapping relationship occurs in the IO group, the logical volume can directly receive the IO of the front-end host; as the service is expanded, a cluster is created at this time, a volume in the IO group needs to be mapped to a front-end host through other nodes inside the cluster, the IO data sent by the front-end host is sent to another IO group in the cluster, the IO group receiving the IO data at this time, that is, a certain controller node in the IO group needs to forward the IO data to a priority node to which the volume in the IO group belongs, and then the IO data is issued from the IO group, where the reason is that a back-end disk can only access in one IO group; if the front-end host is configured with the multipath technology, the volume can receive IO data from different IO groups, and the front-end IO load is balanced.
Optionally, the second judging unit includes:
a back-end abnormality judging subunit, configured to judge whether a back-end abnormality exists at a system back end corresponding to the preferred node;
and the second processing subunit is configured to, if a backend exception exists, forward the IO data to a node in the same cluster where the backend exception does not exist.
This layer of forwarding is also called duplicate forwarding or mirror forwarding (CFW): copy forwarding is a redundant protection technology, and when a mirror module finds that a downstream module, such as a lower layer logical volume or backend software, has a fault or Bug, IO is transferred to a peer node or other IO groups to issue IO data.
The copy forwarding is different from a caching mirroring technology, the copy forwarding does not occur every time, the copy forwarding is enabled only when a module downstream of the IO stack fails or when copy generation is started, data in the copy is stored in a disk instead of a cache, and the copy forwarding is a protection measure in the IO stack.
Optionally, the third determining unit includes:
a Raid abnormity judging subunit, configured to judge whether a Raid abnormity exists in a Raid attributed to the system back end;
and the third processing subunit is configured to forward the IO data to a node in the cluster where the Raid exception does not exist, if the Raid exception exists.
This layer of forwarding is also called managed disk forwarding (MFW): MFW forwarding refers to IO forwarding in a storage pool inside a cluster, and one scenario is that when an MFW module detects that a downstream module Raid fails, IO is transferred to a peer controller and is issued from the peer controller; in another scenario, the managed disk is from another IO group, and the IO needs to be transferred to the IO group where the managed disk is located through the MFW module and then issued.
In other words, MFW forwarding means that when a storage pool is created in the whole cluster, data delivered from the present IO group needs to be forwarded to mdisks in other IO groups in the present cluster for delivery; or when the RAID detects that the module and the downstream module are in fault, the MFW forwards the IO data to the other nodes of the IO group and then sends the data down.
Optionally, the fourth determining unit includes:
the disk abnormity judging subunit is used for judging whether the physical disk forming the Raid has disk abnormity;
and the fourth processing subunit is configured to forward the IO data to a node in the cluster where the disk exception does not exist if the disk exception exists.
This layer of forwarding is also called Disk Forwarding (DFW): the method refers to that when a controller detects that a link of the controller accessing a disk fails or a RAID created by using disks of other IO groups is used, IO data are forwarded to the disk. The DFW forwarding means that the disks forming the Raid are from the same IO group, and at this time, the DFW module is required to forward the IO to the IO group where the disks are located.
The embodiments are described in a progressive manner in the specification, each embodiment focuses on differences from other embodiments, and the same and similar parts among the embodiments are referred to each other. The device disclosed by the embodiment corresponds to the method disclosed by the embodiment, so that the description is simple, and the relevant points can be referred to the method part for description.
Those of skill would further appreciate that the various illustrative elements and algorithm steps described in connection with the embodiments disclosed herein may be implemented as electronic hardware, computer software, or combinations of both, and that the various illustrative components and steps have been described above generally in terms of their functionality in order to clearly illustrate this interchangeability of hardware and software. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the implementation. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present application.
The principles and embodiments of the present application are explained herein using specific examples, which are provided only to help understand the method and the core idea of the present application. It should be noted that, for those skilled in the art, it is possible to make several improvements and modifications to the present application without departing from the principle of the present application, and such improvements and modifications also fall within the scope of the claims of the present application.
It is further noted that, in the present specification, relational terms such as first and second, and the like are used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Also, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other identical elements in a process, method, article, or apparatus that comprises the element.