CN110737716B - A method and device for writing data - Google Patents

Abstract

The application provides a data writing method and device. The method comprises the following steps: the method comprises the steps that a main node stores N sub-logs of a log to the local, the log is generated according to an atomic write request, the atomic write request comprises N sub-writes, information of the sub-writes is recorded in the sub-logs, and N is a positive integer larger than 0; the main node backs up the N sub-logs into a standby node; and the master node marks the logs as valid under the condition that the master node determines that all the N sub-logs are stored in the standby node. The technical scheme provided by the application can ensure the atomicity of a group of write requests through a log mechanism in the data backup process, and avoids the realization of the atomicity through a file system semantic lock in the prior art, so that the process complexity and the system overhead can be reduced.

Description

A method and device for writing data

technical field

The present application relates to the field of storage, and more particularly, to a method and apparatus for writing data.

Background technique

In the storage system, in order to prevent data loss caused by the hardware failure of the operating system or the failure of the storage medium, the locally stored data can be backed up to the standby node. The file system is a software organization that manages and stores file information in the operating system. It is mainly responsible for creating files for users, storing, reading, and dumping files, and controlling file access. In the process of providing data backup storage services, the file system mainly uses atomic operations (that is, it is necessary to ensure that a group of write requests all succeed or fail) to prevent data inconsistency or damage caused by power failure or system crash. For example, network attached storage (NAS) may be a file system that provides efficient and stable storage devices.

In the process of data backup in the prior art, in order to ensure atomic operations between multiple nodes in the file system, it is necessary to add semantic locks in the file system. In the process of data backup, semantic locks are used to ensure the atomic operation of a group of write requests. . However, adding semantic locks to ensure the atomic operation of write requests will complicate the system process. At the same time, after a group of write requests are all written and backed up successfully, semantic locks need to be opened, which increases system overhead.

Therefore, how to ensure the atomic operation of the write request during the data backup process can reduce the process complexity and system overhead.

SUMMARY OF THE INVENTION

The present application provides a data writing method and device, which can ensure the atomicity of a group of write requests through a log mechanism in the process of data backup, avoiding the implementation of atomicity through the semantic lock of the file system in the prior art, so that the Reduce process complexity and system overhead.

In a first aspect, a data writing method is provided, the method comprising: a master node stores N sub-logs of a log locally; the master node backs up the N sub-logs to a standby node; the master node When it is determined that all the N sub-logs are stored in the standby node, the log is marked as valid.

In this embodiment of the present application, a group of write requests may include N sub-writes, where N is a positive integer greater than 0. In order to ensure that the set of write requests all succeed or fail (that is, the set of write requests is one or a series of operations that cannot be interrupted), the set of write requests can be called an atomic write, and the set of write requests can be called an atomic write. It can also be called atomic.

It should be understood that a series of sub-writes in one atomic write can generate one log (log), and N sub-writes in one atomic write can generate N sub-logs (sublog). Among them, the sub-log can record the information of the sub-write (that is, a sub-write can correspond to a sub-log), and the recorded information can include but is not limited to: the stored address (starting address), the written data, the written length of data, etc.

In the embodiments of the present application, the log mechanism is used to ensure the atomicity of a group of write requests in the process of data backup, which avoids implementing the atomicity through the file system semantic lock in the prior art, thereby reducing process complexity and system overhead.

With reference to the first aspect, in some implementations of the first aspect, the method further includes: in the case where the storage of the N sub-logs in the standby node fails, the master node will store the N sub-logs Sublogs are deleted.

Optionally, in some embodiments, if the peer end fails to perform the operation due to resources or various reasons. The embodiment of the present application may adopt a rollback mechanism, which may roll back the log stored at the local end, and ensure the atomicity of the entire system data (either a set of write requests succeed or fail).

In the embodiment of the present application, the atomicity of the write request between nodes is guaranteed by the rollback mechanism, which avoids relying on retry to ensure the atomicity of the write request between the nodes in the prior art, and improves the serviceability of the system.

With reference to the first aspect, in some implementations of the first aspect, the N sub-logs carry the same log identification number ID, and the master node stores the N sub-logs in the master node in an orderly manner according to the ID. in the log storage container of the node.

In the embodiment of the present application, during the data backup process, the standby node serves as the disaster recovery backup of the master node, and the data written by the standby node needs to be written in the order of the master node. The order of data written by nodes needs to be consistent. As an example, this embodiment of the present application may generate an identity number (identity, ID) for each log, and multiple sublogs on the log may also have the same log ID. The local node and the standby node can attach sublogs to the corresponding chunks according to their log IDs, and the sublogs in the sublog linked list in the chunks can be arranged in order according to their log IDs.

With reference to the first aspect, in some implementations of the first aspect, the master node receives a notification message, and the notification message is used to instruct the N sub-logs to be stored in the standby node; The log is marked as valid.

In a second aspect, a data writing method is provided. The method includes: a standby node stores N sub-logs of logs backed up by a master node in the standby node; and stores all the N sub-logs in the standby node. The standby node marks the log as valid; the standby node notifies the master node that all the N sub-logs are stored in the standby node.

It should be understood that the log is generated according to one atomic write request, the one atomic write request includes N sub-writes, the sub-log records the information of the sub-writes, and N is a positive integer greater than 0.

In the embodiments of the present application, the log mechanism is used to ensure the atomicity of a group of write requests in the process of data backup, which avoids implementing the atomicity through the file system semantic lock in the prior art, thereby reducing process complexity and system overhead.

With reference to the second aspect, in some implementations of the second aspect, the N sub-logs carry the same log identification number ID, and the standby node stores the N sub-logs in the standby node in an orderly manner according to the ID. in the log storage container of the node.

With reference to the second aspect, in some implementations of the second aspect, the standby node sends a notification message to the primary node, where the notification message is used to instruct the N sub-logs to be all stored in the log of the standby node in storage container.

In a third aspect, a data writing device is provided, the device comprising: a storage module for storing N sub-logs of a log locally; a sending module for backing up the N sub-logs to a standby node; The first processing module is configured to mark the log as valid when it is determined that all the N sub-logs are stored in the standby node.

It should be understood that the log is generated according to an atomic write request, and the one atomic write request includes N sub-writes, each of the N sub-logs records the information of each sub-write, and N is a positive integer greater than 0 .

The data writing device provided by the embodiments of the present application can ensure the atomicity of a group of write requests through a log mechanism during the data backup process, avoiding the implementation of atomicity through file system semantic locks in the prior art, thereby reducing the process flow complexity and system overhead.

With reference to the third aspect, in some implementations of the third aspect, the apparatus further includes: a second processing module, configured to store the stored N sub-logs in the case of failure to store the N sub-logs in the standby node. The N sub-logs are deleted.

With reference to the third aspect, in some implementations of the third aspect, the N sub-logs carry the same log identification number ID, and the storage module is specifically configured to: the master node stores the N sub-logs according to the IDs The logs are stored in the log storage container of the master node in an orderly manner.

With reference to the third aspect, in some implementations of the third aspect, the first processing module is specifically configured to: receive a notification message, where the notification message is used to instruct all the N sub-logs to be stored in the standby node ; mark the log as valid.

It should be noted that, in the physical device shown in the third aspect, the processor may implement the steps executed by each module by calling the computer program in the memory. For example, the computer instructions stored in the cache may be invoked by the processor to execute the steps required by each module (storage module, sending module, first processing module, and second processing module).

In a fourth aspect, a data writing device is provided, the device comprising: a storage module for storing N sub-logs of a log backed up by a master node in the standby node; a processing module for storing the log in the standby node When all N sub-logs are stored in the standby node, the log is marked as valid; the sending module is used for the standby node to notify the master node that all the N sub-logs are stored in the standby node.

It should be understood that the log is generated according to an atomic write request, and the one atomic write request includes N sub-writes, each of the N sub-logs records the information of each sub-write, and N is a positive integer greater than 0 .

The data writing device provided by the embodiment of the present application can ensure the atomicity of a group of write requests through a log mechanism during the data backup process, avoiding the implementation of atomicity through the semantic lock of the file system in the prior art, thereby reducing the process flow complexity and system overhead.

With reference to the fourth aspect, in some implementations of the fourth aspect, the N sub-logs carry the same log identification number ID, and the storage module is specifically configured to: store the N sub-logs in an orderly manner according to the IDs into the log storage container of the standby node.

With reference to the fourth aspect, in some implementations of the fourth aspect, the sending module is specifically configured to: send a notification message to the master node, where the notification message is used to instruct the N sub-logs to be stored in the in the log storage container of the standby node.

It should be noted that, in the physical device shown in the fourth aspect, the processor may implement the steps executed by each module by calling the computer program in the memory. For example, the computer instructions stored in the cache can be invoked by the processor to execute the steps required by each module (storage module, processing module, and sending module).

In a fifth aspect, a master node is provided, which is characterized by comprising an input and output interface, a processor and a memory, wherein the processor is used to control the input and output interface to send and receive information, the memory is used to store a computer program, and the The processor is configured to call and run the computer program from the memory, so that the master node executes the method described in the first aspect or any possible implementation manner of the first aspect.

Optionally, the processor may be a general-purpose processor, which may be implemented by hardware or software. When implemented by hardware, the processor can be a logic circuit, an integrated circuit, etc.; when implemented by software, the processor can be a general-purpose processor, implemented by reading software codes stored in a memory, which can Integrated in the processor, can be located outside the processor, independent existence.

A sixth aspect provides a standby node, characterized in that it includes an input and output interface, a processor and a memory, wherein the processor is used to control the input and output interface to send and receive information, the memory is used to store a computer program, and the The processor is configured to call and run the computer program from the memory, so that the master node executes the method described in the second aspect or any possible implementation manner of the second aspect.

Optionally, the processor may be a general-purpose processor, which may be implemented by hardware or software. When implemented by hardware, the processor can be a logic circuit, an integrated circuit, etc.; when implemented by software, the processor can be a general-purpose processor, implemented by reading software codes stored in a memory, which can Integrated in the processor, can be located outside the processor, independent existence.

In a seventh aspect, a computer program product is provided, the computer program product comprising: computer program code, which, when the computer program code is run on a computer, causes the computer to perform the methods in the above aspects.

In an eighth aspect, a computer-readable medium is provided, and the computer-readable medium stores program codes, which, when executed on a computer, cause the computer to execute the methods in the above-mentioned aspects.

In a ninth aspect, a system is provided, including the aforementioned master node and a backup node.

Description of drawings

FIG. 1 is a schematic block diagram of a scenario to which this embodiment of the present application is applicable.

FIG. 2 is a schematic flowchart of a data writing method provided by an embodiment of the present application.

FIG. 3 is a schematic structural diagram of a data writing system provided by an embodiment of the present application.

FIG. 4 is a schematic structural diagram of a data writing system provided by another embodiment of the present application.

FIG. 5 is a schematic structural diagram of a data writing system provided by another embodiment of the present application.

FIG. 6 is a schematic flowchart of a data writing method provided by another embodiment of the present application.

FIG. 7 is a schematic structural diagram of a data writing system provided by another embodiment of the present application.

FIG. 8 is a schematic block diagram of a data writing apparatus 800 provided by an embodiment of the present application.

FIG. 9 is a schematic block diagram of a data writing apparatus 900 provided by an embodiment of the present application.

FIG. 10 is a schematic block diagram of a master node 1000 provided by an embodiment of the present application.

FIG. 11 is a schematic block diagram of a standby node 1100 provided by an embodiment of the present application.

Detailed ways

The technical solutions in the present application will be described below with reference to the accompanying drawings.

In a distributed storage system, system availability is one of the important indicators, and it is necessary to ensure that the availability of the system is not affected when a machine fails. In order to ensure system availability, data needs to be backed up (that is, multiple copies of stored data need to be saved, and the data of the multiple copies needs to be consistent), and when a machine failure causes one copy to fail, other copies can still provide services.

FIG. 1 is a schematic block diagram of a scenario to which this embodiment of the present application is applicable. The scene shown in FIG. 1 may include at least two nodes. The source node that can be replicated is generally referred to as the primary node 110 , and the replicated destination node(s) may be referred to as the standby node 120 .

The master node 110 is the management node of the entire file system, and can receive operation requests from users. As an example, after receiving the write request for the primary node 110 , the data can be written locally, and the written data can be backed up on the standby node 120 . The master node 110 may also be referred to as a master replica.

The standby node 120, as a disaster recovery backup based on disk data, enables the standby node 120 to quickly take over applications and application data when the host performs regular planned maintenance or when the host fails (unplanned), so that user access is not possible. Disrupted by planned or unplanned downtime. As an example, the standby node 120 may back up the data written by the primary node 110 . The standby node 120 may also be referred to as a secondary replica.

As a software organization that manages and stores file information in the operating system, the file system is mainly responsible for creating files for users, storing, reading, and dumping files, and controlling access to files.

For example, network attached storage (NAS) may be a computer data storage device connected to a computer network, and is therefore also referred to as "network storage." NAS is a dedicated data storage server, which is data-centric, completely separates the storage device from the server, and centrally manages data. As a business form that is widely used in storage systems, NAS mainly provides an efficient and stable file system for storage devices. NAS devices can usually be used with typical file system sharing protocols to be shared with hosts to provide storage services for a variety of industries.

In many scenarios (for example, a data storage system involving more than 2 nodes) for providing a file system service, it is necessary to ensure that a group of write requests all succeed or all fail. This group of write requests can be called an atomic write. For example, in order to ensure that each update is all successful or all failed even in the unexpected time of power failure, the data write request of the storage system needs to be atomic, and it needs to meet the requirements of atomicity.

It should be understood that an atomic write may also be referred to as a set of write requests being atomic or the set of write requests being an atomic operation (ie, the set of write requests being an uninterruptible one or a series of operations).

In the prior art, the atomicity of a set of write requests is guaranteed by the semantic lock of the file system. However, the increased semantic lock will cause the system process to be complicated and increase the system overhead.

In the process of data backup, in order to ensure the atomicity of write requests between two or more nodes, and also reduce the complexity of the system process and the system overhead, the embodiment of the present application provides a data writing method, which can avoid the existing In the technology, the complex process and large system overhead caused by atomicity are realized through the semantic lock of the file system.

Based on the architecture shown in FIG. 1 , the data writing method according to the embodiment of the present application will be described in detail below with reference to FIG. 2 . FIG. 2 is a schematic flowchart of a data writing method provided by an embodiment of the present application. The method shown in FIG. 2 may include steps 210-230. Steps 210-230 will be described in detail below, respectively.

In step 210, the master node stores the N sub-logs of the log locally.

In this embodiment of the present application, a group of write requests may include N sub-writes, where N is a positive integer greater than 0. In order to ensure that the set of write requests all succeed or fail (that is, the set of write requests is one or a series of operations that cannot be interrupted), the set of write requests can be called an atomic write, and the set of write requests can be called an atomic write. It can also be called atomic.

A series of sub-writes in an atomic write can generate a log (log), and N sub-writes in an atomic write can generate N sublogs (sublogs). Among them, the sub-log can record the information of the sub-write (that is, a sub-write can correspond to a sub-log), and the recorded information can include but is not limited to: the stored address (starting address), the written data, the written length of data, etc.

The master node can store the generated N sub-logs locally. As an example, the master node can store the N sub-logs in the corresponding log storage container according to the starting address of the record. For example, the master node can store the N sub-logs according to the record. The starting address is stored in the corresponding chunk. The writing process of an atomic write request will be described in detail below with reference to FIG. 3 , which will not be repeated here.

In step 220, the master node backs up the N sublogs to the standby node.

In this embodiment of the present application, the master node may back up the generated N sub-logs to the standby node. The standby node can store the N sub-logs written by the primary node in the standby node. As an example, the standby node can store the N sub-logs in the corresponding log storage container according to the starting address of the record. For example, the standby node can store the N sub-logs in the corresponding log storage container. N sublogs are stored in the corresponding chunk according to the starting address of the record. The writing process of an atomic write request will be described in detail below with reference to FIG. 3 , which will not be repeated here.

It should be noted that the address of the chunk in the standby node is the same as the address in the master node, and the slave node can store N sublogs in the corresponding chunk according to the recorded starting address, which is the same as the operation process in the master node.

In step 230, the master node marks the log as valid when it is determined that all the N sub-logs are stored in the standby node.

In the embodiment of the present application, after storing all N sub-logs locally, the master node may determine whether the standby node stores all N sub-logs in the corresponding log storage container. If the master node can determine that the standby node stores all N sublogs in the corresponding log storage container, the master node can mark the log as valid after storing all the N sublogs locally. As an example, the master node may add a flag bit in the log, the master node may modify the state of the flag bit, and the flag bit may be set as valid.

It should be understood that in the embodiment of the present application, when the master node marks the log as valid, it may indicate that the N sub-logs of the log have been stored locally, and the N sub-logs have also been backed up to the standby node, and an atomic write request can be considered to have been All writes to the primary and secondary nodes can ensure the atomicity of a set of write requests between nodes.

In the embodiment of the present application, in the data backup process, the atomicity of the write request between the primary node and the standby node is realized through the log mechanism, which can avoid using the semantic lock of the file system in the prior art, thereby reducing the complexity of the system process and the system overhead.

The specific implementation of the writing process of the atomic write request in the embodiment of the present application is described in more detail below with reference to specific examples. It should be noted that the example in FIG. 3 is only for helping those skilled in the art to understand the embodiments of the present application, and is not intended to limit the embodiments of the present application to specific numerical values or specific scenarios exemplified. Those skilled in the art can obviously make various equivalent modifications or changes according to the example given in the example of FIG. 3 , and such modifications and changes also fall within the scope of the embodiments of the present application.

FIG. 3 is a schematic structural diagram of a data writing system provided by an embodiment of the present application. The system shown in FIG. 3 may include chunks 310 , chunks 320 , and chunks 330 .

The sublogs stored in the chunk 310 in the form of a linked list may include: a sublog (sublog) 312 , a sublog (sublog) 314 , and a sublog (sublog) 316 .

The sublogs stored in the chunk 320 in the form of a linked list may include: a sublog (sublog) 322 , a sublog (sublog) 324 , and a sublog (sublog) 326 .

The sublogs stored in the chunk 330 in the form of a linked list may include: a sublog (sublog) 332 , a sublog (sublog) 334 , and a sublog (sublog) 336 .

It should be understood that in the embodiment of the present application, a series of sub-writes in an atomic write request can generate a log (log), N sub-writes in one atomic write can generate N sub-logs (sublogs), and a sub-log can record one sub-write address, data to be written, length of data to be written, etc. That is to say, an atomic write request can correspond to a log, and each sub-write in an atomic write corresponds to a sublog.

In this embodiment of the present application, a chunk may be a logically divided piece of memory, a basic data index unit, and may be used to store sublogs. The node can store the sublog in the chunk of the corresponding address in the form of a linked list according to the starting address of the sub-write recorded in the sublog.

The sublog generated by the write request can find the corresponding chunk according to the address of its recorded sub-write, and can add it to the sublog list on the chunk. As an example, a read request can find the corresponding chunk through the address and offset recorded in the sublog. As an example, the write request can also find the corresponding chunk according to the address and offset recorded in the sublog.

It should be understood that a linked list may be a non-consecutive and non-sequential storage structure on a physical storage unit, and the logical order of data elements may be sequentially implemented by connecting pointers in the linked list. A linked list can consist of a series of nodes, and a linked list can allow insertion or removal of nodes anywhere on the list. As an example, multiple sublogs on the chunk linked list can be considered as a series of nodes on the chunk linked list, and sublogs on the chunk linked list can be inserted or removed.

The following takes the write request as an example for detailed description.

Referring to FIG. 3 , a log 311 generated by a group of write requests may include sublog 316 , sublog 324 , and sublog 334 , wherein sublog 316 , sublog 324 , and sublog 334 may be organized within log 311 in the form of a linked list.

The sublog can find the corresponding chunk according to its recorded address and offset. For example, sublog316 can be added to the linked list of chunk 310, sublog 324 can be added to the linked list of chunk 320, and sublog334 can be added to the linked list of chunk 320.

It should be noted that the solid line box in the figure can be used to indicate that multiple sublogs included in a log have all been stored in the corresponding chunks, and the multiple sublogs have been marked as valid (a group of write requests have been written in all In memory, the atomicity of this group of write requests can be guaranteed) and can be read by read requests. For example, sublog 312, sublog 314, sublog 322, and sublog 332 have all been written into the corresponding chunks and can be read by read requests. The dotted box in the figure can be used to indicate that multiple sublogs included in a log have not been stored in the corresponding chunk, the log has not been marked as valid, and multiple sublogs on the log cannot be read by read requests. arrive. For example, sublog 316, sublog 324, sublog 334, and sublog 332 included in log 311 have not been marked as valid and cannot be read by read requests.

It should be understood that sublog 322 on the linked list of chunk 320 can be used to indicate that before log 311 is written, it has been stored in the chunk in the form of a linked list, and sublog 326 can be used to indicate that while log 311 is written, there may be other Write data is stored on the chunk 320 linked list.

The node can mark log 311 as valid after all sublog 316, sublog 324, and sublog 334 in log 311 are stored in the corresponding chunk linked list. For example, a marker bit can be added to log 311. After all sublogs in log 311 are stored in the corresponding chunk linked list, the marker bit can be set as valid, and all sublogs in log 311 can be read by a read request. . For details, please refer to the system structure diagram shown in FIG. 4. The log 311 in FIG. 4 has been marked as valid (a group of write requests has been written into the memory, and the atomicity of the group of write requests can be guaranteed). The sublog 316, sublog 324, and sublog 334 can be read by the read request.

Optionally, after a log is marked as valid (that is, all sublogs on the log are stored in the corresponding chunk linked list), an atomic write request operation can be considered to have been completed. The memory space occupied by the log can be released, and the released memory space can be recycled. For details, please refer to the system structure diagram shown in FIG. 5. Compared with the system structure diagram shown in FIG. 4, the sublog 316, sublog 324, and sublog 334 on the log 311 in FIG. 5 have all been stored in the corresponding chunk. , the memory space occupied by log 311 can be released.

It should be understood that if the log of an atomic write request is released, it can also be considered that all sublogs on the log are stored in the corresponding chunk linked list, and an atomic write request has been completed. For example, if the pointer to log is null, an atomic write request can be considered complete and can be read by a read request.

In the embodiment of the present application, the atomic write in the process can be realized through the mechanism of the log and the sub-log, the atomicity of a group of write requests can be guaranteed, and the atomicity of a group of write requests can be avoided in the prior art by using a file system semantic lock , which can reduce process complexity and system overhead.

Optionally, in some embodiments, in the data backup process, the backup node serves as the data backup of the master node, and the data written by the backup node needs to be written in the order of the master node, that is, the data written by the backup node is The order of data and data written by the master node needs to be consistent. As an example, this embodiment of the present application may generate an identity number (identity, ID) for each log, and multiple sublogs on the log may also have the same log ID. The local node and the standby node can attach sublogs to the corresponding chunks according to their log IDs, and the sublogs in the sublog linked list in the chunks can be arranged in order according to their log IDs.

The specific implementation manner of ensuring the atomicity of a group of write requests between the primary node and the backup node in the embodiment of the present application is described in more detail below with reference to specific examples. It should be noted that the example in FIG. 6 is only for helping those skilled in the art to understand the embodiments of the present application, and is not intended to limit the embodiments of the present application to specific numerical values or specific scenarios exemplified. Those skilled in the art can obviously make various equivalent modifications or changes according to the example given in the example of FIG. 6 , and such modifications and changes also fall within the scope of the embodiments of the present application.

Figure 6 shows the atomic write process of the primary node and the standby node in the storage system. It should be understood that the local end in Figure 6 corresponds to the primary node above, and the opposite end corresponds to the secondary node above. The method of FIG. 6 may include steps 610-670, which are described in detail below.

Step 610, the local end writes input/output (input/output, I/O).

The local end (it can be understood that the controller) can receive the write I/O instruction of the operating system and the data to be written. In this embodiment of the present application, a series of sub-writes in a group of write requests can generate a log, and the sub-writes can generate a sublog , in which the data to be written, the length of the data and the address where the data is stored can be recorded in the sublog.

Step 620, the local end generates a log (log) ID.

The local end can generate a log ID for the log after receiving the write I/O command from the operating system, a generated log, and multiple sublogs. Multiple sublogs of the log can have the same log ID as the log, so that the peer end can write data in the order of the local end when writing data, so that the data written by the peer end can be consistent with the data written by the local end.

Step 630, the local end writes the local.

The local end can write the received multiple sublogs locally. For example, the local end can write the sublogs to the corresponding log storage container (for example, a chunk) according to the logID of each sublog. The sublog list in the chunk can be Sort by log ID. Please refer to the data writing method described in FIG. 3 for the processing flow of local writing, which will not be repeated here.

Step 640, write the mirror image to the opposite end.

The opposite end (it can be understood that the controller) can receive multiple sublogs to be backed up sent by the local end. The multiple sublogs can carry log IDs. The opposite end can write the sublogs to the corresponding log storage container according to the log ID of each sublog. (For example, it can be a chunk), the sublog linked list in the chunk can be ordered according to the log ID.

The above process of writing the sublog to the peer end according to the log ID can also be called mirroring, that is, backing up multiple sublogs stored on the local end to the peer end. Please refer to the data writing method described in FIG. 3 for the processing flow of the mirror image at the opposite end, which will not be repeated here.

It should be noted that the local end may only send multiple sublogs to be backed up to the opposite end, and the local end may also send the log and multiple sublogs of the log to the opposite end, which is not specifically limited in this embodiment of the present application. If the peer end only receives multiple sublogs sent by the local end, the peer end can generate corresponding logs.

Step 650, valid for terminal data.

After the peer end stores all the received sublogs in the corresponding chunk according to the log ID, the peer end can mark the log positions of the multiple sublogs as valid. For the specific processing flow, please refer to the data writing method described in FIG. 3 , which will not be repeated here.

The peer end can send a notification message to the local end after storing multiple sublogs in the corresponding chunk, and can notify the local end of the result of this data backup. As an example, if the peer end stores all multiple sublogs in the corresponding chunk and sets the log's mark position to be valid, the notification message sent by the peer end to the local end can be used to indicate that an atomic write request of the peer end has been completed. Finish. As another example, if the mirror-writing operation fails at the opposite end due to resources or various other reasons, the notification message sent by the opposite end to the local end may be used to indicate the failure of the mirror-writing at the opposite end.

In step 660, the local set data is valid.

After the local end determines that the peer end stores all the multiple sublogs in the corresponding chunk, the local end can mark the log positions of the multiple sublogs as valid. For the specific processing flow, please refer to the data writing method described in FIG. 3 , which will not be repeated here.

Step 670, the local end completes.

The local end sets the log generated by the atomic write request to be valid, which can be used to indicate that the atomic write request has been successfully written on two nodes and can be read by the read request.

Optionally, in some embodiments, if the local end does not set the marked position of the log to be valid after determining that the opposite end is completely executed (the process fails to execute), it may be considered that the local end is abnormal. The system can use the peer data as a benchmark. After all the data on the local end is invalid, all the data on the opposite end can be synchronized to the local end.

Optionally, in some embodiments, if the peer end fails to perform the operation due to resources or various reasons. The embodiment of the present application may adopt a rollback mechanism, which may roll back the log stored at the local end, and ensure the atomicity of the entire system data (either a set of write requests succeed or fail).

The specific implementation of the rollback mechanism in the embodiment of the present application is described in more detail below with reference to specific examples. It should be noted that the example in FIG. 7 is only for helping those skilled in the art to understand the embodiments of the present application, and is not intended to limit the embodiments of the present application to specific numerical values or specific scenarios exemplified. Those skilled in the art can obviously make various equivalent modifications or changes according to the example given in the example of FIG. 7 , and such modifications and changes also fall within the scope of the embodiments of the present application.

If the storage system needs to be rolled back, the local end can roll back the log as needed. Through the sublog linked list in the log, the local end can determine the sublog related to it, and can delete the sublog from the chunk linked list in turn to complete the rollback. roll operation.

See Figure 3 for the system before the local end rolls back. As an example, if the peer end fails to store the sublog in log311, the local end can save sublog 316, sublog 324, and sublog334 from the chunk where it is located according to the sublog linked list in log311. Delete on the linked list to complete the rollback operation.

The system after the local end rolls back can be seen in Figure 7. The sublogs in log311 written in Figure 7 have all been deleted from the chunk list where they are located.

In the embodiment of the present application, the atomicity of the write request between nodes is guaranteed by the rollback mechanism, which avoids relying on retry to ensure the atomicity of the write request between the nodes in the prior art, and improves the serviceability of the system.

The data writing method provided by the embodiments of the present application is described in detail above with reference to FIGS. 1 to 7 , and the embodiments of the data writing apparatus of the present application will be described in detail below with reference to FIGS. 8 to 11 . It should be understood that the descriptions of the method embodiments correspond to the descriptions of the apparatus embodiments. Therefore, for the parts not described in detail, reference may be made to the foregoing method embodiments.

FIG. 8 is a schematic block diagram of a data writing apparatus 800 provided by an embodiment of the present application. The data writing apparatus 800 may include:

A storage module 810, configured to store the N sub-logs of the log locally;

A sending module 820, configured to back up the N sub-logs to the standby node;

The first processing module 830 is configured to mark the log as valid when it is determined that all the N sub-logs are stored in the standby node.

It should be understood that the log is generated according to an atomic write request, and the one atomic write request includes N sub-writes, each of the N sub-logs records the information of each sub-write, and N is a positive integer greater than 0

Optionally, in some embodiments, the data writing apparatus 800 may further include:

The second processing module 840 is configured to delete the stored N sub-logs when the storage of the N sub-logs in the standby node fails.

Optionally, in some embodiments, the N sub-logs carry the same log identification number ID, and the storage module 810 is specifically configured to: the master node stores the N sub-logs in an orderly manner according to the ID. in the log storage container of the master node.

Optionally, in some embodiments, the first processing module 830 is specifically configured to: receive a notification message, where the notification message is used to instruct all the N sub-logs to be stored in the standby node; marked as valid.

It should be noted that, in the physical device 800 shown in FIG. 8 , the processor may implement the steps executed by each module by calling the computer program in the memory. For example, the computer instructions stored in the cache may be invoked by the processor to execute the steps required by each module (the storage module 810, the sending module 820, the first processing module 830, and the second processing module 840).

FIG. 9 is a schematic block diagram of a data writing apparatus 900 provided by an embodiment of the present application. The data writing device 900 may include:

A storage module 910, configured to store the N sub-logs of the log backed up by the primary node in the standby node;

A processing module 920, configured to mark the log as valid when all the N sub-logs are stored in the standby node;

The sending module 930 is used for the standby node to notify the master node that all the N sub-logs are stored in the standby node.

It should be understood that the log is generated according to an atomic write request, and the one atomic write request includes N sub-writes, each of the N sub-logs records the information of each sub-write, and N is a positive integer greater than 0 .

Optionally, in some embodiments, the N sub-logs carry the same log identification number ID, and the storage module 910 is specifically configured to: store the N sub-logs in an orderly manner on the standby node according to the IDs in the log storage container.

Optionally, in some embodiments, the sending module 930 is specifically configured to: send a notification message to the master node, where the notification message is used to instruct all the N sub-logs to be stored in the log storage of the standby node in the container.

It should be noted that, in the physical device 900 shown in FIG. 9 , the processor can implement the steps executed by each module by calling the computer program in the memory. For example, the computer instructions stored in the cache may be invoked by the processor to execute the steps required by each module (the storage module 910, the processing module 920, and the sending module 930).

FIG. 10 is a schematic block diagram of a master node 1000 provided by an embodiment of the present application. The master node 1000 may include: memory 1001 , processing 1002 , and input/output interface 1003 .

The memory 1001, the processor 1002 and the input/output interface 1003 are connected through an internal connection path, the memory 1001 is used for storing program instructions, and the processor 1002 is used for executing the program instructions stored in the memory 1001 to control the input/output interface 1003 receives input data and information, and outputs data such as operation results.

It should be understood that, in this embodiment of the present application, the processor 1002 may adopt a central processing unit (central processing unit, CPU), and the processor may also be other general-purpose processors, digital signal processors (digital signal processors, DSP), application-specific integrated circuits (application specific integrated circuit, ASIC), off-the-shelf programmable gate array (field programmable gate Array, FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, and the like. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like. Alternatively, the processor 1002 uses one or more integrated circuits to execute related programs to implement the technical solutions provided by the embodiments of the present application.

The memory 1001 may include read only memory and random access memory, and provides instructions and data to the processor 1002 . A portion of the processor 1002 may also include non-volatile random access memory. For example, the processor 1002 may also store device type information.

In the implementation process, each step of the above-mentioned method can be completed by an integrated logic circuit of hardware in the processor 1002 or an instruction in the form of software. The methods disclosed in conjunction with the embodiments of the present application may be directly embodied as executed by a hardware processor, or executed by a combination of hardware and software modules in the processor. The software modules may be located in random access memory, flash memory, read-only memory, programmable read-only memory or electrically erasable programmable memory, registers and other storage media mature in the art. The storage medium is located in the memory 1001, and the processor 1002 reads the information in the memory 1001, and completes the steps of the above method in combination with its hardware. To avoid repetition, detailed description is omitted here.

It should be understood that the master node 1000 according to the embodiment of the present application may correspond to the apparatus 800 in the embodiment of the present application, and may be used to execute the corresponding processes of each method in FIG. 2 to FIG. 7 in the embodiment of the present application, and the master node 1000 The above-mentioned and other operations and/or functions of each module in FIG. 2 to FIG. 7 in the embodiments of the present application respectively implement the corresponding processes of the methods in FIG. 2 to FIG. 7 , and are not repeated here for the sake of brevity.

FIG. 11 is a schematic block diagram of a standby node 1100 provided by an embodiment of the present application. The standby node 1100 may include: a memory 1101 , a processing 1102 , and an input/output interface 1103 .

The memory 1101, the processor 1102 and the input/output interface 1103 are connected through an internal connection path, the memory 1101 is used to store program instructions, and the processor 1102 is used to execute the program instructions stored in the memory 1101 to control the input/output interface 1103 receives input data and information, and outputs data such as operation results.

It should be understood that, in this embodiment of the present application, the processor 1102 may adopt a central processing unit (central processing unit, CPU), and the processor may also be other general-purpose processors, digital signal processors (digital signal processors, DSP), application-specific integrated circuits (application specific integrated circuit, ASIC), off-the-shelf programmable gate array (field programmable gate Array, FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, and the like. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like. Alternatively, the processor 1102 uses one or more integrated circuits to execute related programs, so as to implement the technical solutions provided by the embodiments of the present application.

The memory 1101 may include read only memory and random access memory, and provides instructions and data to the processor 1102 . A portion of the processor 1102 may also include non-volatile random access memory. For example, the processor 1102 may also store device type information.

In the implementation process, each step of the above-mentioned method can be completed by an integrated logic circuit of hardware in the processor 1102 or an instruction in the form of software. The methods disclosed in conjunction with the embodiments of the present application may be directly embodied as executed by a hardware processor, or executed by a combination of hardware and software modules in the processor. The software modules may be located in random access memory, flash memory, read-only memory, programmable read-only memory or electrically erasable programmable memory, registers and other storage media mature in the art. The storage medium is located in the memory 1101, and the processor 1102 reads the information in the memory 1101, and completes the steps of the above method in combination with its hardware. To avoid repetition, detailed description is omitted here.

It should be understood that the standby node 1100 according to the embodiment of the present application may correspond to the apparatus 900 in the embodiment of the present application, and may be used to execute the corresponding processes of each method in FIG. 2 to FIG. 7 in the embodiment of the present application, and the standby node 1100 The above-mentioned and other operations and/or functions of each module in FIG. 2 to FIG. 7 in the embodiments of the present application respectively implement the corresponding processes of the methods in FIG. 2 to FIG. 7 , and are not repeated here for the sake of brevity.

Through the above description, the primary node 1000 and the backup node 1100 provided by the embodiments of the present application can ensure the atomicity of a group of write requests through a log mechanism during the data backup process, avoiding the implementation of file system semantic locks in the prior art Atomicity, which can reduce process complexity and system overhead.

It should be understood that, in various embodiments of the present application, the size of the sequence numbers of the above-mentioned processes does not mean the sequence of execution, and the execution sequence of each process should be determined by its functions and internal logic, and should not be dealt with in the embodiments of the present application. implementation constitutes any limitation.

Those of ordinary skill in the art can realize that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Skilled artisans may implement the described functionality using different methods for each particular application, but such implementations should not be considered beyond the scope of this application.

Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the above-described systems, devices and units may refer to the corresponding processes in the foregoing method embodiments, which will not be repeated here.

In the several embodiments provided in this application, it should be understood that the disclosed system, apparatus and method may be implemented in other manners. For example, the apparatus embodiments described above are only illustrative. For example, the division of the units is only a logical function division. In actual implementation, there may be other division methods. For example, multiple units or components may be combined or Can be integrated into another system, or some features can be ignored, or not implemented. On the other hand, the shown or discussed mutual coupling or direct coupling or communication connection may be through some interfaces, indirect coupling or communication connection of devices or units, and may be in electrical, mechanical or other forms.

The units described as separate components may or may not be physically separated, and components displayed as units may or may not be physical units, that is, may be located in one place, or may be distributed to multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution in this embodiment.

In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically alone, or two or more units may be integrated into one unit.

The functions, if implemented in the form of software functional units and sold or used as independent products, may be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application can be embodied in the form of a software product in essence, or the part that contributes to the prior art or the part of the technical solution. The computer software product is stored in a storage medium, including Several instructions are used to cause a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present application. The aforementioned storage medium includes: U disk, removable hard disk, read-only memory (ROM), random access memory (RAM), magnetic disk or optical disk and other media that can store program codes .

The above are only specific embodiments of the present application, but the protection scope of the present application is not limited to this. should be covered within the scope of protection of this application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims (18)

1. A method of data writing, the method comprising:

the method comprises the following steps that a main node stores N sub-logs of a log to the local, the log is generated according to an atomic write request, the atomic write request is one or a series of operations which cannot be interrupted, and the log comprises: marking bits, wherein the atomic write request comprises N sub-writes, the sub-log records the information of the sub-writes, and N is a positive integer greater than 0;

the master node backs up the N sub-logs to a standby node;

and the master node modifies the state of the marking bit in the log under the condition that the N sub-logs are all stored in the standby node, so that the log is marked as valid.

2. The method of claim 1, further comprising:

and under the condition that the N sub-logs in the standby node fail to be stored, the main node deletes the stored N sub-logs.

3. The method according to claim 1 or 2, wherein the N sub-logs carry the same log identification number ID,

the main node stores N sub-logs of the log to the local, including:

and the main node orderly stores the N sub-logs into a log storage container of the main node according to the ID.

4. The method according to any one of claims 1 to 3, wherein the master node, in case that it is determined that all of the N sub-logs are stored to the standby node, marking the log as valid comprises:

the main node receives a notification message, wherein the notification message is used for indicating that all the N sub-logs are stored in the standby node;

the master node marks the log as valid.

5. A method of data writing, the method comprising:

the backup node stores N sub-logs of a log backed up by a master node into the backup node, wherein the log is generated according to an atomic write request, the atomic write request is one or a series of operations which cannot be interrupted, and the log comprises: marking bits, wherein the atomic write request comprises N sub-writes, the sub-log records the information of the sub-writes, and N is a positive integer greater than 0;

when the N sub-logs are all stored in the standby node, the standby node modifies the state of the marking bit in the log, so that the log is marked as valid;

and the standby node informs the main node that all the N sub-logs are stored in the standby node.

6. The method of claim 5, wherein the N sub-logs carry the same log identification number ID,

the backup node stores N sub-logs of the log backed up by the master node into the backup node, and the method comprises the following steps:

and the standby node stores the N sub-logs into a log storage container of the standby node in order according to the ID.

7. The method of claim 5 or 6, wherein the notifying, by the standby node, that the N sub-logs are all stored in the standby node to the master node comprises:

and the standby node sends a notification message to the main node, wherein the notification message is used for indicating that all the N sub-logs are stored in a log storage container of the standby node.

8. An apparatus for writing data, the apparatus comprising:

a storage module, configured to store N sub-logs of a log locally, where the log is generated according to an atomic write request, where the atomic write request is one or a series of operations that cannot be interrupted, and the log includes: marking bits, wherein the atomic write request comprises N sub-writes, each sub-log in the N sub-logs records information of each sub-write, and N is a positive integer greater than 0;

the sending module is used for backing up the N sub-logs to a standby node;

and the first processing module is used for modifying the state of the marking bit in the log under the condition that the N sub-logs are all stored in the standby node, so that the log is marked to be valid.

9. The apparatus of claim 8, further comprising:

and the second processing module is used for deleting the N sub-logs stored under the condition that the N sub-logs in the standby node fail to be stored.

10. The apparatus according to claim 8 or 9, wherein the N sub-logs carry the same log identification number ID,

the storage module is specifically configured to:

and the main node orderly stores the N sub-logs into a log storage container of the main node according to the ID.

11. The apparatus according to any one of claims 8 to 10, wherein the first processing module is specifically configured to:

receiving a notification message, wherein the notification message is used for indicating that all the N sub-logs are stored in the standby node;

marking the log as valid.

12. An apparatus for writing data, the method comprising:

a storage module, configured to store N sub-logs of a log backed up by a master node into the backup node, where the log is generated according to an atomic write request, the atomic write request is one or a series of operations that cannot be interrupted, and the log includes: marking bits, wherein the atomic write request comprises N sub-writes, each sub-log in the N sub-logs records information of each sub-write, and N is a positive integer greater than 0;

the processing module is used for modifying the state of the marking bit in the log when the N sub-logs are all stored in the standby node, so that the log is marked to be valid;

and the sending module is used for informing the master node that all the N sub-logs are stored in the standby node by the standby node.

13. The apparatus of claim 12, wherein the N sub-logs carry a same log identification number (ID),

the storage module is specifically configured to:

and orderly storing the N sub-logs into a log storage container of the standby node according to the ID.

14. The apparatus according to claim 12 or 13, wherein the sending module is specifically configured to:

and sending a notification message to the master node, wherein the notification message is used for indicating that all the N sub-logs are stored in a log storage container of the standby node.

15. A master node comprising an input output interface, a processor for controlling the input output interface to transceive information, and a memory for storing a computer program, the processor being configured to retrieve from the memory and execute the computer program such that the master node performs the operational steps of the method of any one of claims 1 to 4.

16. A standby node comprising an input/output interface, a processor for controlling the input/output interface to transceive information, and a memory for storing a computer program, the processor being configured to retrieve and execute the computer program from the memory so that the standby node performs the operation steps of the method of any one of claims 5 to 7.

17. A storage system comprising a master node according to claim 15 and a backup node according to claim 16.

18. A computer-readable storage medium, comprising a computer program which, when run on a computer, causes the computer to perform the method of any one of claims 1 to 7.

Images