Disclosure of Invention
Accordingly, the present invention is directed to a method, a system and a storage medium for managing cache data, which are used for solving the problem that the cache data lacks a good management mode in the prior art.
Based on the above object, the present invention provides a cache data management method, comprising the following steps:
monitoring the cache tasks of all nodes in the distributed storage cluster;
In response to monitoring that a plurality of nodes have cache tasks, pushing each cache task to a cache queue in sequence, and detecting the integrity of each cache task in the cache queue;
In response to detecting that the caching task is complete, analyzing the complete caching task, classifying the complete caching task based on an analysis result, and respectively sending the classified caching task to a database or a distributed caching cluster for storage;
And responding to the fault restoration of the nodes corresponding to the complete cache task, and acquiring the cache task lost by the fault restored nodes from the database and/or the distributed cache cluster.
In some embodiments, in response to monitoring that a number of nodes have cache tasks, pushing each cache task to a cache queue in turn includes:
and in response to monitoring that a plurality of nodes have cache tasks, allocating an identification number for each cache task and sequentially pushing the identification numbers to a cache queue.
In some embodiments, in response to a post-failure repair of a node in the nodes corresponding to the complete cache task, obtaining, from the database and/or the distributed cache cluster, a cache task lost by the post-failure repair node includes:
And responding to the fault restoration of the nodes corresponding to the complete cache task, and acquiring the cache task lost by the fault restoration node from the database and/or the distributed cache cluster according to the identification number.
In some embodiments, detecting the integrity of each cache task in the cache queue comprises:
The integrity of each buffer task is detected by identifying the data length of the buffer task and the information of the head data and the tail data in the buffer queue.
In some embodiments, in response to detecting that the cache task is complete, resolving the complete cache task and classifying based on the resolution result, and sending the classified cache task to the database or the distributed cache cluster for storage respectively includes:
In response to detecting that the buffer task is complete, analyzing the complete buffer task according to the size of the data block, classifying the analyzed buffer task consisting of large data blocks into a first class, and classifying the analyzed buffer task consisting of small data blocks into a second class;
And sending the cache tasks belonging to the first class to the distributed cache cluster for temporary storage, and sending the cache tasks belonging to the second class to the database for storage.
In some embodiments, the method further comprises:
And in response to detecting that the buffer task is incomplete, removing the incomplete buffer task from the buffer queue, and informing a corresponding node to resend the incomplete buffer task.
In some embodiments, the method further comprises:
In response to a node failure, a failure notification is sent to the distributed storage cluster.
In some embodiments, the method further comprises:
and identifying the buffer task needing to be encrypted in the complete buffer task through the buffer basic library, and carrying out encryption processing on the buffer task needing to be encrypted.
In another aspect of the present invention, there is also provided a cache data management system, including:
The cache task monitoring module is configured to monitor the cache task of each node in the distributed storage cluster;
The buffer task detection module is configured to respond to the fact that a plurality of nodes have buffer tasks, push each buffer task to a buffer queue in sequence, and detect the integrity of each buffer task in the buffer queue;
The buffer task classifying module is configured to analyze the complete buffer task and classify the buffer task based on the analysis result in response to detecting that the buffer task is complete, and respectively send the classified buffer task to a database or a distributed buffer cluster for storage, and
And the cache task acquisition module is configured to respond to the failure repair of the node in the nodes corresponding to the complete cache task, and acquire the cache task lost by the failure repaired node from the database and/or the distributed cache cluster.
In yet another aspect of the present invention, there is also provided a computer readable storage medium storing computer program instructions which, when executed, implement any of the methods described above.
In yet another aspect of the present invention, there is also provided a computer device comprising a memory and a processor, the memory storing a computer program which, when executed by the processor, performs any of the methods described above.
The invention has at least the following beneficial technical effects:
According to the method, the system and the device, the cache tasks of all nodes in the distributed storage cluster are monitored, when a plurality of nodes are monitored to have the cache tasks, all the cache tasks are sequentially pushed to the cache queue, then the integrity of all the cache tasks is detected in the cache queue, so that complete cache task information can be screened out, the situation that the cache task information is lost due to network jitter or large delay of the distributed storage cluster is effectively relieved, the complete cache tasks are analyzed and classified, the cache task information can be stored in parallel, and the data storage efficiency is improved. In general, after a node fails, the buffer task is lost, and the buffer task cannot be recovered until the node is recovered to be normal. Therefore, the invention acquires the lost cache task of the node repaired after the fault from different storage devices, and ensures the integrity and stability of the node of the distributed storage cluster recovered after the fault.
Detailed Description
In order to make the objects, technical solutions and advantages of the present invention more apparent, the following embodiments of the present invention will be described in further detail with reference to the accompanying drawings.
It should be noted that, in the embodiments of the present invention, all the expressions "first" and "second" are used to distinguish two non-identical entities with the same name or non-identical parameters, and it is noted that the "first" and "second" are only used for convenience of expression, and should not be construed as limiting the embodiments of the present invention. Furthermore, the terms "comprise" and "have," and any variations thereof, are intended to cover a non-exclusive inclusion, such as a process, method, system, article, or other step or unit that comprises a list of steps or units.
Based on the above objects, a first aspect of the embodiments of the present invention proposes an embodiment of a cache data management method. Fig. 1 is a schematic diagram of an embodiment of a cache data management method provided by the present invention. As shown in fig. 1, the embodiment of the present invention includes the following steps:
Step S10, monitoring the cache tasks of all nodes in the distributed storage cluster;
Step S20, in response to monitoring that a plurality of nodes have cache tasks, pushing each cache task to a cache queue in sequence, and detecting the integrity of each cache task in the cache queue;
step S30, responding to detection that the caching task is complete, analyzing the complete caching task, classifying based on analysis results, and respectively sending the classified caching tasks to a database or a distributed caching cluster for storage;
and step S40, responding to the failure repair of the node in the nodes corresponding to the complete cache task, and acquiring the cache task lost by the failure repaired node from the database and/or the distributed cache cluster.
According to the embodiment of the invention, the buffer tasks of each node in the distributed storage cluster are monitored, when a plurality of nodes are monitored to have the buffer tasks, each buffer task is sequentially pushed to the buffer queue, then the integrity of each buffer task is detected in the buffer queue, so that the complete buffer task information can be ensured to be screened out, the condition that the buffer task information is lost due to network jitter or a large delay of the distributed storage cluster is effectively relieved, and the complete buffer tasks are analyzed and classified, so that the buffer task information is stored in parallel, and the data storage efficiency is improved. In general, after a node fails, the buffer task is lost, and the buffer task cannot be recovered until the node is recovered to be normal. Therefore, the invention acquires the lost cache task of the node repaired after the fault from different storage devices, and ensures the integrity and stability of the node of the distributed storage cluster recovered after the fault.
In some embodiments, in response to monitoring that a plurality of nodes have cache tasks, pushing each cache task to the cache queue in turn includes, in response to monitoring that a plurality of nodes have cache tasks, assigning an identification number to each cache task and pushing each cache task to the cache queue in turn.
In some embodiments, responding to the repair after the node failure in the nodes corresponding to the complete cache task, acquiring the cache task lost by the repaired node after the failure from the database and/or the distributed cache cluster comprises responding to the repair after the node failure in the nodes corresponding to the complete cache task, and acquiring the cache task lost by the repaired node after the failure from the database and/or the distributed cache cluster according to the identification number.
In the above embodiment, the buffer tasks are transmitted in segments in the network, so that each buffer task entering the buffer queue is assigned an identification number. When the repaired node needs to recover the cache data after the fault occurs, the cache tasks in the database and the distributed cache cluster can be identified through the identification number.
In some embodiments, detecting the integrity of each cache task in the cache queue includes detecting the integrity of each cache task in the cache queue by identifying its data length and information of the head data and tail data.
In some embodiments, the method further comprises, in response to detecting that there is a cache task incomplete, removing the incomplete cache task from the cache queue and notifying a corresponding node to resend the incomplete cache task.
In the above embodiment, the cache queue includes cache integrity detection, specifically, by detecting the data length of each cache task and the tag information of the head data and the tail data of the cache task, it can be known whether the cache task is complete, so as to ensure the reliability of the cache task. When the incomplete cache task is detected, discarding the cache task, and notifying a corresponding node to resend the incomplete cache task.
In some embodiments, responding to detection of complete cache tasks, analyzing the complete cache tasks and classifying based on analysis results, and respectively sending the classified cache tasks to a database or a distributed cache cluster for storage comprises responding to detection of complete cache tasks, analyzing the complete cache tasks according to the size of data blocks, classifying the analyzed cache tasks consisting of large data blocks into a first class, classifying the analyzed cache tasks consisting of small data blocks into a second class, sending the cache tasks belonging to the first class to the distributed cache cluster for temporary storage, and sending the cache tasks belonging to the second class to the database for storage.
In this embodiment, the cache task may be parsed by the cache tag information of the cache task, where the parsed cache task is composed of either large data blocks or small data blocks. Regarding the size of the data block, a threshold range is set in advance for the storage capacity occupied by the data block, and the large data block and the small data block respectively belong to storage capacity values in different ranges.
In some embodiments, the method further comprises, in response to a failure of a node, issuing a failure notification to the distributed storage cluster.
In this embodiment, in order to better manage the cache data of the distributed storage cluster, a fault notification flow is set.
In some embodiments, the method further comprises identifying a cache task to be encrypted in the complete cache task through the cache base library, and performing encryption processing on the cache task to be encrypted.
In this embodiment, the cache base may identify a cache task that needs to be encrypted in the cache tasks, if it is identified that a complete cache task needs to be encrypted, then the cache task is encrypted, and correspondingly, when a node that needs to be repaired after a failure is encrypted, then the encrypted cache task needs to be decrypted, and then the node can acquire the encrypted cache task. The method further ensures the safety of the cache task information and is beneficial to the safety management of the distributed storage clusters.
In a second aspect of the embodiment of the present invention, a cache data management system is also provided. FIG. 2 is a schematic diagram of an embodiment of a cache data management system according to the present invention. As shown in FIG. 2, the cache data management system comprises a cache task monitoring module 10 configured to monitor cache tasks of each node in a distributed storage cluster, a cache task detection module 20 configured to sequentially push each cache task to a cache queue and detect the integrity of each cache task in the cache queue in response to monitoring that a plurality of nodes have cache tasks, a cache task classification module 30 configured to parse the complete cache tasks in response to detecting that the cache tasks are complete and classify the complete cache tasks based on parsing results, and send the classified cache tasks to a database or a distributed cache cluster for storage respectively, and a cache task acquisition module 40 configured to repair after a node in the node corresponding to the complete cache task fails, and acquire the cache task lost by the repaired node from the database and/or the distributed cache cluster.
According to the cache data management system, the cache tasks of all nodes in the distributed storage cluster are monitored, when a plurality of nodes are monitored to have the cache tasks, all the cache tasks are sequentially pushed to the cache queue, then the integrity of all the cache tasks is detected in the cache queue, so that the complete cache task information can be ensured to be screened out, the situation that the cache task information is lost due to network jitter or a large delay of the distributed storage cluster is effectively relieved, and the complete cache tasks are analyzed and classified, so that the cache task information is stored in parallel, and the data storage efficiency is improved. In general, after a node fails, the buffer task is lost, and the buffer task cannot be recovered until the node is recovered to be normal. Therefore, the invention acquires the lost cache task of the node repaired after the fault from different storage devices, and ensures the integrity and stability of the node of the distributed storage cluster recovered after the fault.
In some embodiments, the cache task monitoring module 10 includes a cache queue module configured to allocate an identifier for each cache task and push the buffer queue sequentially in response to monitoring that a plurality of nodes have cache tasks.
In some embodiments, the buffer task obtaining module 40 is configured to obtain, in response to a failure repair of a node in the nodes corresponding to the complete buffer task, a buffer task missing from the failed repair node from the database and/or the distributed buffer cluster according to the identification number.
In some embodiments, the cache task detection module 20 includes an integrity detection module configured to detect the integrity of each cache task by identifying its data length and information of the head data and tail data in the cache queue.
In some embodiments, the cache task classification module 30 is configured to parse the complete cache task according to the size of the data block, and divide the parsed cache task composed of large data blocks into a first class and the parsed cache task composed of small data blocks into a second class in response to detecting that the cache task is complete, send the cache task belonging to the first class to the distributed cache cluster for temporary storage, and send the cache task belonging to the second class to the database for storage.
In some embodiments, the system further comprises a task resending module configured to, in response to detecting that the buffered task is incomplete, reject the incomplete buffered task from the buffer queue and notify the corresponding node to resend the incomplete buffered task.
In some embodiments, the system further comprises a failure notification module configured to issue a failure notification to the distributed storage cluster in response to a failure of a node.
In some embodiments, the system further includes an encryption module configured to identify, by the cache base, a cache task that needs to be encrypted among the complete cache tasks, and perform encryption processing on the cache task that needs to be encrypted.
In a third aspect of the embodiment of the present invention, a computer readable storage medium is provided, and fig. 3 shows a schematic diagram of a computer readable storage medium implementing a cache data management method according to an embodiment of the present invention. As shown in fig. 3, the computer-readable storage medium 3 stores computer program instructions 31, which computer program instructions 31, when executed by a processor, implement the steps of:
in some embodiments, in response to monitoring that a plurality of nodes have cache tasks, pushing each cache task to the cache queue in turn includes, in response to monitoring that a plurality of nodes have cache tasks, assigning an identification number to each cache task and pushing each cache task to the cache queue in turn.
In some embodiments, responding to the repair after the node failure in the nodes corresponding to the complete cache task, acquiring the cache task lost by the repaired node after the failure from the database and/or the distributed cache cluster comprises responding to the repair after the node failure in the nodes corresponding to the complete cache task, and acquiring the cache task lost by the repaired node after the failure from the database and/or the distributed cache cluster according to the identification number.
In some embodiments, detecting the integrity of each cache task in the cache queue includes detecting the integrity of each cache task in the cache queue by identifying its data length and information of the head data and tail data.
In some embodiments, responding to detection of complete cache tasks, analyzing the complete cache tasks and classifying based on analysis results, and respectively sending the classified cache tasks to a database or a distributed cache cluster for storage comprises responding to detection of complete cache tasks, analyzing the complete cache tasks according to the size of data blocks, classifying the analyzed cache tasks consisting of large data blocks into a first class, classifying the analyzed cache tasks consisting of small data blocks into a second class, sending the cache tasks belonging to the first class to the distributed cache cluster for temporary storage, and sending the cache tasks belonging to the second class to the database for storage.
In some embodiments, the steps further include, in response to detecting that there is a cache task incomplete, removing the incomplete cache task from the cache queue and notifying the corresponding node to resend the incomplete cache task.
In some embodiments, the steps further comprise, in response to a failure of a node, issuing a failure notification to the distributed storage cluster.
In some embodiments, the method further comprises the steps of identifying a cache task needing encryption in the complete cache task through a cache base library, and conducting encryption processing on the cache task needing encryption.
It should be appreciated that all of the embodiments, features and advantages set forth above with respect to the cache data management method according to the present invention apply equally to the cache data management system and storage medium according to the present invention, without conflicting therewith.
In a fourth aspect of the embodiments of the present invention, there is also provided a computer device comprising a memory 402 and a processor 401, the memory storing a computer program which, when executed by the processor, implements the method of any of the embodiments described above.
Fig. 4 is a schematic hardware structure of an embodiment of a computer device for performing the cache data management method according to the present invention. Taking the example of a computer device as shown in fig. 4, a processor 401 and a memory 402 are included in the computer device, and input means 403 and output means 404 may also be included. The processor 401, memory 402, input device 403, and output device 404 may be connected by a bus or otherwise, for example in fig. 4. The input device 403 may receive input numeric or character information and generate key signal inputs related to user settings and function control of the cache data management system. The output 404 may include a display device such as a display screen.
The memory 402 is used as a non-volatile computer readable storage medium for storing non-volatile software programs, non-volatile computer executable programs, and modules, such as program instructions/modules corresponding to the cache data management method in the embodiment of the present application. The memory 402 may include a storage program area that may store an operating system, an application program required for at least one function, and a storage data area that may store data created by use of a cache data management method, etc. In addition, memory 402 may include high-speed random access memory, and may also include non-volatile memory, such as at least one magnetic disk storage device, flash memory device, or other non-volatile solid-state storage device. In some embodiments, memory 402 may optionally include memory located remotely from processor 401, which may be connected to the local module via a network. Examples of such networks include, but are not limited to, the internet, intranets, local area networks, mobile communication networks, and combinations thereof.
The processor 401 executes various functional applications of the server and data processing, that is, implements the cache data management method of the above-described method embodiment, by running nonvolatile software programs, instructions, and modules stored in the memory 402.
Finally, it should be noted that the computer-readable storage media (e.g., memory) herein can be either volatile memory or nonvolatile memory, or can include both volatile and nonvolatile memory. By way of example, and not limitation, nonvolatile memory can include Read Only Memory (ROM), programmable ROM (PROM), electrically Programmable ROM (EPROM), electrically Erasable Programmable ROM (EEPROM), or flash memory. Volatile memory can include Random Access Memory (RAM), which acts as external cache memory. By way of example, and not limitation, RAM may be available in a variety of forms such as synchronous RAM (DRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous Link DRAM (SLDRAM), and Direct Rambus RAM (DRRAM). The storage devices of the disclosed aspects are intended to comprise, without being limited to, these and other suitable types of memory.
Those of skill would further appreciate that the various illustrative logical blocks, modules, circuits, and algorithm steps described in connection with the disclosure herein may be implemented as electronic hardware, computer software, or combinations of both. To clearly illustrate this interchangeability of hardware and software, various illustrative components, blocks, modules, circuits, and steps have been described above generally in terms of their functionality. Whether such functionality is implemented as software or hardware depends upon the particular application and design constraints imposed on the overall system. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present disclosure.
The various illustrative logical blocks, modules, and circuits described in connection with the disclosure herein may be implemented or performed with a general purpose processor, a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field Programmable Gate Array (FPGA) or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or any combination thereof designed to perform the functions herein. A general purpose processor may be a microprocessor, but in the alternative, the processor may be any conventional processor, controller, microcontroller, or state machine. A processor may also be implemented as a combination of computing devices, e.g., a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors in conjunction with a DSP and/or any other such configuration.
The foregoing is an exemplary embodiment of the present disclosure, but it should be noted that various changes and modifications could be made herein without departing from the scope of the disclosure as defined by the appended claims. The functions, steps and/or actions of the method claims in accordance with the disclosed embodiments described herein need not be performed in any particular order. Furthermore, although elements of the disclosed embodiments may be described or claimed in the singular, the plural is contemplated unless limitation to the singular is explicitly stated.
It should be understood that as used herein, the singular forms "a", "an", and "the" are intended to include the plural forms as well, unless the context clearly supports the exception. It should also be understood that "and/or" as used herein is meant to include any and all possible combinations of one or more of the associated listed items. The foregoing embodiment of the present invention has been disclosed with reference to the number of embodiments for the purpose of description only, and does not represent the advantages or disadvantages of the embodiments.
It will be appreciated by persons skilled in the art that the foregoing discussion of any embodiment is merely exemplary and is not intended to imply that the scope of the disclosure of embodiments of the invention, including the claims, is limited to such examples, that technical features of the above embodiments or different embodiments may be combined and that many other variations of the different aspects of the embodiments of the invention as described above exist within the spirit of the embodiments of the invention, which are not provided in detail for clarity. Therefore, any omission, modification, equivalent replacement, improvement, etc. of the embodiments should be included in the protection scope of the embodiments of the present invention.