[go: up one dir, main page]

CN113568932B - A cache data management method, system and storage medium - Google Patents

A cache data management method, system and storage medium Download PDF

Info

Publication number
CN113568932B
CN113568932B CN202110656323.6A CN202110656323A CN113568932B CN 113568932 B CN113568932 B CN 113568932B CN 202110656323 A CN202110656323 A CN 202110656323A CN 113568932 B CN113568932 B CN 113568932B
Authority
CN
China
Prior art keywords
cache
task
tasks
complete
response
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
CN202110656323.6A
Other languages
Chinese (zh)
Other versions
CN113568932A (en
Inventor
董俊明
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Inspur Jinan data Technology Co ltd
Original Assignee
Inspur Jinan data Technology Co ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Inspur Jinan data Technology Co ltd filed Critical Inspur Jinan data Technology Co ltd
Priority to CN202110656323.6A priority Critical patent/CN113568932B/en
Publication of CN113568932A publication Critical patent/CN113568932A/en
Application granted granted Critical
Publication of CN113568932B publication Critical patent/CN113568932B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Classifications

    • GPHYSICS
    • G06COMPUTING OR CALCULATING; COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/24Querying
    • G06F16/245Query processing
    • G06F16/2455Query execution
    • G06F16/24552Database cache management
    • GPHYSICS
    • G06COMPUTING OR CALCULATING; COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F11/00Error detection; Error correction; Monitoring
    • G06F11/07Responding to the occurrence of a fault, e.g. fault tolerance
    • G06F11/0703Error or fault processing not based on redundancy, i.e. by taking additional measures to deal with the error or fault not making use of redundancy in operation, in hardware, or in data representation
    • G06F11/0706Error or fault processing not based on redundancy, i.e. by taking additional measures to deal with the error or fault not making use of redundancy in operation, in hardware, or in data representation the processing taking place on a specific hardware platform or in a specific software environment
    • G06F11/0709Error or fault processing not based on redundancy, i.e. by taking additional measures to deal with the error or fault not making use of redundancy in operation, in hardware, or in data representation the processing taking place on a specific hardware platform or in a specific software environment in a distributed system consisting of a plurality of standalone computer nodes, e.g. clusters, client-server systems
    • GPHYSICS
    • G06COMPUTING OR CALCULATING; COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F11/00Error detection; Error correction; Monitoring
    • G06F11/07Responding to the occurrence of a fault, e.g. fault tolerance
    • G06F11/0703Error or fault processing not based on redundancy, i.e. by taking additional measures to deal with the error or fault not making use of redundancy in operation, in hardware, or in data representation
    • G06F11/0793Remedial or corrective actions
    • GPHYSICS
    • G06COMPUTING OR CALCULATING; COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/27Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • General Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Databases & Information Systems (AREA)
  • Quality & Reliability (AREA)
  • Data Mining & Analysis (AREA)
  • Computer Hardware Design (AREA)
  • Computing Systems (AREA)
  • Computational Linguistics (AREA)
  • Memory System Of A Hierarchy Structure (AREA)

Abstract

本发明提供了一种缓存数据管理方法、系统及存储介质,方法包括:对分布式存储集群中各节点的缓存任务进行监听;响应于监听到若干节点有缓存任务,将各缓存任务依次推送到缓存队列,并在缓存队列中检测各缓存任务的完整性;响应于检测到有缓存任务完整,将完整的缓存任务进行解析且基于解析结果进行分类,并将分类后的缓存任务分别发送至数据库或分布式缓存集群中进行存储;响应于该完整的缓存任务对应的节点中有节点发生故障后修复,从数据库和/或分布式缓存集群中获取该发生故障后修复的节点所丢失的缓存任务。本发明保证了缓存任务信息的完整性,提高了数据存储的效率,保证了分布式存储集群故障恢复的稳定性。

The present invention provides a cache data management method, system and storage medium, the method comprising: monitoring the cache tasks of each node in a distributed storage cluster; in response to monitoring that several nodes have cache tasks, pushing each cache task to a cache queue in sequence, and detecting the integrity of each cache task in the cache queue; in response to detecting that a cache task is complete, parsing the complete cache task and classifying it based on the parsing result, and sending the classified cache tasks to a database or a distributed cache cluster for storage; in response to a node corresponding to the complete cache task being repaired after a failure, obtaining the cache tasks lost by the node that has been repaired after the failure from the database and/or the distributed cache cluster. The present invention ensures the integrity of cache task information, improves the efficiency of data storage, and ensures the stability of distributed storage cluster failure recovery.

Description

Cache data management method, system and storage medium
Technical Field
The present invention relates to the field of storage technologies, and in particular, to a method, a system, and a storage medium for managing cache data.
Background
Distributed storage is currently applied to the Internet on a large scale, has good expansibility and lower cost, and is therefore widely used for constructing enterprise-level storage capacity. Distributed refers to a unique type of system architecture consisting of a set of computer nodes that communicate over a network and coordinate work to accomplish a common task.
In the current distributed storage cluster environment, there is a lack of a good way to manage the cached data. If a node fails, part of cache data of the node is lost, and the data cannot be automatically recovered after the node is repaired, and because the scene is more complex in the cluster environment, the data quantity and the nodes needing to be managed are more, the data safety of a user can be seriously affected when a large-scale failure occurs. In addition, the cluster network fluctuation can generate larger time delay to cause data loss, and the data safety and performance of the whole cluster can be seriously influenced.
Disclosure of Invention
Accordingly, the present invention is directed to a method, a system and a storage medium for managing cache data, which are used for solving the problem that the cache data lacks a good management mode in the prior art.
Based on the above object, the present invention provides a cache data management method, comprising the following steps:
monitoring the cache tasks of all nodes in the distributed storage cluster;
In response to monitoring that a plurality of nodes have cache tasks, pushing each cache task to a cache queue in sequence, and detecting the integrity of each cache task in the cache queue;
In response to detecting that the caching task is complete, analyzing the complete caching task, classifying the complete caching task based on an analysis result, and respectively sending the classified caching task to a database or a distributed caching cluster for storage;
And responding to the fault restoration of the nodes corresponding to the complete cache task, and acquiring the cache task lost by the fault restored nodes from the database and/or the distributed cache cluster.
In some embodiments, in response to monitoring that a number of nodes have cache tasks, pushing each cache task to a cache queue in turn includes:
and in response to monitoring that a plurality of nodes have cache tasks, allocating an identification number for each cache task and sequentially pushing the identification numbers to a cache queue.
In some embodiments, in response to a post-failure repair of a node in the nodes corresponding to the complete cache task, obtaining, from the database and/or the distributed cache cluster, a cache task lost by the post-failure repair node includes:
And responding to the fault restoration of the nodes corresponding to the complete cache task, and acquiring the cache task lost by the fault restoration node from the database and/or the distributed cache cluster according to the identification number.
In some embodiments, detecting the integrity of each cache task in the cache queue comprises:
The integrity of each buffer task is detected by identifying the data length of the buffer task and the information of the head data and the tail data in the buffer queue.
In some embodiments, in response to detecting that the cache task is complete, resolving the complete cache task and classifying based on the resolution result, and sending the classified cache task to the database or the distributed cache cluster for storage respectively includes:
In response to detecting that the buffer task is complete, analyzing the complete buffer task according to the size of the data block, classifying the analyzed buffer task consisting of large data blocks into a first class, and classifying the analyzed buffer task consisting of small data blocks into a second class;
And sending the cache tasks belonging to the first class to the distributed cache cluster for temporary storage, and sending the cache tasks belonging to the second class to the database for storage.
In some embodiments, the method further comprises:
And in response to detecting that the buffer task is incomplete, removing the incomplete buffer task from the buffer queue, and informing a corresponding node to resend the incomplete buffer task.
In some embodiments, the method further comprises:
In response to a node failure, a failure notification is sent to the distributed storage cluster.
In some embodiments, the method further comprises:
and identifying the buffer task needing to be encrypted in the complete buffer task through the buffer basic library, and carrying out encryption processing on the buffer task needing to be encrypted.
In another aspect of the present invention, there is also provided a cache data management system, including:
The cache task monitoring module is configured to monitor the cache task of each node in the distributed storage cluster;
The buffer task detection module is configured to respond to the fact that a plurality of nodes have buffer tasks, push each buffer task to a buffer queue in sequence, and detect the integrity of each buffer task in the buffer queue;
The buffer task classifying module is configured to analyze the complete buffer task and classify the buffer task based on the analysis result in response to detecting that the buffer task is complete, and respectively send the classified buffer task to a database or a distributed buffer cluster for storage, and
And the cache task acquisition module is configured to respond to the failure repair of the node in the nodes corresponding to the complete cache task, and acquire the cache task lost by the failure repaired node from the database and/or the distributed cache cluster.
In yet another aspect of the present invention, there is also provided a computer readable storage medium storing computer program instructions which, when executed, implement any of the methods described above.
In yet another aspect of the present invention, there is also provided a computer device comprising a memory and a processor, the memory storing a computer program which, when executed by the processor, performs any of the methods described above.
The invention has at least the following beneficial technical effects:
According to the method, the system and the device, the cache tasks of all nodes in the distributed storage cluster are monitored, when a plurality of nodes are monitored to have the cache tasks, all the cache tasks are sequentially pushed to the cache queue, then the integrity of all the cache tasks is detected in the cache queue, so that complete cache task information can be screened out, the situation that the cache task information is lost due to network jitter or large delay of the distributed storage cluster is effectively relieved, the complete cache tasks are analyzed and classified, the cache task information can be stored in parallel, and the data storage efficiency is improved. In general, after a node fails, the buffer task is lost, and the buffer task cannot be recovered until the node is recovered to be normal. Therefore, the invention acquires the lost cache task of the node repaired after the fault from different storage devices, and ensures the integrity and stability of the node of the distributed storage cluster recovered after the fault.
Drawings
In order to more clearly illustrate the embodiments of the invention or the technical solutions in the prior art, the drawings that are necessary for the description of the embodiments or the prior art will be briefly described, it being obvious that the drawings in the following description are only some embodiments of the invention and that other embodiments may be obtained according to these drawings without inventive effort for a person skilled in the art.
FIG. 1 is a schematic diagram of a method for managing cache data according to an embodiment of the present invention;
FIG. 2 is a schematic diagram of a cache data management system/apparatus according to an embodiment of the present invention;
FIG. 3 is a schematic diagram of a computer readable storage medium implementing a method for managing cache data according to an embodiment of the present invention;
fig. 4 is a schematic hardware structure of a computer device for performing a cache data management method according to an embodiment of the present invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the present invention more apparent, the following embodiments of the present invention will be described in further detail with reference to the accompanying drawings.
It should be noted that, in the embodiments of the present invention, all the expressions "first" and "second" are used to distinguish two non-identical entities with the same name or non-identical parameters, and it is noted that the "first" and "second" are only used for convenience of expression, and should not be construed as limiting the embodiments of the present invention. Furthermore, the terms "comprise" and "have," and any variations thereof, are intended to cover a non-exclusive inclusion, such as a process, method, system, article, or other step or unit that comprises a list of steps or units.
Based on the above objects, a first aspect of the embodiments of the present invention proposes an embodiment of a cache data management method. Fig. 1 is a schematic diagram of an embodiment of a cache data management method provided by the present invention. As shown in fig. 1, the embodiment of the present invention includes the following steps:
Step S10, monitoring the cache tasks of all nodes in the distributed storage cluster;
Step S20, in response to monitoring that a plurality of nodes have cache tasks, pushing each cache task to a cache queue in sequence, and detecting the integrity of each cache task in the cache queue;
step S30, responding to detection that the caching task is complete, analyzing the complete caching task, classifying based on analysis results, and respectively sending the classified caching tasks to a database or a distributed caching cluster for storage;
and step S40, responding to the failure repair of the node in the nodes corresponding to the complete cache task, and acquiring the cache task lost by the failure repaired node from the database and/or the distributed cache cluster.
According to the embodiment of the invention, the buffer tasks of each node in the distributed storage cluster are monitored, when a plurality of nodes are monitored to have the buffer tasks, each buffer task is sequentially pushed to the buffer queue, then the integrity of each buffer task is detected in the buffer queue, so that the complete buffer task information can be ensured to be screened out, the condition that the buffer task information is lost due to network jitter or a large delay of the distributed storage cluster is effectively relieved, and the complete buffer tasks are analyzed and classified, so that the buffer task information is stored in parallel, and the data storage efficiency is improved. In general, after a node fails, the buffer task is lost, and the buffer task cannot be recovered until the node is recovered to be normal. Therefore, the invention acquires the lost cache task of the node repaired after the fault from different storage devices, and ensures the integrity and stability of the node of the distributed storage cluster recovered after the fault.
In some embodiments, in response to monitoring that a plurality of nodes have cache tasks, pushing each cache task to the cache queue in turn includes, in response to monitoring that a plurality of nodes have cache tasks, assigning an identification number to each cache task and pushing each cache task to the cache queue in turn.
In some embodiments, responding to the repair after the node failure in the nodes corresponding to the complete cache task, acquiring the cache task lost by the repaired node after the failure from the database and/or the distributed cache cluster comprises responding to the repair after the node failure in the nodes corresponding to the complete cache task, and acquiring the cache task lost by the repaired node after the failure from the database and/or the distributed cache cluster according to the identification number.
In the above embodiment, the buffer tasks are transmitted in segments in the network, so that each buffer task entering the buffer queue is assigned an identification number. When the repaired node needs to recover the cache data after the fault occurs, the cache tasks in the database and the distributed cache cluster can be identified through the identification number.
In some embodiments, detecting the integrity of each cache task in the cache queue includes detecting the integrity of each cache task in the cache queue by identifying its data length and information of the head data and tail data.
In some embodiments, the method further comprises, in response to detecting that there is a cache task incomplete, removing the incomplete cache task from the cache queue and notifying a corresponding node to resend the incomplete cache task.
In the above embodiment, the cache queue includes cache integrity detection, specifically, by detecting the data length of each cache task and the tag information of the head data and the tail data of the cache task, it can be known whether the cache task is complete, so as to ensure the reliability of the cache task. When the incomplete cache task is detected, discarding the cache task, and notifying a corresponding node to resend the incomplete cache task.
In some embodiments, responding to detection of complete cache tasks, analyzing the complete cache tasks and classifying based on analysis results, and respectively sending the classified cache tasks to a database or a distributed cache cluster for storage comprises responding to detection of complete cache tasks, analyzing the complete cache tasks according to the size of data blocks, classifying the analyzed cache tasks consisting of large data blocks into a first class, classifying the analyzed cache tasks consisting of small data blocks into a second class, sending the cache tasks belonging to the first class to the distributed cache cluster for temporary storage, and sending the cache tasks belonging to the second class to the database for storage.
In this embodiment, the cache task may be parsed by the cache tag information of the cache task, where the parsed cache task is composed of either large data blocks or small data blocks. Regarding the size of the data block, a threshold range is set in advance for the storage capacity occupied by the data block, and the large data block and the small data block respectively belong to storage capacity values in different ranges.
In some embodiments, the method further comprises, in response to a failure of a node, issuing a failure notification to the distributed storage cluster.
In this embodiment, in order to better manage the cache data of the distributed storage cluster, a fault notification flow is set.
In some embodiments, the method further comprises identifying a cache task to be encrypted in the complete cache task through the cache base library, and performing encryption processing on the cache task to be encrypted.
In this embodiment, the cache base may identify a cache task that needs to be encrypted in the cache tasks, if it is identified that a complete cache task needs to be encrypted, then the cache task is encrypted, and correspondingly, when a node that needs to be repaired after a failure is encrypted, then the encrypted cache task needs to be decrypted, and then the node can acquire the encrypted cache task. The method further ensures the safety of the cache task information and is beneficial to the safety management of the distributed storage clusters.
In a second aspect of the embodiment of the present invention, a cache data management system is also provided. FIG. 2 is a schematic diagram of an embodiment of a cache data management system according to the present invention. As shown in FIG. 2, the cache data management system comprises a cache task monitoring module 10 configured to monitor cache tasks of each node in a distributed storage cluster, a cache task detection module 20 configured to sequentially push each cache task to a cache queue and detect the integrity of each cache task in the cache queue in response to monitoring that a plurality of nodes have cache tasks, a cache task classification module 30 configured to parse the complete cache tasks in response to detecting that the cache tasks are complete and classify the complete cache tasks based on parsing results, and send the classified cache tasks to a database or a distributed cache cluster for storage respectively, and a cache task acquisition module 40 configured to repair after a node in the node corresponding to the complete cache task fails, and acquire the cache task lost by the repaired node from the database and/or the distributed cache cluster.
According to the cache data management system, the cache tasks of all nodes in the distributed storage cluster are monitored, when a plurality of nodes are monitored to have the cache tasks, all the cache tasks are sequentially pushed to the cache queue, then the integrity of all the cache tasks is detected in the cache queue, so that the complete cache task information can be ensured to be screened out, the situation that the cache task information is lost due to network jitter or a large delay of the distributed storage cluster is effectively relieved, and the complete cache tasks are analyzed and classified, so that the cache task information is stored in parallel, and the data storage efficiency is improved. In general, after a node fails, the buffer task is lost, and the buffer task cannot be recovered until the node is recovered to be normal. Therefore, the invention acquires the lost cache task of the node repaired after the fault from different storage devices, and ensures the integrity and stability of the node of the distributed storage cluster recovered after the fault.
In some embodiments, the cache task monitoring module 10 includes a cache queue module configured to allocate an identifier for each cache task and push the buffer queue sequentially in response to monitoring that a plurality of nodes have cache tasks.
In some embodiments, the buffer task obtaining module 40 is configured to obtain, in response to a failure repair of a node in the nodes corresponding to the complete buffer task, a buffer task missing from the failed repair node from the database and/or the distributed buffer cluster according to the identification number.
In some embodiments, the cache task detection module 20 includes an integrity detection module configured to detect the integrity of each cache task by identifying its data length and information of the head data and tail data in the cache queue.
In some embodiments, the cache task classification module 30 is configured to parse the complete cache task according to the size of the data block, and divide the parsed cache task composed of large data blocks into a first class and the parsed cache task composed of small data blocks into a second class in response to detecting that the cache task is complete, send the cache task belonging to the first class to the distributed cache cluster for temporary storage, and send the cache task belonging to the second class to the database for storage.
In some embodiments, the system further comprises a task resending module configured to, in response to detecting that the buffered task is incomplete, reject the incomplete buffered task from the buffer queue and notify the corresponding node to resend the incomplete buffered task.
In some embodiments, the system further comprises a failure notification module configured to issue a failure notification to the distributed storage cluster in response to a failure of a node.
In some embodiments, the system further includes an encryption module configured to identify, by the cache base, a cache task that needs to be encrypted among the complete cache tasks, and perform encryption processing on the cache task that needs to be encrypted.
In a third aspect of the embodiment of the present invention, a computer readable storage medium is provided, and fig. 3 shows a schematic diagram of a computer readable storage medium implementing a cache data management method according to an embodiment of the present invention. As shown in fig. 3, the computer-readable storage medium 3 stores computer program instructions 31, which computer program instructions 31, when executed by a processor, implement the steps of:
in some embodiments, in response to monitoring that a plurality of nodes have cache tasks, pushing each cache task to the cache queue in turn includes, in response to monitoring that a plurality of nodes have cache tasks, assigning an identification number to each cache task and pushing each cache task to the cache queue in turn.
In some embodiments, responding to the repair after the node failure in the nodes corresponding to the complete cache task, acquiring the cache task lost by the repaired node after the failure from the database and/or the distributed cache cluster comprises responding to the repair after the node failure in the nodes corresponding to the complete cache task, and acquiring the cache task lost by the repaired node after the failure from the database and/or the distributed cache cluster according to the identification number.
In some embodiments, detecting the integrity of each cache task in the cache queue includes detecting the integrity of each cache task in the cache queue by identifying its data length and information of the head data and tail data.
In some embodiments, responding to detection of complete cache tasks, analyzing the complete cache tasks and classifying based on analysis results, and respectively sending the classified cache tasks to a database or a distributed cache cluster for storage comprises responding to detection of complete cache tasks, analyzing the complete cache tasks according to the size of data blocks, classifying the analyzed cache tasks consisting of large data blocks into a first class, classifying the analyzed cache tasks consisting of small data blocks into a second class, sending the cache tasks belonging to the first class to the distributed cache cluster for temporary storage, and sending the cache tasks belonging to the second class to the database for storage.
In some embodiments, the steps further include, in response to detecting that there is a cache task incomplete, removing the incomplete cache task from the cache queue and notifying the corresponding node to resend the incomplete cache task.
In some embodiments, the steps further comprise, in response to a failure of a node, issuing a failure notification to the distributed storage cluster.
In some embodiments, the method further comprises the steps of identifying a cache task needing encryption in the complete cache task through a cache base library, and conducting encryption processing on the cache task needing encryption.
It should be appreciated that all of the embodiments, features and advantages set forth above with respect to the cache data management method according to the present invention apply equally to the cache data management system and storage medium according to the present invention, without conflicting therewith.
In a fourth aspect of the embodiments of the present invention, there is also provided a computer device comprising a memory 402 and a processor 401, the memory storing a computer program which, when executed by the processor, implements the method of any of the embodiments described above.
Fig. 4 is a schematic hardware structure of an embodiment of a computer device for performing the cache data management method according to the present invention. Taking the example of a computer device as shown in fig. 4, a processor 401 and a memory 402 are included in the computer device, and input means 403 and output means 404 may also be included. The processor 401, memory 402, input device 403, and output device 404 may be connected by a bus or otherwise, for example in fig. 4. The input device 403 may receive input numeric or character information and generate key signal inputs related to user settings and function control of the cache data management system. The output 404 may include a display device such as a display screen.
The memory 402 is used as a non-volatile computer readable storage medium for storing non-volatile software programs, non-volatile computer executable programs, and modules, such as program instructions/modules corresponding to the cache data management method in the embodiment of the present application. The memory 402 may include a storage program area that may store an operating system, an application program required for at least one function, and a storage data area that may store data created by use of a cache data management method, etc. In addition, memory 402 may include high-speed random access memory, and may also include non-volatile memory, such as at least one magnetic disk storage device, flash memory device, or other non-volatile solid-state storage device. In some embodiments, memory 402 may optionally include memory located remotely from processor 401, which may be connected to the local module via a network. Examples of such networks include, but are not limited to, the internet, intranets, local area networks, mobile communication networks, and combinations thereof.
The processor 401 executes various functional applications of the server and data processing, that is, implements the cache data management method of the above-described method embodiment, by running nonvolatile software programs, instructions, and modules stored in the memory 402.
Finally, it should be noted that the computer-readable storage media (e.g., memory) herein can be either volatile memory or nonvolatile memory, or can include both volatile and nonvolatile memory. By way of example, and not limitation, nonvolatile memory can include Read Only Memory (ROM), programmable ROM (PROM), electrically Programmable ROM (EPROM), electrically Erasable Programmable ROM (EEPROM), or flash memory. Volatile memory can include Random Access Memory (RAM), which acts as external cache memory. By way of example, and not limitation, RAM may be available in a variety of forms such as synchronous RAM (DRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous Link DRAM (SLDRAM), and Direct Rambus RAM (DRRAM). The storage devices of the disclosed aspects are intended to comprise, without being limited to, these and other suitable types of memory.
Those of skill would further appreciate that the various illustrative logical blocks, modules, circuits, and algorithm steps described in connection with the disclosure herein may be implemented as electronic hardware, computer software, or combinations of both. To clearly illustrate this interchangeability of hardware and software, various illustrative components, blocks, modules, circuits, and steps have been described above generally in terms of their functionality. Whether such functionality is implemented as software or hardware depends upon the particular application and design constraints imposed on the overall system. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present disclosure.
The various illustrative logical blocks, modules, and circuits described in connection with the disclosure herein may be implemented or performed with a general purpose processor, a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field Programmable Gate Array (FPGA) or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or any combination thereof designed to perform the functions herein. A general purpose processor may be a microprocessor, but in the alternative, the processor may be any conventional processor, controller, microcontroller, or state machine. A processor may also be implemented as a combination of computing devices, e.g., a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors in conjunction with a DSP and/or any other such configuration.
The foregoing is an exemplary embodiment of the present disclosure, but it should be noted that various changes and modifications could be made herein without departing from the scope of the disclosure as defined by the appended claims. The functions, steps and/or actions of the method claims in accordance with the disclosed embodiments described herein need not be performed in any particular order. Furthermore, although elements of the disclosed embodiments may be described or claimed in the singular, the plural is contemplated unless limitation to the singular is explicitly stated.
It should be understood that as used herein, the singular forms "a", "an", and "the" are intended to include the plural forms as well, unless the context clearly supports the exception. It should also be understood that "and/or" as used herein is meant to include any and all possible combinations of one or more of the associated listed items. The foregoing embodiment of the present invention has been disclosed with reference to the number of embodiments for the purpose of description only, and does not represent the advantages or disadvantages of the embodiments.
It will be appreciated by persons skilled in the art that the foregoing discussion of any embodiment is merely exemplary and is not intended to imply that the scope of the disclosure of embodiments of the invention, including the claims, is limited to such examples, that technical features of the above embodiments or different embodiments may be combined and that many other variations of the different aspects of the embodiments of the invention as described above exist within the spirit of the embodiments of the invention, which are not provided in detail for clarity. Therefore, any omission, modification, equivalent replacement, improvement, etc. of the embodiments should be included in the protection scope of the embodiments of the present invention.

Claims (7)

1.一种缓存数据管理方法,其特征在于,包括以下步骤:1. A cache data management method, characterized in that it comprises the following steps: 对分布式存储集群中各节点的缓存任务进行监听,其中,所述缓存任务解析后由大数据块或小数据块组成;Monitor the cache tasks of each node in the distributed storage cluster, wherein the cache tasks are composed of large data blocks or small data blocks after parsing; 响应于监听到若干节点有缓存任务,将各缓存任务依次推送到缓存队列,并在所述缓存队列中检测各缓存任务的完整性;In response to monitoring that several nodes have cache tasks, push each cache task to a cache queue in sequence, and detect the integrity of each cache task in the cache queue; 响应于检测到有缓存任务完整,将完整的缓存任务进行解析且基于解析结果进行分类,并将分类后的缓存任务分别发送至数据库或分布式缓存集群中进行存储;In response to detecting that a cache task is complete, the complete cache task is parsed and classified based on the parsing result, and the classified cache tasks are respectively sent to a database or a distributed cache cluster for storage; 响应于所述完整的缓存任务对应的节点中有节点发生故障后修复,从所述数据库和/或分布式缓存集群中获取所述发生故障后修复的节点所丢失的缓存任务;In response to a node among the nodes corresponding to the complete cache task being repaired after a failure, obtaining, from the database and/or the distributed cache cluster, cache tasks lost by the node that has been repaired after the failure; 其中,在所述缓存队列中检测各缓存任务的完整性包括:在所述缓存队列中通过对各缓存任务的数据长度以及头部数据和尾部数据的信息进行识别以检测其完整性;响应于检测到有缓存任务不完整,将不完整的缓存任务从所述缓存队列中剔除,并通知相应的节点对所述不完整的缓存任务进行重新发送;Wherein, detecting the integrity of each cache task in the cache queue includes: detecting the integrity of each cache task by identifying the data length and the information of the header data and the tail data in the cache queue; in response to detecting that a cache task is incomplete, removing the incomplete cache task from the cache queue, and notifying the corresponding node to resend the incomplete cache task; 响应于检测到有缓存任务完整,将完整的缓存任务进行解析且基于解析结果进行分类,并将分类后的缓存任务分别发送至数据库或分布式缓存集群中进行存储包括:响应于检测到有缓存任务完整,对完整的缓存任务按照数据块大小进行解析,并将解析出的由大数据块组成的缓存任务分为第一类,以及将解析出的由小数据块组成的缓存任务分为第二类;将属于第一类的缓存任务发送至所述分布式缓存集群中暂时存储,并将属于第二类的缓存任务发送至所述数据库中存储。In response to detecting that a cache task is complete, the complete cache task is parsed and classified based on the parsing results, and the classified cache tasks are sent to the database or distributed cache cluster for storage respectively, including: in response to detecting that a cache task is complete, the complete cache task is parsed according to the data block size, and the parsed cache tasks composed of large data blocks are classified into a first category, and the parsed cache tasks composed of small data blocks are classified into a second category; the cache tasks belonging to the first category are sent to the distributed cache cluster for temporary storage, and the cache tasks belonging to the second category are sent to the database for storage. 2.根据权利要求1所述的方法,其特征在于,响应于监听到若干节点有缓存任务,将各缓存任务依次推送到缓存队列包括:2. The method according to claim 1, characterized in that, in response to monitoring that a plurality of nodes have cache tasks, pushing each cache task to the cache queue in sequence comprises: 响应于监听到若干节点有缓存任务,为各缓存任务分配识别号并依次推送到缓存队列。In response to monitoring that several nodes have cache tasks, an identification number is assigned to each cache task and pushed to the cache queue in sequence. 3.根据权利要求2所述的方法,其特征在于,响应于所述完整的缓存任务对应的节点中有节点发生故障后修复,从所述数据库和/或分布式缓存集群中获取所述发生故障后修复的节点所丢失的缓存任务包括:3. The method according to claim 2, characterized in that, in response to a node among the nodes corresponding to the complete cache task being repaired after a failure, obtaining from the database and/or the distributed cache cluster the cache task lost by the node that has been repaired after the failure comprises: 响应于所述完整的缓存任务对应的节点中有节点发生故障后修复,根据识别号从所述数据库和/或分布式缓存集群中获取所述发生故障后修复的节点所丢失的缓存任务。In response to a node among the nodes corresponding to the complete cache task being repaired after a failure, the cache task lost by the node that has been repaired after the failure is obtained from the database and/or the distributed cache cluster according to the identification number. 4.根据权利要求1所述的方法,其特征在于,还包括:4. The method according to claim 1, further comprising: 响应于有节点发生故障,向所述分布式存储集群发出故障通知。In response to a node failure, a failure notification is sent to the distributed storage cluster. 5.根据权利要求1所述的方法,其特征在于,还包括:5. The method according to claim 1, further comprising: 通过缓存基础库识别所述完整的缓存任务中需要加密的缓存任务,并对所述需要加密的缓存任务进行加密处理。The cache tasks that need to be encrypted in the complete cache tasks are identified through the cache basic library, and the cache tasks that need to be encrypted are encrypted. 6.一种缓存数据管理系统,其特征在于,包括:6. A cache data management system, comprising: 缓存任务监听模块,配置用于对分布式存储集群中各节点的缓存任务进行监听,其中,所述缓存任务解析后由大数据块或小数据块组成;A cache task monitoring module, configured to monitor cache tasks of each node in the distributed storage cluster, wherein the cache tasks are composed of large data blocks or small data blocks after parsing; 缓存任务检测模块,配置用于响应于监听到若干节点有缓存任务,将各缓存任务依次推送到缓存队列,并在所述缓存队列中检测各缓存任务的完整性;A cache task detection module, configured to push each cache task to a cache queue in sequence in response to monitoring that a plurality of nodes have cache tasks, and detect the integrity of each cache task in the cache queue; 缓存任务分类模块,配置用于响应于检测到有缓存任务完整,将完整的缓存任务进行解析且基于解析结果进行分类,并将分类后的缓存任务分别发送至数据库或分布式缓存集群中进行存储;以及a cache task classification module configured to, in response to detecting that a cache task is complete, parse the complete cache task and classify it based on the parsing result, and send the classified cache tasks to a database or a distributed cache cluster for storage; and 缓存任务获取模块,配置用于响应于所述完整的缓存任务对应的节点中有节点发生故障后修复,从所述数据库和/或分布式缓存集群中获取所述发生故障后修复的节点所丢失的缓存任务;A cache task acquisition module, configured to, in response to a node among the nodes corresponding to the complete cache task being repaired after a failure, acquire from the database and/or the distributed cache cluster the cache tasks lost by the node that has been repaired after the failure; 其中,所述缓存任务检测模块还配置用于在所述缓存队列中通过对各缓存任务的数据长度以及头部数据和尾部数据的信息进行识别以检测其完整性;响应于检测到有缓存任务不完整,将不完整的缓存任务从所述缓存队列中剔除,并通知相应的节点对所述不完整的缓存任务进行重新发送;The cache task detection module is further configured to detect the integrity of each cache task in the cache queue by identifying the data length and the information of the header data and the tail data; in response to detecting that a cache task is incomplete, the incomplete cache task is removed from the cache queue, and the corresponding node is notified to resend the incomplete cache task; 所述缓存任务分类模块还配置用于响应于检测到有缓存任务完整,将完整的缓存任务进行解析且基于解析结果进行分类,并将分类后的缓存任务分别发送至数据库或分布式缓存集群中进行存储包括:响应于检测到有缓存任务完整,对完整的缓存任务按照数据块大小进行解析,并将解析出的由大数据块组成的缓存任务分为第一类,以及将解析出的由小数据块组成的缓存任务分为第二类;将属于第一类的缓存任务发送至所述分布式缓存集群中暂时存储,并将属于第二类的缓存任务发送至所述数据库中存储。The cache task classification module is also configured to parse the complete cache task in response to detecting that a cache task is complete and classify the tasks based on the parsing results, and send the classified cache tasks to the database or distributed cache cluster for storage, including: in response to detecting that a cache task is complete, parsing the complete cache task according to the data block size, and classifying the parsed cache tasks composed of large data blocks into a first category, and classifying the parsed cache tasks composed of small data blocks into a second category; sending the cache tasks belonging to the first category to the distributed cache cluster for temporary storage, and sending the cache tasks belonging to the second category to the database for storage. 7.一种计算机可读存储介质,其特征在于,存储有计算机程序指令,所述计算机程序指令被处理器执行时实现如权利要求1-5任意一项所述的方法。7. A computer-readable storage medium, characterized in that it stores computer program instructions, and when the computer program instructions are executed by a processor, they implement the method according to any one of claims 1 to 5.
CN202110656323.6A 2021-06-11 2021-06-11 A cache data management method, system and storage medium Active CN113568932B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202110656323.6A CN113568932B (en) 2021-06-11 2021-06-11 A cache data management method, system and storage medium

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202110656323.6A CN113568932B (en) 2021-06-11 2021-06-11 A cache data management method, system and storage medium

Publications (2)

Publication Number Publication Date
CN113568932A CN113568932A (en) 2021-10-29
CN113568932B true CN113568932B (en) 2025-03-14

Family

ID=78161999

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202110656323.6A Active CN113568932B (en) 2021-06-11 2021-06-11 A cache data management method, system and storage medium

Country Status (1)

Country Link
CN (1) CN113568932B (en)

Families Citing this family (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN115190125B (en) * 2022-06-27 2024-09-20 京东科技控股股份有限公司 Monitoring method and device for cache cluster
CN116910076A (en) * 2023-07-21 2023-10-20 济南浪潮数据技术有限公司 Resource caching method, system, storage medium and equipment

Family Cites Families (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US7680879B2 (en) * 1996-07-18 2010-03-16 Computer Associates Think, Inc. Method and apparatus for maintaining data integrity across distributed computer systems
US8108623B2 (en) * 2008-10-26 2012-01-31 Microsoft Corporation Poll based cache event notifications in a distributed cache
US10228995B2 (en) * 2016-07-28 2019-03-12 Hewlett Packard Enterprise Development Lp Last writers of datasets in storage array errors
CN107707662A (en) * 2017-10-16 2018-02-16 大唐网络有限公司 A kind of distributed caching method based on node, device and storage medium
CN108363764A (en) * 2018-02-05 2018-08-03 山东地主网络科技创新有限公司 A kind of distributed caching management system and method
CN110196759B (en) * 2018-06-20 2022-12-06 腾讯科技(深圳)有限公司 Distributed transaction processing method and device, storage medium and electronic device
CN110032558B (en) * 2019-04-12 2020-12-22 重庆天蓬网络有限公司 Data synchronization method, device and system and storage medium
CN112486948B (en) * 2020-11-25 2022-05-13 福建省数字福建云计算运营有限公司 Real-time data processing method

Also Published As

Publication number Publication date
CN113568932A (en) 2021-10-29

Similar Documents

Publication Publication Date Title
US12124326B2 (en) Automatic correlation of dynamic system events within computing devices
CN108881294B (en) Method and device for generating IP profile of attack source based on network attack behavior
CN113568932B (en) A cache data management method, system and storage medium
CN110995468A (en) System fault processing method, device, equipment and storage medium of system to be analyzed
US9210057B2 (en) Cross-cutting event correlation
CN112866319B (en) Log data processing method, system and storage medium
CN105656699A (en) Alarm management method and system for content distribution network
CN109918196B (en) System resource allocation method, device, computer equipment and storage medium
CN113938306A (en) Credible authentication method and system based on data cleaning rule
CN111858588A (en) Distributed application index service platform and data processing method
CN119512855A (en) Abnormal data monitoring method, device, equipment and medium
CN113806204B (en) Method, device, system and storage medium for evaluating message segment correlation
WO2023014391A1 (en) Non-deterministic finite automata tree structure application apparatus and method
CN107612755A (en) The management method and its device of a kind of cloud resource
CN112583825A (en) Method and device for detecting abnormality of industrial system
WO2021139280A1 (en) Log processing method and apparatus based on zookeeper, and computer device and storage medium
CN114726756B (en) Configuration monitoring method and device
CN117714325A (en) Network monitoring method, device, electronic equipment and storage medium for server cluster
CN114003313B (en) Cluster management method, system, storage medium and device
CN105099781A (en) IT infrastructure patrolling method and system
CN113392079B (en) A distributed storage cluster log storage optimization method, system and terminal
CN113852487B (en) Methods and systems for facilitating self-healing networks
CN103944777A (en) Distributed monitoring system information processing method and system
CN115934487A (en) Log monitoring and warning method, device, computer equipment and storage medium
JP5686001B2 (en) Information processing apparatus, message isolation method, and message isolation program

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant