CN118519820B - Cloud platform-based data storage abnormality detection method and equipment - Google Patents
Cloud platform-based data storage abnormality detection method and equipment Download PDFInfo
- Publication number
- CN118519820B CN118519820B CN202410995977.5A CN202410995977A CN118519820B CN 118519820 B CN118519820 B CN 118519820B CN 202410995977 A CN202410995977 A CN 202410995977A CN 118519820 B CN118519820 B CN 118519820B
- Authority
- CN
- China
- Prior art keywords
- data
- storage
- abnormality detection
- request information
- data storage
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
- G06F11/0703—Error or fault processing not based on redundancy, i.e. by taking additional measures to deal with the error or fault not making use of redundancy in operation, in hardware, or in data representation
- G06F11/0706—Error or fault processing not based on redundancy, i.e. by taking additional measures to deal with the error or fault not making use of redundancy in operation, in hardware, or in data representation the processing taking place on a specific hardware platform or in a specific software environment
- G06F11/0727—Error or fault processing not based on redundancy, i.e. by taking additional measures to deal with the error or fault not making use of redundancy in operation, in hardware, or in data representation the processing taking place on a specific hardware platform or in a specific software environment in a storage system, e.g. in a DASD or network based storage system
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
- G06F11/0703—Error or fault processing not based on redundancy, i.e. by taking additional measures to deal with the error or fault not making use of redundancy in operation, in hardware, or in data representation
- G06F11/079—Root cause analysis, i.e. error or fault diagnosis
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
- G06F11/0703—Error or fault processing not based on redundancy, i.e. by taking additional measures to deal with the error or fault not making use of redundancy in operation, in hardware, or in data representation
- G06F11/0793—Remedial or corrective actions
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
- G06F3/0614—Improving the reliability of storage systems
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
- G06F3/0614—Improving the reliability of storage systems
- G06F3/0619—Improving the reliability of storage systems in relation to data integrity, e.g. data losses, bit errors
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
- G06F3/0638—Organizing or formatting or addressing of data
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/50—Network services
- H04L67/56—Provisioning of proxy services
- H04L67/568—Storing data temporarily at an intermediate stage, e.g. caching
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Physics & Mathematics (AREA)
- Human Computer Interaction (AREA)
- Quality & Reliability (AREA)
- Signal Processing (AREA)
- Computer Networks & Wireless Communication (AREA)
- Computer Security & Cryptography (AREA)
- Health & Medical Sciences (AREA)
- Biomedical Technology (AREA)
- Alarm Systems (AREA)
- Debugging And Monitoring (AREA)
Abstract
The invention provides a cloud platform-based data storage abnormality detection method and equipment, which relate to the technical field of data processing and are characterized in that received data storage writing request information is analyzed and judged by acquiring the data storage writing request information, if the data storage writing request information is a periodic writing request, storage abnormality detection is carried out according to a periodic abnormality detection channel, a first abnormality detection result is acquired, if the data storage writing request information is an aperiodic writing request, storage abnormality detection is carried out according to an aperiodic abnormality detection channel, a second abnormality detection result is acquired, and data to be written of the data storage writing request information is written into a memory according to the first abnormality detection result or the second abnormality detection result. The method solves the technical problem that the prior art has insufficient data security and reliability caused by direct writing of data into a storage system. The technical effects of improving the safety and the reliability of data and realizing efficient data processing and storage are achieved.
Description
Technical Field
The invention relates to the technical field of data processing, in particular to a cloud platform-based data storage abnormality detection method and equipment.
Background
In current data storage technology, data is often written directly into memory, lacking the necessary intermediate processing links. This direct write approach, while simplifying the operational flow, presents a significant data security risk. Due to the lack of real-time analysis and monitoring of data content, potentially malicious data or anomaly information may be stored without detection, posing a threat to system security and user privacy. In addition, direct writing may also result in inefficient data processing because once the data is written, additional reading and processing is required if subsequent data cleaning, verification, or conversion is required, adding to the overall burden of the system. Therefore, the direct writing method in the prior art has obvious defects in terms of data security and data reliability.
In the prior art, data is directly written into a storage system, abnormal data are mixed, and the technical problems of insufficient data safety and reliability are caused.
Disclosure of Invention
The application provides a cloud platform-based data storage abnormality detection method and equipment, which are used for solving the technical problems that in the prior art, data are directly written into a storage system, abnormal data are mixed, and the data safety and reliability are insufficient.
In view of the above problems, the application provides a cloud platform-based data storage abnormality detection method and equipment.
The first aspect of the application provides a cloud platform-based data storage abnormality detection method, which comprises the following steps: acquiring data storage write-in request information; setting a storage transfer module, wherein the storage transfer module is arranged in a cloud platform, a storage abnormality detection model is embedded in the storage transfer module, and the storage abnormality detection model comprises a periodic abnormality detection channel and an aperiodic abnormality detection channel; analyzing the received data storage writing request information through the storage transfer module, and judging whether the data storage writing request information is a periodic writing request or not; if the data storage writing request information is a periodic writing request, carrying out storage abnormality detection according to the periodic abnormality detection channel to obtain a first abnormality detection result; if the data storage writing request information is an aperiodic writing request, carrying out storage abnormality detection according to the aperiodic abnormality detection channel to obtain a second abnormality detection result; and according to the first abnormality detection result or the second abnormality detection result, the storage transfer module writes the data to be written of the data storage write request information into a memory.
In a second aspect of the present application, there is provided a cloud platform-based data storage abnormality detection apparatus, the apparatus comprising: the request information acquisition module is used for acquiring data storage writing request information; the storage transfer module is used for setting the storage transfer module, wherein the storage transfer module is arranged in the cloud platform, a storage abnormality detection model is embedded in the storage transfer module, and the storage abnormality detection model comprises a periodic abnormality detection channel and an aperiodic abnormality detection channel; the request information judging module is used for analyzing the received data storage writing request information through the storage transfer module and judging whether the data storage writing request information is a periodic writing request or not; the first abnormal detection result acquisition module is used for judging that if the data storage writing request information is a periodic writing request, carrying out storage abnormal detection according to the periodic abnormal detection channel and acquiring a first abnormal detection result; the second abnormality detection result acquisition module is used for judging that if the data storage writing request information is an aperiodic writing request, carrying out storage abnormality detection according to the aperiodic abnormality detection channel and acquiring a second abnormality detection result; the data to be written is written into the memory according to the first abnormality detection result or the second abnormality detection result, and the data to be written into of the data storage writing request information is written into the memory by the storage transfer module.
One or more technical schemes provided by the application have at least the following technical effects or advantages:
According to the method provided by the embodiment of the application, the data storage writing request information is acquired, the storage transfer module is arranged, the received data storage writing request information is analyzed, whether the data storage writing request information is a periodic writing request or not is judged, if the data storage writing request information is the periodic writing request, the storage abnormality detection is carried out according to the periodic abnormality detection channel, the first abnormality detection result is acquired, if the data storage writing request information is the non-periodic writing request, the storage abnormality detection is carried out according to the non-periodic abnormality detection channel, the second abnormality detection result is acquired, and the storage transfer module writes the data to be written of the data storage writing request information into the memory according to the first abnormality detection result or the second abnormality detection result. The technical effects of efficiently detecting abnormal data, improving data safety and reliability and realizing efficient data processing and storage are achieved through storage transfer preprocessing.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following description will briefly explain the drawings needed in the description of the embodiments, which are merely examples of the present invention, and other drawings can be obtained according to these drawings without inventive effort for a person skilled in the art.
FIG. 1 is a schematic flow chart of a data storage abnormality detection method based on a cloud platform;
FIG. 2 is a schematic flow chart of a method for detecting data storage anomalies based on a cloud platform to obtain a first anomaly detection result;
Fig. 3 is a schematic structural diagram of a cloud platform-based data storage abnormality detection device provided by the application.
Reference numerals illustrate: the system comprises a request information acquisition module 11, a storage transit module 12, a request information judgment module 13, a first abnormality detection result acquisition module 14, a second abnormality detection result acquisition module 15 and a data writing module 16.
Detailed Description
The application provides a cloud platform-based data storage abnormality detection method and equipment, which are used for solving the technical problems that in the prior art, data are directly written into a storage system, abnormal data are mixed, and the data safety and reliability are insufficient. The technical effects of efficiently detecting abnormal data, improving data safety and reliability and realizing efficient data processing and storage are achieved through storage transfer preprocessing.
In the following, the technical solutions of the present invention will be clearly and completely described with reference to the accompanying drawings, and it should be understood that the described embodiments are only some embodiments of the present invention, but not all embodiments of the present invention, and that the present invention is not limited by the exemplary embodiments described herein. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention. It should be further noted that, for convenience of description, only some, but not all of the drawings related to the present invention are shown.
In a first embodiment, as shown in fig. 1, the present application provides a method for detecting data storage abnormality based on a cloud platform, the method comprising:
And acquiring the data storage write request information.
Specifically, there are one or more interfaces in the cloud platform for receiving data storage write requests from different data sources or applications, and the interfaces support multiple data formats, such as JSON, XML, binary, etc., to accommodate the data representation of the different data sources. Meanwhile, each interface definitely requests the data type and structure of the parameter, including the specific content, data type, format, size, etc. information of the data to be written. The cloud platform interface is used for receiving and acquiring the data storage writing request information from the client or the application program, wherein the data storage writing request information comprises the data content to be stored and related metadata, so that the cloud platform can comprehensively know the writing activity of the data storage, and a necessary data basis is provided for subsequent anomaly detection and security analysis.
The method comprises the steps of setting a storage transfer module, wherein the storage transfer module is arranged in a cloud platform, a storage abnormality detection model is embedded in the storage transfer module, and the storage abnormality detection model comprises a periodic abnormality detection channel and an aperiodic abnormality detection channel.
Specifically, in order to enhance the reliability and security of data storage, a storage transfer module is provided, where the storage transfer module is deployed on a cloud platform and used as an intermediate processing layer for checking the writing mode of the data storage and updating the storage protocol. The storage transfer module is embedded with a storage abnormality detection model which is used for detecting whether written data are abnormal or not. The stored anomaly detection model includes two main channels: a periodic abnormality detection channel for detecting abnormalities in data having a periodicity law, and a non-periodic abnormality detection channel capable of identifying data points that do not conform to the periodicity law, i.e., abnormal data, by analyzing a periodic characteristic of the data, such as a daily, weekly, or monthly periodic variation. The aperiodic anomaly detection channel is different from the periodic channel in that it does not depend on the periodicity law of the data, but detects anomalies based on a broader data characteristic and statistical law. A variety of algorithms are employed, such as distance-based anomaly detection, density-based anomaly detection, cluster-based anomaly detection, and the like, to identify isolated points, clusters of anomalies, or data points that do not conform to the overall data distribution in the data. By combining the periodic anomaly detection channel and the aperiodic anomaly detection channel in the storage transfer module, the anomaly data in the data storage request can be more comprehensively and accurately identified, so that the interference of the anomaly data on subsequent processing and analysis is avoided, and the reliability and the data quality of data storage are improved.
And analyzing the received data storage write request information through the storage transfer module, and judging whether the data storage write request information is a periodic write request or not.
Specifically, after the storage transfer module receives the data storage writing request information, the data storage writing request information is analyzed in detail, and the time sequence characteristics of the data storage writing request information are analyzed. And determining whether the data storage write-in request belongs to a periodic write-in request or not according to an analysis result, wherein the periodic write-in request refers to a repeated data storage request according to a fixed time interval, and data carried by the periodic write-in request has a certain periodicity rule. By analyzing and judging the data storage writing request information, the periodic writing request and the aperiodic writing request can be distinguished, and the storage transfer module can more reasonably allocate storage resources and improve the processing efficiency.
And if the data storage writing request information is a periodic writing request, carrying out storage abnormality detection according to the periodic abnormality detection channel to obtain a first abnormality detection result.
And if the data storage writing request information is an aperiodic writing request, carrying out storage abnormality detection according to the aperiodic abnormality detection channel, and obtaining a second abnormality detection result.
Specifically, when the decision determines that the data storage write request information is a periodic write request, the storage relay module will use the periodic anomaly detection channel for storage anomaly detection. The periodic anomaly detection channel utilizes the periodic characteristics of the data to analyze various attributes of the request, including, but not limited to, the timestamp of the request, the data content, the source of the request, the destination address, the storage permission level, and the like. By analyzing the data store write request information, the aperiodic anomaly detection channel can generate a stored anomaly detection result, i.e., a first anomaly detection result.
When the data storage writing request information is judged to be not the periodic writing request, namely the data storage writing request information is the aperiodic writing request, the storage transfer module selects an aperiodic anomaly detection channel to analyze and detect the data storage writing request information. The aperiodic abnormal detection does not depend on the periodicity rule of the data, and the aperiodic write-in request information is detected by adopting various algorithms and technologies, such as a density-based abnormal detection algorithm, a distance-based isolated point detection method, a statistical distribution-based abnormal recognition method and the like, so as to obtain a second abnormal detection result.
Through the analysis and judgment, the storage transfer module can intelligently select a proper abnormality detection channel according to the characteristics of the data storage writing request information, so that the accurate abnormality detection of the periodic writing request and the aperiodic writing request is realized, the efficiency and the accuracy of data storage are improved, and the safety and the reliability of data are enhanced.
And according to the first abnormality detection result or the second abnormality detection result, the storage transfer module writes the data to be written of the data storage write request information into a memory.
Specifically, the storage relay module judges according to a first abnormality detection result from the periodic abnormality detection channel or a second abnormality detection result of the non-periodic abnormality detection channel, wherein the first abnormality detection result and the second abnormality detection result contain information about whether the request has an abnormality or not. If the abnormality detection result indicates that the request is not abnormal, namely the request passes detection, the storage transfer module continues to execute data writing operation, and writes the data to be written in the data storage writing request information into the memory. If the abnormality detection result shows that the request has abnormality, the storage transfer module refuses to write the data storage writing request information, does not execute writing operation, and starts an abnormality processing flow, such as recording the abnormality information into a log file or sending an alarm to a system administrator, so as to prevent potential safety risk or data damage. By determining whether to write the data to be written in the data storage write request information into the memory according to the abnormality detection result, the data which passes through the abnormality detection can be written, so that the safety and reliability of the data storage are improved, the occupation of the abnormal data write into the memory and the operation space is effectively avoided, and the efficient data processing and storage are realized.
Further, the storage transfer module includes a data backup module, and the method further includes: if the storage transfer module receives a plurality of concurrent writing request information; generating a write request queue according to the multiple concurrent write request messages, wherein the write request queue is connected with the data backup module; detecting backup conditions of all requests in the write request queue, and if the backup conditions are met, activating the data backup module to perform data backup on the requests meeting the backup conditions; and performing anomaly detection on the backed-up data to be written, and if the anomaly detection is passed, writing the backed-up data to be written in the data backup module into the memory by the storage transfer module.
Specifically, the storage transfer module further comprises a data backup module, and the data backup module is responsible for copying the data to be written to a safe storage medium so as to ensure the restorability of the data. The storage transfer module is used as a data processing center to receive a plurality of writing request information from different sources or applications, and when the storage transfer module receives a plurality of data writing operation requests at the same time in a very short time interval, all writing requests are stored according to the order of the plurality of writing request information, so as to generate a writing request queue. The write request queue is a first-in first-out data structure, and a connection is established with the data backup module, and each time a new write request enters the queue, the storage transfer module informs the data backup module so as to call the backup function when needed.
Next, a reasonable failure rate threshold is set according to the traffic demand and the data importance. The failure rate threshold is used for judging whether the reliability of the requested writing data belongs to the tolerant maximum failure rate level. And comparing each request in the write request queue with a failure rate threshold, and when detecting that the failure rate of a certain request or some requests exceeds a preset failure rate threshold, considering that the request meets a backup condition, and activating a data backup module to perform data backup on the request.
After the data is backed up to the data backup module, an abnormality detection flow is carried out on the backed-up data to be written, so that the quality of the backed-up data is ensured. If the backed-up data to be written passes the abnormality detection, the storage transfer module writes the backed-up data to be written in the data backup module into the memory. Ensuring that the original data is lost or damaged in the writing process, and also can be recovered by backing up the data. Through the backup and re-detection processes, the reliability and the safety of the data can be ensured, and the risk of data loss is reduced.
Further, the method for detecting the backup condition of each request in the write request queue comprises the following steps: the storage transfer module performs data extraction on the data storage writing request information to acquire data to be written; performing failure analysis on the data to be written according to the request response time of the write request queue, and obtaining data failure probability; judging the data failure probability, and backing up the data to be written according to the data backup module if the data failure probability is larger than the expected failure probability.
Specifically, after receiving the data storage writing request information, the storage transfer module firstly performs data extraction operation, and analyzes the data to be written from the storage writing request information. The response time of each request in the write request queue, which refers to the time interval from the request commit to the data actually being written to memory, is then analyzed. For data with timeliness, a longer response time may cause the value of the data to be reduced or completely fail, so that failure analysis is performed on the data to be written according to the request response time of the write request queue, and the data failure probability is calculated, wherein the data failure probability is used for reflecting the possibility that the write data fails under the current response time. The calculated data failure probability is then compared with a predetermined expected failure probability, i.e. a failure rate threshold set according to traffic demand and data importance. If the calculated data failure probability is larger than the expected failure probability, the fact that the current data writing request has higher failure risk is indicated, a data backup flow is triggered, and the current data to be written is backed up according to the data backup module. After the data backup is completed, the storage transfer module continues to process the next request in the write request queue, so that timely backup of the data sensitive to time efficiency is ensured, the data failure risk caused by overlong response time is reduced, and the reliability and accuracy of the data are improved.
In one embodiment, as shown in fig. 2, the method includes performing storage anomaly detection according to the periodic anomaly detection channel to obtain a first anomaly detection result, where the method includes: analyzing the data storage writing request information to acquire data to be written, a request address, a storage authority level and a protocol period; obtaining a detection sample under a characteristic period according to the periodic abnormal detection channel; and carrying out anomaly detection on the data to be written, the request address, the storage authority level and the protocol period according to the detection sample under the characteristic period to obtain a first anomaly detection result, wherein the first anomaly detection result comprises a data anomaly detection result, an address anomaly detection result, an authority anomaly detection result and a period anomaly detection result.
Specifically, the data storage writing request information is analyzed in detail, and key information is extracted, wherein the key information comprises data to be written, a request address, a storage authority level and a protocol period, the data to be written is the actual data to be stored, the request address is the position where the data should be written into the memory, the storage authority level is the data authority level written by a requester at the address, and the protocol period is the periodicity rule followed by the request, such as every hour, every day and the like. The periodic anomaly detection channel is responsible for monitoring and identifying those factors that may deviate from the normal periodic pattern. In order to perform abnormality detection, a detection sample matched with a specific periodic characteristic is acquired according to the periodic abnormality detection channel. The detection samples under the characteristic period are from historical data or other reliable data sources and represent data characteristics under normal periodic behavior for subsequent anomaly detection processes to identify and determine whether the current data storage write request deviates from the expected periodic pattern.
The method comprises the steps that a detection sample under a characteristic period is utilized, a storage transfer module carries out anomaly detection on extracted data to be written, a request address, a storage authority level and a protocol period respectively, an anomaly point which is inconsistent with a normal mode is identified, a first anomaly detection result is generated, and the first anomaly detection result comprises four parts: the data processing method comprises the steps of data anomaly detection results, address anomaly detection results, authority anomaly detection results and periodic anomaly detection results, wherein the data anomaly detection results are whether data have problems, such as data format errors, data damage and the like, the address anomaly detection results are whether a request address is legal or authorized to be written, the authority anomaly detection results are whether a requester has authority to write data at the address, and the periodic anomaly detection results are whether the request follows a correct periodic rule. The method and the device have the advantages that the request information is subjected to comprehensive abnormality detection based on the detection sample under the characteristic period, various types of abnormalities of the data storage and writing request information can be effectively detected, and then the safety and the accuracy of the written data are guaranteed.
Further, according to the aperiodic anomaly detection channel, storage anomaly detection is performed to obtain a second anomaly detection result, and the method includes: extracting characteristics of the data storage writing request information to obtain request frequency, data quantity change index and access mode change degree; the aperiodic abnormality detection channel obtains LOF scores corresponding to each feature based on local abnormality factors; and acquiring a preset LOF score, and performing anomaly detection on the LOF score corresponding to each feature according to the preset LOF score to acquire a second anomaly detection result.
Specifically, the non-periodic writing request data is analyzed in detail, key characteristic information is extracted, the key characteristic information comprises request frequency, a data quantity change index and an access mode change degree, the request frequency is the number of requests received in unit time, the data quantity change index is the change condition of the data quantity contained in the requests, for example, the data quantity related to each writing request is analyzed, statistics such as an average value, a standard deviation, a maximum value, a minimum value and the like of the data quantity are calculated to measure the change condition of the data quantity, and the access mode change degree is the change of a mode or a path of requesting access storage.
And calculating the local anomaly factor (LOF) score of each feature based on the request frequency, the data volume change index and the access mode change degree of the data storage writing request information by utilizing an aperiodic anomaly detection channel in the storage transit module, and obtaining the LOF score corresponding to one feature, wherein the greater the LOF score is, the more likely the data point is an anomaly point. Based on the local anomaly factor, an algorithm for anomaly detection, anomalies are identified by calculating the local density deviation of a data point from other data points in its neighborhood. And setting one or more preset LOF score thresholds according to the historical data and the business requirements, wherein the LOF score thresholds are used for judging whether the data points are abnormal or not. And comparing the LOF score corresponding to each feature with a preset LOF score threshold. If the LOF score of a certain feature exceeds a corresponding threshold value, the feature is considered to be abnormal, and the abnormal detection results of all the features are synthesized to obtain a second abnormal detection result. And the LOF algorithm is utilized to comprehensively analyze a plurality of characteristics, so that the accuracy and the efficiency of anomaly detection are improved, and the situations of false alarm and missing report are reduced.
Further, the LOF score corresponding to each feature is obtained, and the method comprises the following steps: determining a neighbor number K value, wherein the K value determines the number of adjacent points considered in calculating the local density of each feature; calculating K nearest neighbor distances corresponding to each feature, and acquiring the local density of each feature and the adjacent local density according to the K nearest neighbor distances; and obtaining the ratio of the local density corresponding to each feature to the adjacent local density, and obtaining the LOF score corresponding to each feature.
Specifically, the K value of the suitable neighbor number is determined according to user experience or historical experiments, the K value determines the number of the adjacent points considered in calculating the local density of each feature, the K value is not too large or too small, the abnormal points in the dense area are not easy to identify due to too large K value, and the normal points are misjudged to be abnormal due to too small K value. For each feature in the request frequency, the data volume change index, the access pattern change level, the LOF algorithm calculates the distances between that feature and all other features, which are derived by calculating euclidean distances or other similarity measures in the multidimensional feature space of the data points, which measure the similarity or difference between the features. And sorting all the features according to the calculated distances, and selecting the K nearest features as K nearest neighbors of the features. Then, using the distances of the K nearest neighbors, a local density of each feature and a nearest neighbor local density are calculated, wherein the local density is an index for measuring the tightness degree between a data point and the K nearest neighbors thereof, the local density is calculated and obtained through the sum of the inverse numbers of the distances, and the higher the local density is, the closer the data point is to the nearest neighbors thereof. The nearest neighbor local density is a density determined by the distance of each feature from its own K nearest neighbors. And finally, calculating the ratio of the local density corresponding to each feature to the local density of the adjacent feature to obtain LOF score corresponding to each feature, wherein the higher the LOF score is, the more likely the data point is an abnormal point relative to the local neighborhood of the data point.
Further, determining whether the data storage write request information is a periodic write request, the method comprising: performing time sequence identification on the data storage writing request information to acquire time sequence repeatability; and carrying out cluster analysis on the data storage writing request information according to time sequence repeatability to obtain classification results, wherein the classification results comprise periodic writing requests and aperiodic writing requests.
Specifically, timestamp information of all data store write requests is first collected, with the timestamp recording the specific time at which each write operation occurred. The time stamps are converted into time series data suitable for analysis, i.e. the time stamps are converted into offsets relative to a certain reference point in time, or the time is divided into finer granularity units, e.g. minutes, hours. Periodic patterns in the write requests, such as repeated patterns per hour, per few hours, per day, per week, per month or longer, are detected using statistical methods or time series analysis techniques, such as autocorrelation analysis, fourier transforms, seasonal decomposition, etc. And for the detected periodic pattern, quantifying the degree of repeatability of the periodic pattern by calculating the similarity, stability index or periodic intensity of the writing request in the period, and obtaining the time sequence repeatability.
After the time sequence repeatability is identified, the data storage writing request information is clustered according to a clustering algorithm, wherein the clustering algorithm comprises K-means, hierarchical clustering, DBSCAN and the like. In the clustering process, the clustering algorithm groups similar writing requests together according to the extracted features, so that the features of the periodic writing requests and the aperiodic writing requests can be distinguished, and a classification result is obtained, wherein the classification result comprises the periodic writing requests and the aperiodic writing requests. By carrying out time sequence recognition and cluster analysis on the data storage writing request, whether the data storage writing request information is a periodic writing request or not can be judged efficiently and accurately, and further the accuracy and reliability of subsequent data storage abnormality detection are improved, and efficient data processing and storage are realized.
In a second embodiment, based on the same inventive concept as the data storage abnormality detection method based on the cloud platform in the foregoing embodiment, as shown in fig. 3, the present application provides a data storage abnormality detection device based on the cloud platform, where the device includes:
a request information acquisition module 11, where the request information acquisition module 11 is configured to acquire data storage write request information;
The storage transfer module 12 is used for setting a storage transfer module, wherein the storage transfer module is arranged in the cloud platform, a storage abnormality detection model is embedded in the storage transfer module, and the storage abnormality detection model comprises a periodic abnormality detection channel and an aperiodic abnormality detection channel;
A request information judging module 13, where the request information judging module 13 is configured to analyze, by using the storage relay module, the received data storage write request information, and judge whether the data storage write request information is a periodic write request;
the first abnormality detection result obtaining module 14, where the first abnormality detection result obtaining module 14 is configured to determine that if the data storage write request information is a periodic write request, perform storage abnormality detection according to the periodic abnormality detection channel, and obtain a first abnormality detection result;
the second abnormality detection result obtaining module 15, where the second abnormality detection result obtaining module 15 is configured to determine that if the data storage write request information is an aperiodic write request, perform storage abnormality detection according to the aperiodic abnormality detection channel, and obtain a second abnormality detection result;
The data to be written writing module 16, where the data to be written writing module 16 is configured to write the data to be written of the data storage writing request information into a memory according to the first anomaly detection result or the second anomaly detection result by using the storage relay module.
Further, the data writing module to be written 16 is further configured to perform the following steps: if the storage transfer module receives a plurality of concurrent writing request information; generating a write request queue according to the multiple concurrent write request messages, wherein the write request queue is connected with the data backup module; detecting backup conditions of all requests in the write request queue, and if the backup conditions are met, activating the data backup module to perform data backup on the requests meeting the backup conditions; and performing anomaly detection on the backed-up data to be written, and if the anomaly detection is passed, writing the backed-up data to be written in the data backup module into the memory by the storage transfer module.
Further, the data writing module to be written 16 is further configured to perform the following steps: the storage transfer module performs data extraction on the data storage writing request information to acquire data to be written; performing failure analysis on the data to be written according to the request response time of the write request queue, and obtaining data failure probability; judging the data failure probability, and backing up the data to be written according to the data backup module if the data failure probability is larger than the expected failure probability.
Further, the first abnormality detection result obtaining module 14 is further configured to perform the following steps: analyzing the data storage writing request information to acquire data to be written, a request address, a storage authority level and a protocol period; obtaining a detection sample under a characteristic period according to the periodic abnormal detection channel; and carrying out anomaly detection on the data to be written, the request address, the storage authority level and the protocol period according to the detection sample under the characteristic period to obtain a first anomaly detection result, wherein the first anomaly detection result comprises a data anomaly detection result, an address anomaly detection result, an authority anomaly detection result and a period anomaly detection result.
Further, the second abnormality detection result obtaining module 15 is further configured to perform the following steps: extracting characteristics of the data storage writing request information to obtain request frequency, data quantity change index and access mode change degree; the aperiodic abnormality detection channel obtains LOF scores corresponding to each feature based on local abnormality factors; and acquiring a preset LOF score, and performing anomaly detection on the LOF score corresponding to each feature according to the preset LOF score to acquire a second anomaly detection result.
Further, the second abnormality detection result obtaining module 15 is further configured to perform the following steps: determining a neighbor number K value, wherein the K value determines the number of adjacent points considered in calculating the local density of each feature; calculating K nearest neighbor distances corresponding to each feature, and acquiring the local density of each feature and the adjacent local density according to the K nearest neighbor distances; and obtaining the ratio of the local density corresponding to each feature to the adjacent local density, and obtaining the LOF score corresponding to each feature.
Further, the request information determining module 13 is further configured to perform the following steps: performing time sequence identification on the data storage writing request information to acquire time sequence repeatability; and carrying out cluster analysis on the data storage writing request information according to time sequence repeatability to obtain classification results, wherein the classification results comprise periodic writing requests and aperiodic writing requests.
The foregoing description of the preferred embodiments of the application is not intended to limit the application to the precise form disclosed, and any such modifications, equivalents, and alternatives falling within the spirit and scope of the application are intended to be included within the scope of the application.
The specification and figures are merely exemplary illustrations of the present application and are considered to cover any and all modifications, variations, combinations, or equivalents that fall within the scope of the application. It will be apparent to those skilled in the art that various modifications and variations can be made to the present application without departing from the scope of the application. Thus, it is intended that the present application cover the modifications and variations of this application provided they come within the scope of the application and its equivalents.
Claims (5)
1. The data storage abnormality detection method based on the cloud platform is characterized by comprising the following steps of:
Acquiring data storage write-in request information;
setting a storage transfer module, wherein the storage transfer module is arranged in a cloud platform, a storage abnormality detection model is embedded in the storage transfer module, and the storage abnormality detection model comprises a periodic abnormality detection channel and an aperiodic abnormality detection channel;
analyzing the received data storage writing request information through the storage transfer module, and judging whether the data storage writing request information is a periodic writing request or not;
If the data storage writing request information is a periodic writing request, carrying out storage abnormality detection according to the periodic abnormality detection channel to obtain a first abnormality detection result;
if the data storage writing request information is an aperiodic writing request, carrying out storage abnormality detection according to the aperiodic abnormality detection channel to obtain a second abnormality detection result;
according to the first abnormality detection result or the second abnormality detection result, the storage transfer module writes the data to be written of the data storage write request information into a memory;
The method comprises the steps of performing storage abnormality detection according to the periodic abnormality detection channel to obtain a first abnormality detection result, and comprises the following steps:
Analyzing the data storage writing request information to acquire data to be written, a request address, a storage authority level and a protocol period;
Obtaining a detection sample under a characteristic period according to the periodic abnormal detection channel;
Performing anomaly detection on the data to be written, the request address, the storage authority level and the protocol period according to the detection sample under the characteristic period to obtain a first anomaly detection result, wherein the first anomaly detection result comprises a data anomaly detection result, an address anomaly detection result, an authority anomaly detection result and a period anomaly detection result;
The method comprises the steps of performing storage anomaly detection according to the aperiodic anomaly detection channel to obtain a second anomaly detection result, and comprises the following steps:
extracting characteristics of the data storage writing request information to obtain request frequency, data quantity change index and access mode change degree;
The aperiodic abnormality detection channel obtains LOF scores corresponding to each feature based on local abnormality factors;
Acquiring a preset LOF score, and performing anomaly detection on the LOF score corresponding to each feature according to the preset LOF score to acquire a second anomaly detection result;
The aperiodic anomaly detection channel obtains LOF scores corresponding to each feature based on local anomaly factors, and the method comprises the following steps:
Determining a neighbor number K value, wherein the K value determines the number of adjacent points considered in calculating the local density of each feature;
calculating K nearest neighbor distances corresponding to each feature, and acquiring the local density of each feature and the adjacent local density according to the K nearest neighbor distances;
And obtaining the ratio of the local density corresponding to each feature to the adjacent local density, and obtaining the LOF score corresponding to each feature.
2. The cloud platform based data storage anomaly detection method of claim 1, wherein the storage relay module comprises a data backup module, the method further comprising:
If the storage transfer module receives a plurality of concurrent writing request information;
generating a write request queue according to the multiple concurrent write request messages, wherein the write request queue is connected with the data backup module;
Detecting backup conditions of all requests in the write request queue, and if the backup conditions are met, activating the data backup module to perform data backup on the requests meeting the backup conditions;
And performing anomaly detection on the backed-up data to be written, and if the anomaly detection is passed, writing the backed-up data to be written in the data backup module into the memory by the storage transfer module.
3. The cloud platform-based data storage anomaly detection method of claim 2, wherein backup condition detection is performed for each request in the write request queue, the method comprising:
the storage transfer module performs data extraction on the data storage writing request information to acquire data to be written;
performing failure analysis on the data to be written according to the request response time of the write request queue, and obtaining data failure probability;
judging the data failure probability, and backing up the data to be written according to the data backup module if the data failure probability is larger than the expected failure probability.
4. The cloud platform based data storage anomaly detection method of claim 1, wherein determining whether the data storage write request information is a periodic write request comprises:
performing time sequence identification on the data storage writing request information to acquire time sequence repeatability;
And carrying out cluster analysis on the data storage writing request information according to time sequence repeatability to obtain classification results, wherein the classification results comprise periodic writing requests and aperiodic writing requests.
5. A cloud platform based data storage anomaly detection device, characterized by comprising the steps for implementing the method of any one of claims 1 to 4:
the request information acquisition module is used for acquiring data storage writing request information;
the storage transfer module is used for setting the storage transfer module, wherein the storage transfer module is arranged in the cloud platform, a storage abnormality detection model is embedded in the storage transfer module, and the storage abnormality detection model comprises a periodic abnormality detection channel and an aperiodic abnormality detection channel;
The request information judging module is used for analyzing the received data storage writing request information through the storage transfer module and judging whether the data storage writing request information is a periodic writing request or not;
The first abnormal detection result acquisition module is used for judging that if the data storage writing request information is a periodic writing request, carrying out storage abnormal detection according to the periodic abnormal detection channel and acquiring a first abnormal detection result;
the second abnormality detection result acquisition module is used for judging that if the data storage writing request information is an aperiodic writing request, carrying out storage abnormality detection according to the aperiodic abnormality detection channel and acquiring a second abnormality detection result;
the data to be written is written into the memory according to the first abnormality detection result or the second abnormality detection result, and the data to be written into of the data storage writing request information is written into the memory by the storage transfer module.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202410995977.5A CN118519820B (en) | 2024-07-24 | 2024-07-24 | Cloud platform-based data storage abnormality detection method and equipment |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202410995977.5A CN118519820B (en) | 2024-07-24 | 2024-07-24 | Cloud platform-based data storage abnormality detection method and equipment |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN118519820A CN118519820A (en) | 2024-08-20 |
| CN118519820B true CN118519820B (en) | 2024-09-27 |
Family
ID=92281493
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202410995977.5A Active CN118519820B (en) | 2024-07-24 | 2024-07-24 | Cloud platform-based data storage abnormality detection method and equipment |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN118519820B (en) |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112818066A (en) * | 2019-11-15 | 2021-05-18 | 深信服科技股份有限公司 | Time sequence data anomaly detection method and device, electronic equipment and storage medium |
| CN113377568A (en) * | 2021-06-29 | 2021-09-10 | 北京同创永益科技发展有限公司 | Abnormity detection method and device, electronic equipment and storage medium |
| CN116028263A (en) * | 2021-10-25 | 2023-04-28 | 中移(苏州)软件技术有限公司 | Database data backup method and device, computing equipment and storage medium |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116483623A (en) * | 2023-04-17 | 2023-07-25 | 中科易存软件江苏有限公司 | Data storage system and storage equipment based on cloud computing |
-
2024
- 2024-07-24 CN CN202410995977.5A patent/CN118519820B/en active Active
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112818066A (en) * | 2019-11-15 | 2021-05-18 | 深信服科技股份有限公司 | Time sequence data anomaly detection method and device, electronic equipment and storage medium |
| CN113377568A (en) * | 2021-06-29 | 2021-09-10 | 北京同创永益科技发展有限公司 | Abnormity detection method and device, electronic equipment and storage medium |
| CN116028263A (en) * | 2021-10-25 | 2023-04-28 | 中移(苏州)软件技术有限公司 | Database data backup method and device, computing equipment and storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN118519820A (en) | 2024-08-20 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111045894B (en) | Database abnormality detection method, database abnormality detection device, computer device and storage medium | |
| CN102257520B (en) | Application performance analysis | |
| CN111177714B (en) | Abnormal behavior detection method and device, computer equipment and storage medium | |
| CN111612038B (en) | Abnormal user detection method and device, storage medium and electronic equipment | |
| CN111782484B (en) | Anomaly detection method and device | |
| EP3465509A1 (en) | Classification of log data | |
| CN119011279B (en) | Network security log management method and system | |
| Gurumdimma et al. | Towards detecting patterns in failure logs of large-scale distributed systems | |
| CN119865372A (en) | DNS resolution-oriented secure access control method and system | |
| CN113220585A (en) | Automatic fault diagnosis method and related device | |
| JP2007243459A (en) | Traffic state extraction apparatus and method, and computer program | |
| CN119383169B (en) | IP recovery method, system, equipment and medium for regional service interruption | |
| CN119273364A (en) | A threshold management method and system based on financial risk | |
| CN118519820B (en) | Cloud platform-based data storage abnormality detection method and equipment | |
| CN112924743B (en) | Instrument state detection method based on current data | |
| CN114168422A (en) | Operation and maintenance data anomaly detection method, device, equipment and medium | |
| CN119127630A (en) | Illegal behavior identification method, device, computer equipment and storage medium | |
| CN112445687A (en) | Blocking detection method of computing equipment and related device | |
| US12118084B2 (en) | Automatic selection of data for target monitoring | |
| US20240086271A1 (en) | Organization segmentation for anomaly detection | |
| CN118094383A (en) | Multidimensional anomaly detection method and device based on fusion model | |
| KR20210033839A (en) | System and method for performance monitoring | |
| CN113360313B (en) | Behavior analysis method based on massive system logs | |
| CN111798237A (en) | Abnormal transaction diagnosis method and system based on application log | |
| CN120256986B (en) | A method and device for detecting insulation status of a high-voltage composite lightning arrester |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |