CN115481095A - Heat monitoring-based file access path optimization method in integral system - Google Patents
Heat monitoring-based file access path optimization method in integral system Download PDFInfo
- Publication number
- CN115481095A CN115481095A CN202210690625.XA CN202210690625A CN115481095A CN 115481095 A CN115481095 A CN 115481095A CN 202210690625 A CN202210690625 A CN 202210690625A CN 115481095 A CN115481095 A CN 115481095A
- Authority
- CN
- China
- Prior art keywords
- heat
- access
- information
- node
- file
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/18—File system types
- G06F16/182—Distributed file systems
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
- G06F3/061—Improving I/O performance
- G06F3/0611—Improving I/O performance in relation to response time
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
- G06F3/0629—Configuration or reconfiguration of storage systems
- G06F3/0635—Configuration or reconfiguration of storage systems by changing the path, e.g. traffic rerouting, path reconfiguration
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
- G06F3/0638—Organizing or formatting or addressing of data
- G06F3/0643—Management of files
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0668—Interfaces specially adapted for storage systems adopting a particular infrastructure
- G06F3/067—Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Human Computer Interaction (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- Computer Networks & Wireless Communication (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention provides a file access path optimization method based on heat monitoring in an integral system. In the integral data storage system, fragment storage information of a file main copy is constructed based on file metadata; and monitoring the access heat of each fragment storage node in real time, comparing the access heat with a set heat model, inquiring the lowest heat information of the copy storage node corresponding to the fragment under the condition that the access heat is higher than a heat threshold value, and updating the fragment access address in the metadata information into the copy storage node with low heat if the lowest heat of the copy storage node is lower than the heat of the current main storage node. The system comprises an integral file metadata fragment storage information module, a file fragment storage node access heat monitoring module, a heat evaluation model and path adjusting module and a system management configuration module.
Description
Technical Field
The invention relates to the field of integration, in particular to a file access path optimization method based on heat monitoring in an integration system.
Background
Due to the rapid development of internet services in recent years, massive data is generated every day. With the development and application of cloud computing and big data, more and more internet service systems adopt a distributed storage technology to store mass data. In the distributed storage technology, disk space on each machine in the system is fully used through a network, and the distributed storage resources form a virtual storage device, so that data is stored in each corner of an enterprise in a distributed manner. Distributed storage has good advantages, including scalability: distributed storage systems can scale to hundreds or even thousands of such cluster sizes, and the overall performance of the system can grow linearly; high availability: in the distributed file system, the high availability comprises two layers, namely the availability of the whole file system and the completeness and consistency of data; the cost is low: automatic fault tolerance and automatic load balancing of distributed storage systems allows for the construction of distributed storage systems on lower cost servers. In addition, linear scalability can also increase and decrease the cost of the server; and the elastic storage can flexibly increase or reduce the data storage and increase or delete the resources in the storage pool according to the service requirement without interrupting the system operation and the like.
In the distributed storage technology, a fragmentation storage technology is generally adopted, namely, data in a large database is divided into a plurality of data fragments according to a certain rule, and then the data fragments are respectively stored in different servers, so that the data access pressure of each server is reduced, and the performance of the whole database system is improved.
The integral service system is applied in a plurality of ways, has frequent access and large data volume, and also constructs a distributed fragment storage technology. However, the node performance and the access heat of the file fragmentation storage are uneven, so that the problem that the whole file access efficiency is reduced due to over-high heat and low reading speed of part of nodes in the file access process is caused. Usually, one file has multiple copies, and the copies can be preset in the low-frequency access node, so that the low-frequency access node can be adjusted, and the problem of low efficiency of file access caused by the high-temperature access node is effectively solved. The patent of the invention is a method and a system which are formally constructed based on the background and the thought of the requirement.
Disclosure of Invention
The invention provides a file access path optimization method based on heat monitoring in an integral system. In the integral data storage system, fragment storage information of a file main copy is constructed based on file metadata; and monitoring the access heat of each fragment storage node in real time, comparing the access heat with a set heat model, inquiring the lowest heat information of the copy storage node corresponding to the fragment under the condition that the access heat is higher than a heat threshold value, and updating the fragment access address in the metadata information into the copy storage node with low heat if the lowest heat of the copy storage node is lower than the heat of the current main storage node. The system comprises an integral file metadata fragment storage information module, a file fragment storage node access heat monitoring module, a heat evaluation model and path adjusting module and a system management configuration module.
1. The score file metadata fragment storage information module comprises:
in the metadata information of the integral file, access address information stored in fragments is constructed, wherein the access address information comprises access address information of each sequential fragment, storage address information of all master copies, storage node information and master copy corresponding relation information, which are provided when external access is carried out.
2. The file fragment storage node access heat monitoring module comprises:
the module monitors the access heat of the global file fragment storage node. The information accessed by the node at any time is recorded in the heat monitoring data. And setting a unit time window for heat monitoring, and continuously recording and counting access heat information H (n) of each node through the time window and sending the access heat information H (n) to a heat evaluation model and a path adjusting module.
3. The heat evaluation model and path adjusting module:
in the module, a node heat threshold F model is set according to the operation, operation and maintenance experience of the integral service system. The node performance can be uniformly set (for example, F0 can be set when the node performance configuration is uniform), or different node threshold settings (F (n)) can be performed according to different node performances.
When receiving all node access statistical data H (n) continuously sent by a file fragment storage node access heat monitoring module, aiming at the node n, if:
(1) H (n) < F (n), it means no optimization is needed, and no adjustment is made;
(2) H (n) ≧ F (n), this means the heat is too high, needs to adjust optimization. At this time, firstly, searching real-time heat information H (x) of a node stored in each copy corresponding to the segment by a module for storing information in the metadata segment of the integral file, and finding out a minimum heat value MIN (H (x)) of other corresponding copy storage nodes:
A. if MIN (H (x)) < F (n), extracting copy access address information corresponding to the minimum value, and updating the fragment access address information in the integral file metadata fragment storage information module;
if MIN (H (x)) > F (n), comparing the sizes of H (n) and MIN (H (x)), and if H (n) > MIN (H (x)), indicating that the minimum hot degree of a replica node is lower than the hot degree of the current main slice, extracting replica access address information corresponding to the minimum hot degree, and updating the slice access address information in the score file metadata slice storage information module; otherwise, the access heat of the current main partition is too high, but the mode is still relatively optimal compared with the heat of the node where other copies are located, and the access address is not updated.
Drawings
Fig. 1 is a system structure diagram of a method for optimizing a file access path based on heat monitoring in an integral system.
Detailed Description
The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
Referring to fig. 1, in an embodiment of the present invention, a method for optimizing a file access path based on heat monitoring in a point system is provided, where in a point data storage system, fragmented storage information of a primary copy of a file is constructed based on file metadata; and monitoring the access heat of each fragment storage node in real time, comparing the access heat with a set heat model, inquiring the lowest heat information of the copy storage node corresponding to the fragment under the condition that the access heat is higher than a heat threshold value, and updating the fragment access address in the metadata information into the copy storage node with low heat if the lowest heat of the copy storage node is lower than the heat of the current main storage node. The system comprises an integral file metadata fragment storage information module (S01), a file fragment storage node access heat monitoring module (S02), a heat evaluation model and path adjusting module (S03) and a system management configuration module (S04).
1. A score file metadata fragment storage information module (S01):
in the metadata information of the integral file, access address information stored in fragments is constructed, wherein the access address information comprises access address information of each sequential fragment, storage address information of all master copies, storage node information and master copy corresponding relation information, which are provided when external access is carried out.
2. A file fragment storage node access heat monitoring module (S02):
the module monitors the access heat of the global file fragment storage nodes. The information accessed by the node at any time is recorded in the heat monitoring data. And setting a unit time window for heat monitoring, and continuously recording and counting access heat information H (n) of each node through the time window and sending the access heat information H (n) to a heat evaluation model and a path adjusting module (S03).
3. A popularity evaluation model and path adjustment module (S03):
in the module, a node heat threshold F model is set according to the operation, operation and maintenance experience of an integral service system. The node performance can be uniformly set (for example, F0 can be set when the node performance configuration is uniform), or different node threshold settings (F (n)) can be performed according to different node performances.
When receiving all node access statistical data H (n) continuously sent by a file fragmentation storage node access heat monitoring module (S02), aiming at the node n, if:
(1) H (n) < F (n), which means no optimization is needed and no adjustment is needed;
(2) H (n) ≧ F (n), this means the heat is too high, needs to adjust optimization. At this time, firstly, the module (S01) for storing information in metadata fragments of an integral file is used to query the real-time heat information H (x) of the nodes stored in each copy corresponding to the fragment, and find out the minimum heat value MIN (H (x)) of the storage nodes of other corresponding copies:
A. if MIN (H (x)) < F (n), extracting the copy access address information corresponding to the minimum value, and updating the fragment access address information in the score file metadata fragment storage information module (S01);
B. if MIN (H (x)) > F (n), comparing the sizes of H (n) and MIN (H (x)), and if H (n) > MIN (H (x)), indicating that the minimum hot degree of a replica node is lower than the hot degree of the current main partition, extracting replica access address information corresponding to the minimum hot degree, and updating the partition access address information in the integral file metadata partition storage information module (S01); otherwise, the access heat of the current main partition is too high, but the mode is still relatively optimal compared with the heat of the node where other copies are located, and the access address is not updated.
4. System management configuration module (S04):
the module can be uniformly configured for the global module. The method comprises the step of configuring information of each fragment node, main/sub fragment storage address information and fragment access address information for external access/reading in a score file metadata fragment storage information module (S01). And configuring a time window for carrying out heat monitoring statistics on the access heat monitoring module (S02) of the file fragmentation storage node. The heat evaluation model and the heat valve value in the path adjusting module (S03) are uniformly preset, or personalized setting is carried out according to different nodes, or the module can be switched on and off, which indicates that the file path optimization processing can be started integrally or terminated integrally.
Claims (5)
1. A file access path optimization method based on heat monitoring in an integral system is characterized by comprising the following steps: in the integral data storage system, fragment storage information of a file main copy is constructed based on file metadata; monitoring the access heat of each fragment storage node in real time, comparing the access heat with a set heat model, inquiring the lowest heat information of the copy storage node corresponding to the fragment under the condition that the access heat is higher than a heat threshold value, and updating a fragment access address in the metadata information into a low-heat copy storage node if the lowest heat of the copy storage node is lower than the heat of the current main storage node; the system comprises an integral file metadata fragment storage information module, a file fragment storage node access heat monitoring module, a heat evaluation model and path adjusting module and a system management configuration module.
2. The method for optimizing the access path of the file based on the heat monitoring in the integrating system according to claim 1, wherein: the score file metadata fragment storage information module comprises: in the metadata information of the integral file, access address information stored in fragments is constructed, wherein the access address information comprises access address information of each sequential fragment, storage address information of all master copies, storage node information and master copy corresponding relation information, which are provided when external access is provided.
3. The method for optimizing the access path of the file based on the heat monitoring in the integrating system according to claim 1, wherein: the file fragment storage node access heat monitoring module comprises: the module monitors the access heat of the global file fragment storage node; the information accessed by the node at any time is recorded into the heat monitoring data; and setting a unit time window for heat monitoring, and continuously recording and counting access heat information H (n) of each node through the time window and sending the access heat information H (n) to a heat evaluation model and a path adjusting module.
4. The method for optimizing the access path of the file based on the heat monitoring in the integrating system according to claim 1, wherein: the heat evaluation model and path adjusting module: in the module, a node heat threshold value F model is set according to the operation, operation and maintenance experience of an integral service system; the node performance can be uniformly set (for example, when the node performance configuration is uniform, the node performance configuration can be set to be F0), or different node threshold settings (F (n)) are carried out according to different node performances; when receiving all node access statistical data H (n) continuously sent by a file fragmentation storage node access heat monitoring module, aiming at the node n, if: h (n) < F (n), which means no optimization is needed and no adjustment is needed; h (n) ≧ F (n), this means the heat is too high, needs to adjust and optimize. At this time, firstly, searching real-time heat information H (x) of a node stored in each copy corresponding to the segment by a module for storing information in the metadata segment of the integral file, and finding out a minimum heat value MIN (H (x)) of other corresponding copy storage nodes: if MIN (H (x)) < F (n), extracting the copy access address information corresponding to the minimum value, and updating the fragment access address information in the integral file metadata fragment storage information module; if MIN (H (x)) > F (n), comparing the sizes of H (n) and MIN (H (x)), and if H (n) > MIN (H (x)), indicating that the minimum hot degree of a replica node is lower than the hot degree of the current main partition, extracting replica access address information corresponding to the minimum hot degree, and updating the partition access address information in the integral file metadata partition storage information module; otherwise, the access heat of the current main partition is too high, but the mode is still relatively optimal compared with the heat of the node where other copies are located, and the access address is not updated.
5. The method for optimizing the file access path based on heat monitoring in the scoring system as claimed in claim 1, wherein: a system management configuration module: the module can be uniformly configured for the global module. The method comprises the step of configuring information of each fragment node, main/auxiliary fragment storage address information and fragment access address information for external access/reading in a fragment storage information module of the metadata of the integral file. Configuring a time window for carrying out heat monitoring statistics on the access heat monitoring module of the file fragment storage node; the heat evaluation model and the heat threshold value in the path adjusting module are uniformly preset, or personalized setting is carried out according to different nodes, or switch setting can be carried out on the module, so that the document path optimization processing can be started integrally or terminated integrally.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210690625.XA CN115481095A (en) | 2022-06-18 | 2022-06-18 | Heat monitoring-based file access path optimization method in integral system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210690625.XA CN115481095A (en) | 2022-06-18 | 2022-06-18 | Heat monitoring-based file access path optimization method in integral system |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN115481095A true CN115481095A (en) | 2022-12-16 |
Family
ID=84421893
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210690625.XA Pending CN115481095A (en) | 2022-06-18 | 2022-06-18 | Heat monitoring-based file access path optimization method in integral system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115481095A (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN119739915A (en) * | 2024-12-10 | 2025-04-01 | 中鉴认证有限责任公司 | Cloud storage and artificial intelligent authentication based archive management method and system |
| CN119743394A (en) * | 2024-12-23 | 2025-04-01 | 学科网(北京)股份有限公司 | Policy adjustment method, device and storage medium |
-
2022
- 2022-06-18 CN CN202210690625.XA patent/CN115481095A/en active Pending
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN119739915A (en) * | 2024-12-10 | 2025-04-01 | 中鉴认证有限责任公司 | Cloud storage and artificial intelligent authentication based archive management method and system |
| CN119743394A (en) * | 2024-12-23 | 2025-04-01 | 学科网(北京)股份有限公司 | Policy adjustment method, device and storage medium |
| CN119743394B (en) * | 2024-12-23 | 2025-09-16 | 学科网(北京)股份有限公司 | Policy adjustment method, device and storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US8972488B2 (en) | System, methods, and media for providing in-memory non-relational databases | |
| EP1892921B1 (en) | Method and system for managing distributed content and related metadata | |
| DE60131900T2 (en) | METHOD AND SYSTEM FOR MANAGING DISTRIBUTED CONTENT AND RELATED METADATA | |
| US10769126B1 (en) | Data entropy reduction across stream shard | |
| US9413671B2 (en) | Scaling redundancy elimination middleboxes | |
| CN112559459B (en) | Cloud computing-based self-adaptive storage layering system and method | |
| US11029891B2 (en) | Hybrid distributed storage system to dynamically modify storage overhead and improve access performance | |
| CN115481095A (en) | Heat monitoring-based file access path optimization method in integral system | |
| CN111651302A (en) | Distributed database backup method, device and system | |
| CN111897813B (en) | Flow control method and device for database resources | |
| US20200065306A1 (en) | Bloom filter partitioning | |
| US6973536B1 (en) | Self-adaptive hybrid cache | |
| CN110798222B (en) | Data compression method and device | |
| KR20100073154A (en) | Method for data processing and asymmetric clustered distributed file system using the same | |
| CN117806546B (en) | Data processing methods and related products | |
| CN118353910B (en) | Method and system for improving cache hit rate of edge CDN sink node | |
| CN114722078A (en) | Data statistics method, apparatus, equipment, storage medium and program product | |
| CN118643102A (en) | Data sending method, device, computer equipment and medium | |
| CN113360576A (en) | Power grid mass data real-time processing method and device based on Flink Streaming | |
| CN112667747A (en) | Dynamic configuration multi-database distributed persistence method supporting user-defined plug-in | |
| CN117909302A (en) | Global data deduplication method and system suitable for distributed object storage | |
| KR102571783B1 (en) | Search processing system performing high-volume search processing and control method thereof | |
| CN120085810B (en) | Data storage method, electronic device, storage medium, and program product | |
| CN114385079A (en) | Data writing method, data reading method and data reading and writing system | |
| CN120448460A (en) | Database cutting and storing method and system based on single data source |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication |