CN117312328B - Self-adaptive bottom storage configuration method, device, system and medium - Google Patents
Self-adaptive bottom storage configuration method, device, system and medium Download PDFInfo
- Publication number
- CN117312328B CN117312328B CN202311603289.1A CN202311603289A CN117312328B CN 117312328 B CN117312328 B CN 117312328B CN 202311603289 A CN202311603289 A CN 202311603289A CN 117312328 B CN117312328 B CN 117312328B
- Authority
- CN
- China
- Prior art keywords
- storage
- performance
- detection period
- configuration
- preset
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/22—Indexing; Data structures therefor; Storage structures
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Databases & Information Systems (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Software Systems (AREA)
- Computing Systems (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a self-adaptive bottom storage configuration method, a device, a system and a medium, wherein the method comprises the following steps: detecting the type of bottom layer storage in the distributed database system at a specified time according to a preset detection period, and acquiring the storage performance of the current detection period; confirming whether the type of the bottom layer storage and/or the storage performance change or not in the current detection period; if the storage performance is changed, matching the type of the bottom layer storage and/or the storage performance with a preset configuration strategy library to obtain a matched target configuration strategy; and adaptively adjusting storage configuration parameters of the distributed database system according to the target configuration strategy. By detecting the type and the performance change of the bottom layer storage in the distributed database system, the corresponding configuration strategy is automatically matched and the storage configuration parameters are adjusted, so that the self-adaptive storage configuration based on the bottom layer storage change is realized, the storage performance is utilized to the maximum extent, and the overall performance of the distributed database system is ensured.
Description
Technical Field
The present invention relates to the field of database technologies, and in particular, to a method, an apparatus, a system, and a medium for configuring a self-adaptive underlying storage.
Background
In a distributed database system, storage performance is one of important factors affecting database performance, correct storage configuration storage plays a critical role in whether storage performance can play a role to the greatest extent, the conventional method for storage configuration generally solidifies during installation and deployment, and if the condition of bottom storage change or storage performance decay and the like occurs in the later operation period, manual intervention modification is not needed, so that the performance of the distributed database is reduced.
Disclosure of Invention
In view of the foregoing deficiencies of the prior art, an object of the present invention is to provide a method, apparatus, system and medium for adaptive underlying storage configuration applicable to financial science and technology or other related fields, which aims to implement adaptive storage configuration based on underlying storage changes, so as to maximize storage performance, and ensure overall performance of a distributed database system.
The technical scheme of the invention is as follows:
an adaptive underlying storage configuration method, comprising:
detecting the type of bottom layer storage in the distributed database system at a specified time according to a preset detection period, and acquiring the storage performance of the current detection period;
confirming whether the type of the bottom layer storage and/or the storage performance change or not in the current detection period;
if the storage performance is changed, matching the type of the bottom layer storage and/or the storage performance with a preset configuration strategy library to obtain a matched target configuration strategy;
and adaptively adjusting storage configuration parameters of the distributed database system according to the target configuration strategy.
In one embodiment, the acquiring the storage performance of the current detection period specifically includes:
and collecting and counting performance index data stored in the bottom layer, time consumption of a full flow stage of storage access and time consumption of a full flow stage of application layer access in the current monitoring period, and taking the collected and counted performance index data, the time consumption of the full flow stage of storage access and the time consumption of the full flow stage of application layer access as storage performance of the current detection period.
In one embodiment, said determining whether said storage performance has changed for the current detection period comprises:
calculating the storage performance characteristics of the current detection period according to the performance index stored in the bottom layer, the time consumption of the storage access full-flow stage and the time consumption of the application layer access full-flow stage;
comparing the storage performance characteristic of the current detection period with the previous detection period, and confirming the storage performance fluctuation value;
and when the storage performance fluctuation value exceeds a preset fluctuation range, confirming that the storage performance of the current detection period changes.
In one embodiment, the matching the type of the underlying storage and/or the storage performance with a preset configuration policy library to obtain a matched target configuration policy includes:
matching the type of the bottom layer storage with a preset configuration strategy library to obtain a configuration strategy of a matched read-write queue algorithm; and/or the number of the groups of groups,
matching the performance index stored in the bottom layer with a preset configuration strategy library to obtain a configuration strategy of matching disk queue depth and IO scheduling queue size; and/or the number of the groups of groups,
matching the time consumption of the whole flow stage of the memory access with a preset configuration strategy library to obtain a matched IO scheduling queue size and a configuration strategy of a read-write queue algorithm; and/or the number of the groups of groups,
and matching the time consumption of the application layer in the whole process of access with a preset configuration strategy library to obtain the configuration strategy of the matched database storage engine.
In one embodiment, the adaptively adjusting the storage configuration parameters of the distributed database system according to the target configuration policy specifically includes:
and according to the target configuration strategy, the storage configuration of the server where each main and standby library is located in the distributed database system is adaptively adjusted according to the sequence of the first standby library and the main library.
In one embodiment, before detecting the type of the bottom layer storage in the distributed database system at a specified time according to the preset detection period and acquiring the storage performance of the current detection period, the method further includes:
initializing storage configuration parameters of the distributed database system.
In one embodiment, the preset detection period specifically refers to: the detection is performed at intervals of a preset time, or at fixed time intervals configured in advance, or at dynamic time intervals based on historical configuration data.
An adaptive underlying storage configuration apparatus, comprising:
the detection module is used for detecting the type of the bottom layer storage in the distributed database system at the appointed time according to the preset detection period and acquiring the storage performance of the current detection period;
the change analysis module is used for confirming whether the type of the bottom layer storage and/or the storage performance change or not in the current detection period;
the configuration matching module is used for matching the type of the bottom layer storage and/or the storage performance with a preset configuration strategy library if the bottom layer storage is changed to obtain a matched target configuration strategy;
and the self-adaptive adjustment module is used for self-adaptively adjusting the storage configuration parameters of the distributed database system according to the target configuration strategy.
An adaptive underlying storage configuration system, the system comprising at least one processor; the method comprises the steps of,
a memory communicatively coupled to the at least one processor; wherein,
the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the adaptive underlying storage configuration method described above.
A non-transitory computer-readable storage medium storing computer-executable instructions that, when executed by one or more processors, cause the one or more processors to perform the adaptive underlying storage configuration method described above.
The beneficial effects are that: compared with the prior art, the embodiment of the invention automatically matches corresponding configuration strategies and adjusts storage configuration parameters by detecting the type and performance change of the bottom storage in the distributed database system, thereby realizing the self-adaptive storage configuration based on the bottom storage change, maximizing the utilization of storage performance and ensuring the overall performance of the distributed database system.
Drawings
The invention will be further described with reference to the accompanying drawings and examples, in which:
FIG. 1 is an application environment diagram of a method for configuring an adaptive underlying storage according to an embodiment of the present invention;
FIG. 2 is a flowchart of a method for configuring an adaptive underlying storage according to an embodiment of the present invention;
FIG. 3 is a flowchart of step S200 in the adaptive underlying storage configuration method according to the embodiment of the present invention;
FIG. 4 is a schematic diagram of a functional module of an adaptive underlying storage configuration device according to an embodiment of the present invention;
fig. 5 is a schematic hardware structure of an adaptive underlying storage configuration system according to an embodiment of the present invention.
Detailed Description
The present invention will be described in further detail below in order to make the objects, technical solutions and effects of the present invention more clear and distinct. It should be understood that the specific embodiments described herein are for purposes of illustration only and are not intended to limit the scope of the invention. Embodiments of the present invention are described below with reference to the accompanying drawings.
The self-adaptive bottom layer storage configuration method provided by the invention can be applied to a distributed database storage system shown in figure 1, and comprises a distributed database storage component, a storage identification module, a storage performance statistics module, a storage data acquisition module, a storage management module and a storage configuration strategy library; the distributed database storage component is responsible for lasting data of the distributed database, the storage base supports various types of storage, and the system is accessed by a unified interface; the storage identification module is responsible for identifying the storage type of the bottom layer of the storage component in the distributed database and reporting the storage type to the storage data acquisition module; the storage performance statistics module is responsible for counting time consumption of each stage in the storage access flow, performance changes such as the whole macroscopic application layer access statistics information and the like, and reporting the performance changes to the storage data acquisition module; the storage data acquisition module is responsible for reporting the acquired bottom storage architecture and performance change to the storage management module; the storage configuration policy library stores related configuration policies in various storage scenes, so that the storage management module can read and issue the configuration policies to the storage component according to the underlying storage architecture and the performance change.
Referring to fig. 2, fig. 2 is a flowchart of an embodiment of a self-adaptive underlying storage configuration method according to the present invention, where the method is applied to the distributed database storage system of fig. 1 for illustration. As shown in fig. 2, the method specifically includes the following steps:
s100, detecting the type of the bottom layer storage in the distributed database system at the appointed time according to the preset detection period, and acquiring the storage performance of the current detection period.
In this embodiment, during the running process of the distributed database system, the type of the bottom layer storage of the distributed database system is detected at a specified time according to a preset detection period, and the storage performance of the current detection period is acquired and obtained, so that the hardware attribute and performance condition of the bottom layer storage are automatically perceived after the system runs, so that flexible storage configuration adjustment is performed based on the actual running condition.
Wherein, the preset detection period specifically refers to: the detection is performed at intervals of a preset time, or at fixed time intervals configured in advance, or at dynamic time intervals based on historical configuration data. In this embodiment, the detection of the bottom storage type and the storage performance can be performed at regular preset time intervals according to the fixed detection frequency, so as to ensure that the self-adaptive bottom storage configuration is performed in time; the system can also detect at a preset fixed time according to the running characteristics of the system, for example, the system can be analyzed according to running data to run in a peak period and a valley period, and the system can detect in the valley period to complete the self-adaptive bottom storage configuration, so that the influence of the self-adaptive configuration on the normal read-write business in the peak period is reduced; the detection can be performed according to dynamic time intervals based on historical configuration data, namely the specified time of the detection is not a fixed interval or a fixed time, the detection time is flexibly changed according to the historical configuration data, for example, time intervals with different lengths are preset, the configuration adjustment frequency is determined based on the historical configuration data, if the configuration adjustment frequency is larger than a threshold value, the current possible unstable performance or frequent replacement of the bottom storage type is indicated, the length of the time interval is reduced to perform the detection, the bottom storage configuration is ensured to be adjusted in time, and if the configuration adjustment frequency is larger than the threshold value, the current system operation is indicated to be stable, the length of the time interval is increased to perform the detection, so that system resources are saved.
S200, confirming whether the type of the bottom layer storage and/or the storage performance change in the current detection period;
s300, if the storage performance is changed, matching the type of the bottom layer storage and/or the storage performance with a preset configuration strategy library to obtain a matched target configuration strategy;
s400, self-adaptively adjusting storage configuration parameters of the distributed database system according to the target configuration strategy.
In this embodiment, based on the detection result of each detection period, it is determined whether the type and/or storage performance of the underlying storage in the current detection period change, that is, whether at least one of the attribute and the storage performance of the underlying storage changes, so that when any one of the attribute and the storage performance changes, the subsequent adaptive policy matching can be performed in time.
Based on the configuration strategies under various storage scenes stored in a preset configuration strategy library, matching to obtain a target configuration strategy which is matched with the current detection period, so that under the condition that storage hardware characteristics and performance change occur after the distributed database system operates, the target configuration strategy which is most suitable for the current bottom storage type and storage performance can be automatically matched based on the changed storage environment, and the target configuration strategy is used for controlling the storage parameter configuration of the storage component, so that the target configuration strategy is issued to the storage component, the storage configuration parameters of the distributed database system are adaptively adjusted based on the target configuration strategy, the adaptive storage configuration based on the bottom storage change is realized, the storage performance is utilized to the maximum, and the overall performance of the distributed database system is ensured.
In one embodiment, step S400 specifically includes:
and according to the target configuration strategy, the storage configuration of the server where each main and standby library is located in the distributed database system is adaptively adjusted according to the sequence of the first standby library and the main library.
In this embodiment, in order to ensure data consistency of the distributed database system, when the adaptive underlying storage parameters are adjusted, the backup database is modified according to the order of the backup database and the master database, then the backup database is switched to the master database, the original master database is switched to the backup database, and on the premise of ensuring data consistency, the relevant configuration of the servers of each master and backup database in the distributed database for the storage type is modified, thereby ensuring the performance and reliability of the database.
In one embodiment, prior to step S100, the method further comprises:
initializing storage configuration parameters of the distributed database system.
In this embodiment, in the initialization period of the distributed database system, storage configuration parameters are initialized based on an initial underlying storage architecture, for example, a read-write queue algorithm of the underlying storage is configured according to a historical experience value based on the type of the initial underlying storage, a disk performance test tool is utilized to simulate a service read-write model, and configuration is validated by adjusting and setting the depth of a disk queue, the size of an IO scheduling queue, and setting the depth of a relevant queue and the size of the IO scheduling queue when the performance tool has the maximum performance.
In one embodiment, the obtaining the storage performance of the current detection period specifically includes:
and collecting and counting performance index data stored in the bottom layer, time consumption of a full flow stage of storage access and time consumption of a full flow stage of application layer access in the current monitoring period, and taking the collected and counted performance index data, the time consumption of the full flow stage of storage access and the time consumption of the full flow stage of application layer access as storage performance of the current detection period.
In this embodiment, the storage performance of the system may be represented by multi-angle storage statistics, including performance index data of the bottom layer storage, such as IOPS (number of IO operations per second), read-write bandwidth, and the like; the time consumption of the whole flow stage of the storage access, namely the time consumption of each stage of disk reading and writing, such as IO queue time consumption, drive to disk stage time consumption and the like; the application layer accesses the whole flow stage time consuming, such as the data when the application layer accesses and the read-write time consuming of the log file submitting stage. The fluctuation monitoring of the storage performance is realized by counting the time consumption of each stage in the storage access flow and the access statistical information of the whole macroscopic application layer and combining the performance index data stored in the bottom layer.
In one embodiment, as shown in fig. 3, determining in step S200 whether the storage performance of the current detection period has changed includes:
s201, calculating the storage performance characteristics of the current detection period according to the performance index stored in the bottom layer, the time consumption of the full flow stage of storage access and the time consumption of the full flow stage of application layer access;
s202, comparing the storage performance characteristic of the current detection period with the previous detection period, and confirming the storage performance fluctuation value;
and S203, when the storage performance fluctuation value exceeds a preset fluctuation range, confirming that the storage performance of the current detection period changes.
In this embodiment, since the performance index of the bottom layer storage, the time consumption of the full flow stage of the storage access, the time consumption of the full flow stage of the application layer access, and the like store statistics data, the system still has normal fluctuation when operating normally, and therefore when determining whether the storage performance changes, the storage performance characteristics are calculated based on each item of the storage statistics data to represent the storage performance change trend. Specifically, each item of storage statistical data can be converted into corresponding performance scores according to a preset mapping table, each item of performance scores are directly summarized or weighted summation is carried out on each item of performance scores according to preset weights, and the like, so that the storage performance characteristics of the current detection period are obtained. Namely, the fluctuation difference in the normal range is eliminated through the performance scores corresponding to the various storage performance parameters stored in the mapping table in different size ranges.
And (3) comparing the storage performance characteristic of the current detection period with the previous detection period along with the continuous progress of the self-adaptive bottom storage configuration, confirming the fluctuation proportion of the storage performance fluctuation value such as the storage performance characteristic, and if the fluctuation proportion exceeds the preset fluctuation range, such as-5% to 5%, confirming the change of the storage performance of the current detection period, thereby realizing the quantitative confirmation of the change of the storage performance, and timely detecting the change of the performance on the basis of normal fluctuation of storage statistical data.
In one embodiment, step S300 includes:
matching the type of the bottom layer storage with a preset configuration strategy library to obtain a configuration strategy of a matched read-write queue algorithm; and/or the number of the groups of groups,
matching the performance index stored in the bottom layer with a preset configuration strategy library to obtain a configuration strategy of matching disk queue depth and IO scheduling queue size; and/or the number of the groups of groups,
matching the time consumption of the whole flow stage of the memory access with a preset configuration strategy library to obtain a matched IO scheduling queue size and a configuration strategy of a read-write queue algorithm; and/or the number of the groups of groups,
and matching the time consumption of the application layer in the whole process of access with a preset configuration strategy library to obtain the configuration strategy of the matched database storage engine.
In this embodiment, when the detected type and storage performance of the underlying storage are matched with the preset configuration policy library, the configuration policy specifically includes a storage read-write queue algorithm, a disk queue depth, an IO scheduling queue size, etc. at the operating system level, and a configuration policy such as a buffer pool size, a redox log disk flushing policy, whether to write a disk synchronously, a dirty page proportion, and an IO performance index with the largest background process related to the database storage engine.
Wherein, the operating system level:
(1) Based on the information of the change of the storage type read from the operating system, matching the current bottom storage type with a preset configuration strategy library to obtain a configuration strategy of a matched read-write queue algorithm, for example, SATA SSD uses readline, NVME SSD uses noop and the like;
(2) Matching the detected performance indexes such as IOPS, read-write bandwidth and the like of the bottom layer storage with a preset configuration strategy library to obtain a configuration strategy of matching disk queue depth and IO scheduling queue size, so that the performance indexes are maximized;
(3) The time consumption condition of the whole process stage of the disk read-write is recorded in real time through the whole process tracking tool of the disk read-write, the time consumption of the whole process stage of the storage access is matched with a preset configuration strategy library, and the matched IO scheduling queue size and the configuration strategy of the read-write queue algorithm are obtained, so that corresponding configuration parameters can be specifically adjusted aiming at time consumption abnormal scenes. For example: the IO queue consumption time becomes large, and the IO queue size is increased; and when the consumption time of the drive to the disk stage is increased, adjusting a read-write algorithm, if the adjustment fails, reporting an alarm to a management platform, and timely reminding a manager of performing fault detection processing when a bottom storage hardware layer fails.
A distributed database storage engine level:
(1) When the iops capacity monitored by the operating system level changes, triggering the configuration of the maximum IO performance index of the background process related to the storage engine of the distributed database at the same time;
(2) Monitoring the data submitting frequency of the distributed storage engine, if the data submitting frequency is too high, for example, more than 70% of the bottom layer energy storage capacity is exceeded, adjusting the size of a storage engine buffer pool, merging IO, and reducing the IO times of a disk;
(3) The time consumption of the application layer obtained through statistics in the whole process stage is matched with a preset configuration strategy library to obtain a configuration strategy of a matched database storage engine, for example, the time consumption of the whole process stage of SQL submission is monitored, specifically, the time consumption of reading and writing in the data and log file submission stage can be used, if the time consumption of reading and writing is within a certain range of a threshold value, the transaction submission is started while refreshing the disk strategy, otherwise, the transaction submission is closed while refreshing the disk strategy and reporting the alarm at the same time, so that the data integrity can be ensured within the allowable range of the disk performance, and meanwhile, the service availability is ensured and the alarm is timely reported when the bottleneck occurs in the disk performance, and the operation and maintenance processing related faults are notified;
(4) And monitoring the performance use condition of the bottom storage, such as IOPS, read-write bandwidth and other data, if the performance use condition is lower than a threshold value, reducing the dirty page proportion of the distributed storage engine, triggering the engine to refresh the dirty page into the disk as soon as possible, otherwise, reducing the dirty page proportion, reserving the disk IO performance to important files such as a redo log and the like to a greater extent, and ensuring the overall performance and the stability of the storage engine of the distributed database.
Another embodiment of the present invention provides an adaptive underlying storage configuration apparatus, as shown in fig. 4, the apparatus 1 includes:
the detection module 11 is used for detecting the type of the bottom layer storage in the distributed database system at the appointed time according to the preset detection period and acquiring the storage performance of the current detection period;
a change analysis module 12, configured to confirm whether the type of the underlying storage and/or the storage performance change in the current detection period;
the configuration matching module 13 is configured to match the type of the bottom layer storage and/or the storage performance with a preset configuration policy library if the bottom layer storage is changed, so as to obtain a matched target configuration policy;
an adaptive adjustment module 14, configured to adaptively adjust storage configuration parameters of the distributed database system according to the target configuration policy.
The modules referred to in the present invention refer to a series of computer program instruction segments capable of completing specific functions, and are more suitable for describing the execution process of the adaptive bottom storage configuration than programs, and specific implementation manners of each module refer to the above corresponding method embodiments and are not repeated herein.
In one embodiment, the detection module 11 includes:
and the storage performance statistics unit is used for collecting and counting performance index data stored in the bottom layer, time consumption of a full flow stage of storage access and time consumption of a full flow stage of application layer access in the current monitoring period, and the performance index data, the time consumption of the full flow stage of storage access and the time consumption of the full flow stage of application layer access are used as the storage performance of the current detection period.
In one embodiment, the change analysis module 12 includes:
the performance characteristic unit is used for calculating the storage performance characteristic of the current detection period according to the performance index stored in the bottom layer, the time consumption of the storage access full-flow stage and the time consumption of the application layer access full-flow stage;
the comparison unit is used for comparing the storage performance characteristic of the current detection period with the previous detection period and confirming the storage performance fluctuation value;
and the performance change analysis unit is used for confirming that the storage performance of the current detection period changes when the storage performance fluctuation value exceeds a preset fluctuation range.
In one embodiment, the configuration matching module 13 includes:
the first matching unit is used for matching the type of the bottom layer storage with a preset configuration strategy library to obtain a configuration strategy of a matched read-write queue algorithm; and/or the number of the groups of groups,
the second matching unit is used for matching the performance index stored in the bottom layer with a preset configuration strategy library to obtain a configuration strategy of matching the depth of the disk queue and the size of the IO scheduling queue; and/or the number of the groups of groups,
the third matching unit is used for matching the time consumption of the whole flow stage of the storage access with a preset configuration strategy library to obtain a matched IO scheduling queue size and a configuration strategy of a read-write queue algorithm; and/or the number of the groups of groups,
and the fourth matching unit is used for matching the time consumption of the application layer in the whole process of access with a preset configuration strategy library to obtain the configuration strategy of the matched database storage engine.
In one embodiment, the adaptive adjustment module 14 is specifically configured to:
and according to the target configuration strategy, the storage configuration of the server where each main and standby library is located in the distributed database system is adaptively adjusted according to the sequence of the first standby library and the main library.
In one embodiment, the device 1 further comprises:
and the initialization module is used for initializing the storage configuration parameters of the distributed database system.
In one embodiment, the preset detection period specifically refers to: the detection is performed at intervals of a preset time, or at fixed time intervals configured in advance, or at dynamic time intervals based on historical configuration data.
Another embodiment of the present invention provides an adaptive underlying storage configuration system, as shown in fig. 5, the system 10 comprises:
one or more processors 110 and a memory 120, one processor 110 being illustrated in fig. 5, the processors 110 and the memory 120 being coupled via a bus or other means, the bus coupling being illustrated in fig. 5.
Processor 110 is used to implement various control logic for system 10, which may be a general purpose processor, a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field Programmable Gate Array (FPGA), a single-chip microcomputer, ARM (Acorn RISC Machine) or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or any combination of these components. Also, the processor 110 may be any conventional processor, microprocessor, or state machine. The processor 110 may also be implemented as a combination of computing devices, e.g., a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors in conjunction with a DSP, and/or any other such configuration.
The memory 120 is used as a non-volatile computer readable storage medium for storing non-volatile software programs, non-volatile computer executable programs, and modules, such as program instructions corresponding to the adaptive underlying storage configuration method in the embodiment of the present invention. The processor 110 executes various functional applications of the system 10 and data processing, i.e., implements the adaptive underlying storage configuration method in the method embodiments described above, by running non-volatile software programs, instructions, and units stored in the memory 120.
Memory 120 may include a storage program area that may store an operating system, at least one application program required for functionality, and a storage data area; the storage data area may store data created from the use of system 10, etc. In addition, memory 120 may include high-speed random access memory, and may also include non-volatile memory, such as at least one magnetic disk storage device, flash memory device, or other non-volatile solid-state storage device. In some embodiments, memory 120 may optionally include memory located remotely from processor 110, which may be connected to system 10 via a network. Examples of such networks include, but are not limited to, the internet, intranets, local area networks, mobile communication networks, and combinations thereof.
One or more units are stored in memory 120 that, when executed by one or more processors 110, perform the steps of:
detecting the type of bottom layer storage in the distributed database system at a specified time according to a preset detection period, and acquiring the storage performance of the current detection period;
confirming whether the type of the bottom layer storage and/or the storage performance change or not in the current detection period;
if the storage performance is changed, matching the type of the bottom layer storage and/or the storage performance with a preset configuration strategy library to obtain a matched target configuration strategy;
and adaptively adjusting storage configuration parameters of the distributed database system according to the target configuration strategy.
In one embodiment, the acquiring the storage performance of the current detection period specifically includes:
and collecting and counting performance index data stored in the bottom layer, time consumption of a full flow stage of storage access and time consumption of a full flow stage of application layer access in the current monitoring period, and taking the collected and counted performance index data, the time consumption of the full flow stage of storage access and the time consumption of the full flow stage of application layer access as storage performance of the current detection period.
In one embodiment, said determining whether said storage performance has changed for the current detection period comprises:
calculating the storage performance characteristics of the current detection period according to the performance index stored in the bottom layer, the time consumption of the storage access full-flow stage and the time consumption of the application layer access full-flow stage;
comparing the storage performance characteristic of the current detection period with the previous detection period, and confirming the storage performance fluctuation value;
and when the storage performance fluctuation value exceeds a preset fluctuation range, confirming that the storage performance of the current detection period changes.
In one embodiment, the matching the type of the underlying storage and/or the storage performance with a preset configuration policy library to obtain a matched target configuration policy includes:
matching the type of the bottom layer storage with a preset configuration strategy library to obtain a configuration strategy of a matched read-write queue algorithm; and/or the number of the groups of groups,
matching the performance index stored in the bottom layer with a preset configuration strategy library to obtain a configuration strategy of matching disk queue depth and IO scheduling queue size; and/or the number of the groups of groups,
matching the time consumption of the whole flow stage of the memory access with a preset configuration strategy library to obtain a matched IO scheduling queue size and a configuration strategy of a read-write queue algorithm; and/or the number of the groups of groups,
and matching the time consumption of the application layer in the whole process of access with a preset configuration strategy library to obtain the configuration strategy of the matched database storage engine.
In one embodiment, the adaptively adjusting the storage configuration parameters of the distributed database system according to the target configuration policy specifically includes:
and according to the target configuration strategy, the storage configuration of the server where each main and standby library is located in the distributed database system is adaptively adjusted according to the sequence of the first standby library and the main library.
In one embodiment, before detecting the type of the bottom layer storage in the distributed database system at a specified time according to the preset detection period and acquiring the storage performance of the current detection period, the method further includes:
initializing storage configuration parameters of the distributed database system.
In one embodiment, the preset detection period specifically refers to: the detection is performed at intervals of a preset time, or at fixed time intervals configured in advance, or at dynamic time intervals based on historical configuration data.
Embodiments of the present invention provide a non-transitory computer-readable storage medium storing computer-executable instructions for execution by one or more processors, e.g., to perform the method steps S100-S400 of fig. 2 described above.
By way of example, nonvolatile storage media can include Read Only Memory (ROM), programmable ROM (PROM), electrically Programmable ROM (EPROM), electrically erasable ROM (EEPROM), or flash memory. Volatile memory can include Random Access Memory (RAM), which acts as external cache memory. By way of illustration and not limitation, RAM may be available in many forms such as Synchronous RAM (SRAM), dynamic RAM, (DRAM), synchronous DRAM (SDRAM), double Data Rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchlink DRAM (SLDRAM), and Direct Rambus RAM (DRRAM). The disclosed memory components or memories of the operating environments described herein are intended to comprise one or more of these and/or any other suitable types of memory.
In summary, in the self-adaptive bottom layer storage configuration method, device, system and medium disclosed by the invention, the method detects the type of bottom layer storage in a distributed database system at a specified time according to a preset detection period, and acquires the storage performance of the current detection period; confirming whether the type of the bottom layer storage and/or the storage performance change or not in the current detection period; if the storage performance is changed, matching the type of the bottom layer storage and/or the storage performance with a preset configuration strategy library to obtain a matched target configuration strategy; and adaptively adjusting storage configuration parameters of the distributed database system according to the target configuration strategy. By detecting the type and the performance change of the bottom layer storage in the distributed database system, the corresponding configuration strategy is automatically matched and the storage configuration parameters are adjusted, so that the self-adaptive storage configuration based on the bottom layer storage change is realized, the storage performance is utilized to the maximum extent, and the overall performance of the distributed database system is ensured.
Of course, those skilled in the art will appreciate that implementing all or part of the above described methods may be accomplished by way of a computer program stored on a non-volatile computer readable storage medium, which when executed may comprise the steps of the above described method embodiments, to instruct related hardware (e.g., processors, controllers, etc.). The storage medium may be a memory, a magnetic disk, a floppy disk, a flash memory, an optical memory, etc.
It is to be understood that the invention is not limited in its application to the examples described above, but is capable of modification and variation in light of the above teachings by those skilled in the art, and that all such modifications and variations are intended to be included within the scope of the appended claims.
Claims (8)
1. An adaptive underlying storage configuration method, comprising:
detecting the type of bottom layer storage in the distributed database system at a specified time according to a preset detection period, and acquiring the storage performance of the current detection period;
confirming whether the type of the bottom layer storage and/or the storage performance change or not in the current detection period;
if the storage performance is changed, matching the type of the bottom layer storage and/or the storage performance with a preset configuration strategy library to obtain a matched target configuration strategy;
adaptively adjusting storage configuration parameters of the distributed database system according to the target configuration strategy;
the obtaining the storage performance of the current detection period specifically includes:
collecting and counting performance index data stored in the bottom layer, time consumption of a full flow stage of storage access and time consumption of a full flow stage of application layer access in a current monitoring period, and taking the collected and counted performance index data, the time consumption of the full flow stage of storage access and the time consumption of the full flow stage of application layer access as storage performance of the current detection period;
the determining whether the storage performance of the current detection period changes includes:
calculating the storage performance characteristics of the current detection period according to the performance index data stored in the bottom layer, the time consumption of the storage access full-flow stage and the time consumption of the application layer access full-flow stage;
comparing the storage performance characteristic of the current detection period with the previous detection period, and confirming the storage performance fluctuation value;
when the storage performance fluctuation value exceeds a preset fluctuation range, confirming that the storage performance of the current detection period changes;
converting each item of storage performance data into corresponding performance scores according to a preset mapping table, wherein the corresponding performance scores of each item of storage performance parameters in different size ranges are stored in the mapping table;
summarizing the performance scores of all the items or carrying out weighted summation on the performance scores according to preset weights to obtain the storage performance characteristics of the current detection period.
2. The method for configuring an adaptive underlying storage according to claim 1, wherein said matching the type of the underlying storage and/or the storage performance with a preset configuration policy library to obtain a matched target configuration policy includes:
matching the type of the bottom layer storage with a preset configuration strategy library to obtain a configuration strategy of a matched read-write queue algorithm; and/or the number of the groups of groups,
matching the performance index stored in the bottom layer with a preset configuration strategy library to obtain a configuration strategy of matching disk queue depth and IO scheduling queue size; and/or the number of the groups of groups,
matching the time consumption of the whole flow stage of the memory access with a preset configuration strategy library to obtain a matched IO scheduling queue size and a configuration strategy of a read-write queue algorithm; and/or the number of the groups of groups,
and matching the time consumption of the application layer in the whole process of access with a preset configuration strategy library to obtain the configuration strategy of the matched database storage engine.
3. The method for adaptively configuring the storage configuration of the bottom layer according to claim 1, wherein the adaptively adjusting the storage configuration parameters of the distributed database system according to the target configuration policy specifically comprises:
and according to the target configuration strategy, the storage configuration of the server where each main and standby library is located in the distributed database system is adaptively adjusted according to the sequence of the first standby library and the main library.
4. The method for configuring an adaptive underlying storage according to claim 1, wherein before detecting the type of the underlying storage in the distributed database system at a specified time according to a preset detection period and acquiring the storage performance of the current detection period, the method further comprises:
initializing storage configuration parameters of the distributed database system.
5. The method for configuring an adaptive underlying storage according to any one of claims 1 to 4, wherein the preset detection period specifically refers to: the detection is performed at intervals of a preset time, or at fixed time intervals configured in advance, or at dynamic time intervals based on historical configuration data.
6. An adaptive underlying storage configuration device, comprising:
the detection module is used for detecting the type of the bottom layer storage in the distributed database system at the appointed time according to the preset detection period and acquiring the storage performance of the current detection period;
the change analysis module is used for confirming whether the type of the bottom layer storage and/or the storage performance change or not in the current detection period;
the configuration matching module is used for matching the type of the bottom layer storage and/or the storage performance with a preset configuration strategy library if the bottom layer storage is changed to obtain a matched target configuration strategy;
the self-adaptive adjustment module is used for self-adaptively adjusting the storage configuration parameters of the distributed database system according to the target configuration strategy;
the obtaining the storage performance of the current detection period specifically includes:
collecting and counting performance index data stored in the bottom layer, time consumption of a full flow stage of storage access and time consumption of a full flow stage of application layer access in a current monitoring period, and taking the collected and counted performance index data, the time consumption of the full flow stage of storage access and the time consumption of the full flow stage of application layer access as storage performance of the current detection period;
the determining whether the storage performance of the current detection period changes includes:
calculating the storage performance characteristics of the current detection period according to the performance index stored in the bottom layer, the time consumption of the storage access full-flow stage and the time consumption of the application layer access full-flow stage;
comparing the storage performance characteristic of the current detection period with the previous detection period, and confirming the storage performance fluctuation value;
when the storage performance fluctuation value exceeds a preset fluctuation range, confirming that the storage performance of the current detection period changes;
converting each item of storage performance data into corresponding performance scores according to a preset mapping table, wherein the corresponding performance scores of each item of storage performance parameters in different size ranges are stored in the mapping table;
summarizing the performance scores of all the items or carrying out weighted summation on the performance scores according to preset weights to obtain the storage performance characteristics of the current detection period.
7. An adaptive underlying storage configuration system, said system comprising at least one processor; the method comprises the steps of,
a memory communicatively coupled to the at least one processor; wherein,
the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the adaptive underlying storage configuration method of any one of claims 1-5.
8. A non-transitory computer-readable storage medium storing computer-executable instructions which, when executed by one or more processors, cause the one or more processors to perform the adaptive underlying storage configuration method of any one of claims 1-5.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311603289.1A CN117312328B (en) | 2023-11-28 | 2023-11-28 | Self-adaptive bottom storage configuration method, device, system and medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311603289.1A CN117312328B (en) | 2023-11-28 | 2023-11-28 | Self-adaptive bottom storage configuration method, device, system and medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN117312328A CN117312328A (en) | 2023-12-29 |
| CN117312328B true CN117312328B (en) | 2024-03-01 |
Family
ID=89260741
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202311603289.1A Active CN117312328B (en) | 2023-11-28 | 2023-11-28 | Self-adaptive bottom storage configuration method, device, system and medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN117312328B (en) |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112559230A (en) * | 2019-09-10 | 2021-03-26 | 无锡江南计算技术研究所 | Automatic identification method for abnormal performance degradation problem of super-large scale storage system |
| WO2023185770A1 (en) * | 2022-03-28 | 2023-10-05 | 阿里巴巴（中国）有限公司 | Cloud data caching method and apparatus, device and storage medium |
Family Cites Families (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP6260407B2 (en) * | 2014-03-28 | 2018-01-17 | 富士通株式会社 | Storage management device, performance adjustment method, and performance adjustment program |
| US20180121237A1 (en) * | 2016-10-27 | 2018-05-03 | Nutanix, Inc. | Life cycle management of virtualized storage performance |
| US10740042B2 (en) * | 2016-12-30 | 2020-08-11 | Western Digital Technologies, Inc. | Scheduling access commands for data storage devices |
| US10891052B2 (en) * | 2017-06-26 | 2021-01-12 | Western Digital Technologies, Inc. | Adaptive system for optimization of non-volatile storage operational parameters |
| US11954511B2 (en) * | 2021-04-16 | 2024-04-09 | EMC IP Holding Company LLC | Automated storage system performance validation |
-
2023
- 2023-11-28 CN CN202311603289.1A patent/CN117312328B/en active Active
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112559230A (en) * | 2019-09-10 | 2021-03-26 | 无锡江南计算技术研究所 | Automatic identification method for abnormal performance degradation problem of super-large scale storage system |
| WO2023185770A1 (en) * | 2022-03-28 | 2023-10-05 | 阿里巴巴（中国）有限公司 | Cloud data caching method and apparatus, device and storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN117312328A (en) | 2023-12-29 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN105339913B (en) | Managing write performance for asymmetric memory systems | |
| US9110797B1 (en) | Correlated failure zones for data storage | |
| EP2488929B1 (en) | Memory object relocation for power savings | |
| US8645734B2 (en) | Power aware memory allocation | |
| US11137926B1 (en) | Systems and methods for automatic storage tiering | |
| CN105653591A (en) | Hierarchical storage and migration method of industrial real-time data | |
| US11775438B2 (en) | Intelligent cache warm-up on data protection systems | |
| US10949115B2 (en) | Flash memory polling | |
| CN113722072A (en) | Storage system file merging method and device based on intelligent distribution | |
| WO2021126398A1 (en) | Behavior-driven die management on solid-state drives | |
| US20180060205A1 (en) | Forecasting Resource Utilization | |
| CN119473142A (en) | A ZNS SSD garbage collection perception method based on MLP | |
| CN118245887A (en) | Distributed financial data management method and system under cloud infrastructure | |
| CN118377427A (en) | Control method and device for computing resource allocation, storage medium and electronic equipment | |
| CN116820339A (en) | Method and device for determining disk status, storage medium and electronic device | |
| US20210224205A1 (en) | Tuning data storage equipment based on comparing observed i/o statistics with expected i/o statistics which are defined by operating settings that control operation | |
| CN117312328B (en) | Self-adaptive bottom storage configuration method, device, system and medium | |
| CN119512816A (en) | Data backup method, system, electronic device and storage medium | |
| CN115827594A (en) | Time sequence data synchronization method and device, electronic equipment and storage medium | |
| CN118245350B (en) | Solid-state storage health condition detection method and device | |
| CN119537120B (en) | Verification frequency determining method, device, equipment and medium based on equipment score | |
| CN119357085B (en) | A dynamic caching method, device and medium for supply chain financial factoring orders | |
| CN116820861B (en) | Method and device for testing enterprise-level solid state disk garbage collection mechanism | |
| CN116204134B (en) | Cache management method, device, equipment, storage system and storage medium | |
| CN116630034B (en) | Wind control data processing system and method |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |