CN108108384B - A data storage method and device - Google Patents

Abstract

The invention discloses a data storage method and device. The method includes: extracting a set amount of data from the to-be-stored data as test data for current data to be stored; The test type for the database to be tested; according to the test type and the test data, test each database to be stored, and determine the test parameters corresponding to each database to be stored; according to the test parameters corresponding to each database to be stored and the preset performance requirements, determine a target database, and store the data to be stored in the target database. It is used to solve the problem that the database selected in the prior art for storing the data to be stored is unreasonable, cannot meet the user's requirements for the performance of the database storing the data to be stored, and affects the user's experience.

Description

A data storage method and device

technical field

The present invention relates to the technical field of data storage, and in particular, to a data storage method and device.

Background technique

With the rapid development of information technology, data is exploding, the amount of data that needs to be stored is also increasing, and the data types are becoming more and more complex. In the face of massive data, a single database can no longer meet the storage needs of all data. When the existing storage technology is used for data storage, multiple databases are generally set up. When the current database for data storage is full of data, the Start the next database for data storage.

However, in the prior art, after the data to be stored is received, the data is stored according to the storage sequence of the database, without considering the current storage environment requirements and the performance requirements of the user when storing the stored data, so the existing The data storage method in the technology causes the selected database to be unreasonable, cannot meet the performance requirements of the user and the storage environment for the database for storing the data to be stored, and affects the user's experience.

SUMMARY OF THE INVENTION

The present invention provides a data storage method and device, which are used to solve the problem that the database selected in the prior art for storing data to be stored is unreasonable, cannot meet the user's performance requirements for the database to be stored, and affects the user's experience. .

In order to achieve the above purpose, an embodiment of the present invention discloses a data storage method, which is applied to an electronic device, and the method includes:

For the current data to be stored, extract a set amount of data from the data to be stored as test data;

According to the preset performance requirements, determine the test type to be tested against the database to be stored;

According to the test type and the test data, test each database to be stored, and determine the test parameters corresponding to each database to be stored;

According to the test parameters corresponding to each database to be stored and the preset performance requirements, a target database is determined, and the data to be stored is stored in the target database.

Further, before determining the test type to be tested on the database to be stored according to the preset performance requirements, the method further includes:

According to the type of the data to be stored and the data type for each storage database, a database corresponding to the type of the data to be stored is selected as the database to be stored.

Further, the performance requirements include:

Load performance priority, storage efficiency priority, query performance priority.

Further, determining the test type to be tested on the database to be stored according to the preset performance requirements includes:

If the preset performance requirement prioritizes loading performance or storage efficiency, determining to perform a load test on the database to be stored;

If the preset performance requirement prioritizes query performance, it is determined to perform a query test on the database to be stored.

Further, if the test type is a loading test, according to the test type and the test data, each database to be stored is tested, and the test parameters corresponding to each database to be stored are determined to include:

Control each of the to-be-stored databases to load the test data, and determine the load time, occupied storage space, and average CPU utilization of each of the to-be-stored databases to load the test data.

Further, if the test type is a query test, the test is performed on each database to be stored according to the test type and the test data, and the test parameters corresponding to each database to be stored are determined to include:

Query each piece of data in the test data in each of the to-be-stored databases, and determine the average response time, average network data transmission volume, and average CPU utilization of each to-be-stored database for querying each piece of data in the test data Rate.

Further, if the preset performance requirement prioritizes loading performance, determining the target database according to the test parameters corresponding to each to-be-stored database and the preset performance requirement includes:

For each database to be stored, determine the first product result of the load time corresponding to the test data loaded into the database to be stored, the occupied storage space and the average utilization rate of the CPU;

Sort the first product results in descending order, and select a first set number of databases to be stored with smaller first product results as the first candidate database;

The database to be stored with the smallest loading time among the first candidate databases is determined as the target database.

Further, if the preset performance requirement is storage efficiency priority, determining the target database according to the test parameters corresponding to each database to be stored and the preset performance requirement includes:

For each database to be stored, determine the first product result of the load time corresponding to the test data loaded into the database to be stored, the occupied storage space and the average utilization rate of the CPU;

Sort the first product results in descending order, and select a second set number of databases to be stored with smaller first product results as the second candidate database;

The database to be stored with the smallest storage space occupied in the second candidate database is determined as the target database.

Further, if the preset performance requirement is query performance priority, determining the target database according to the test parameters corresponding to each to-be-stored database and the preset performance requirement includes:

For each database to be stored, determine the second product result of the average response time, the average network data transmission volume and the average CPU utilization rate corresponding to the query of each piece of data in the test data in the database to be stored;

Sort the second product results in descending order, and select a third set number of databases to be stored with smaller second product results as the third candidate database;

The database to be stored with the smallest average response time among the third candidate databases is determined as the target database.

The embodiment of the present invention discloses a data storage device, which is applied to electronic equipment, and the device includes:

an extraction module, for extracting a set amount of data from the data to be stored as test data for the current data to be stored;

The first determination module is used to determine the test type to be tested on the database to be stored according to the preset performance requirements;

The second determination module is used to test each database to be stored according to the test type and the test data, and determine the test parameters corresponding to each database to be stored;

The third determining module is configured to determine a target database according to the test parameters corresponding to each database to be stored and the preset performance requirements, and store the data to be stored in the target database.

Further, the device also includes:

The selecting module is configured to select the database corresponding to the type of the data to be stored as the database to be stored according to the type of the data to be stored and the data type for each storage database.

Further, the first determining module is specifically configured to determine to perform a load test on the to-be-stored database if the preset performance requirement is priority on loading performance or priority on storage efficiency; It is required that the query performance is given priority, and it is determined to perform a query test on the database to be stored.

Further, the second determination module is specifically configured to control each of the to-be-stored databases to load the test data if the test type is a load test, and to determine the loading time for each of the to-be-stored databases to load the test data , occupied storage space and average CPU utilization.

Further, the second determination module is specifically used to query each piece of data in the test data in each of the databases to be stored if the test type is a query test, and to determine that each database to be stored is responsible for the test data. Average response time, average network data transfer volume, and average CPU utilization for each piece of data in the query.

Further, the third determining module is specifically configured to determine, for each database to be stored, the loading time and occupation of the test data corresponding to the database to be stored if the preset performance requirement is priority for loading performance. The first product result of the storage space and the average utilization rate of the CPU; sort the first product results in descending order, and select the first set number of databases to be stored with smaller first product results as the first A candidate database; the database to be stored with the smallest loading time among the first candidate databases is determined as the target database.

Further, the third determining module is specifically configured to determine, for each database to be stored, the loading time and occupation of the test data corresponding to the database to be stored if the preset performance requirement is storage efficiency priority. The first product result of the storage space and the average utilization rate of the CPU; the first product result is sorted in descending order, and the second set number of databases to be stored where the first product result is smaller is selected as the first product result. Two candidate databases; the database to be stored that occupies the smallest storage space in the second candidate database is determined as the target database.

Further, the third determination module is specifically configured to, for each database to be stored, determine that each piece of data in the test data is to be performed in the database to be stored if the preset performance requirement is query performance priority. Query the second product result of the corresponding average response time, average network data transmission volume, and average CPU utilization; sort the second product results in descending order, and select the third device with the smaller second product result. A certain number of databases to be stored are used as third candidate databases; the database to be stored with the smallest average response time among the third candidate databases is determined as the target database.

The embodiment of the present invention discloses a data storage method and device. The method includes: for the current data to be stored, extracting a set amount of data from the to-be-stored data as test data; according to a preset performance requirement, determining The test type for testing the database to be stored; according to the test type and the test data, each database to be stored is tested, and the test parameters corresponding to each database to be stored are determined; according to the corresponding test parameters of each database to be stored Test parameters and the preset performance requirements, determine a target database, and store the data to be stored in the target database. Because in the embodiment of the present invention, a set amount of data is extracted from the data to be stored as test data, and according to the preset performance requirements, the corresponding test type is used to test each database to be stored, so that according to each The test parameters corresponding to the database to be stored determine the target database. Therefore, the selected target database is a database that meets the current performance requirements, thereby making data storage more reasonable and improving user experience.

Description of drawings

1 is a schematic diagram of a data storage process according to Embodiment 1 of the present invention;

2 is a schematic diagram of a data storage process according to Embodiment 2 of the present invention;

3 is a schematic diagram of a data storage process according to Embodiment 4 of the present invention;

4 is a schematic diagram of a data storage process according to Embodiment 5 of the present invention;

5 is a schematic diagram of a data storage process according to Embodiment 6 of the present invention;

FIG. 6 is a schematic structural diagram of a data storage device according to Embodiment 7 of the present invention.

Detailed ways

The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, but not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

Example 1:

FIG. 1 is a schematic diagram of a data storage process provided by an embodiment of the present invention, and the process includes:

S101: For the current data to be stored, extract a set amount of data from the data to be stored as test data.

The data storage method provided by the embodiment of the present invention is applied to an electronic device, and the electronic device may be a personal computer, a tablet computer, or a server and other devices.

Specifically, for the current data to be stored, a set amount of data is extracted from the data to be stored as test data, and the set number can be 5, 10, etc. For example, the current data to be stored contains a total of 100 pieces of data Data, the set number is 10, and 10 pieces of data are randomly extracted from the data to be stored as test data. Of course, the set number can also be the ratio of extracting the data to be stored, for example: the received data to be stored The data contains a total of 100 pieces of data, the set quantity is 10% of the data to be stored, and 10% of the data is randomly extracted from the data to be stored, that is, 10 pieces of data are randomly extracted as test data.

In the embodiment of the present invention, randomly extracting a set amount of data from the to-be-stored data as the test data is the prior art, which is not repeated in the embodiment of the present invention.

S102: Determine a test type to be tested for the database to be stored according to a preset performance requirement.

A database is a warehouse that organizes, stores, and manages data according to data structures. Between databases of different types or different architectures of the same type, because the architecture of the database is different and the structure of the stored data is different, when different databases load or read the same data, the load time, read time, and memory usage are different. Parameters such as rate will also be different.

In the embodiment of the present invention, a variety of performance requirements are stored in the electronic device. When receiving data to be stored, the electronic device can select the performance requirements according to the current storage environment and the number of data pieces contained in the data to be stored, For example, if the storage space of the database to be stored is about to be filled up, the selected performance requirement is storage efficiency first, so as to achieve the purpose of minimizing the storage space of data. If the number of data to be stored exceeds the set threshold, the selected The performance requirements are given priority to query performance, so as to achieve the goal of improving data query efficiency. Of course, users can also choose performance requirements according to their needs. Each performance requirement corresponds to a type of test. For example, if the performance requirement is query performance priority, it corresponds to the query test, and if the performance requirement is read memory performance priority, it corresponds to the read test.

Specifically, the test type to be tested for the database to be stored is determined according to a preset performance requirement. For example, the preset performance requirement is that memory performance is prioritized during reading, and it is determined to perform a read test on the database to be stored.

S103: Test each database to be stored according to the test type and the test data, and determine test parameters corresponding to each database to be stored.

Specifically, the test parameter is the operation parameter of the electronic device where the database to be stored is located when each database to be stored is tested according to the test type and the test data, and the operation parameter can reflect that the database is running on the database to be stored. The influence on the performance of electronic equipment when the corresponding operation is performed on the test data.

The test parameters may be, for example, parameters such as the average usage rate of a central processing unit (Central Processing Unit, CPU), the average occupancy of memory, and the like. If the test type is a read test, and the database to be stored includes database A, database B, database C, and database D, store the test data in database A, database B, database C, and database D respectively, and record database A. , database B, database C, and database D to read parameters such as the reading time, memory occupancy, and average CPU usage corresponding to the test data.

S104: Determine a target database according to the test parameters corresponding to each to-be-stored database and the preset performance requirements, and store the to-be-stored data in the target database.

Specifically, specific parameters in the test parameters corresponding to each of the databases to be stored can be compared according to preset performance requirements to determine the target database. For example, the databases to be stored include database A, database B, and database C. The parameters corresponding to database A are read time 1 second, average CPU utilization 10%, memory usage 30M, database B corresponding parameters are read time 2 seconds, average CPU utilization 15%, memory usage 30M, database C The corresponding parameters are the read time of 2.5 seconds, the average CPU utilization rate of 10%, and the memory footprint of 30M. The preset performance requirements are read time priority. Select the read times corresponding to database A, database B, and database C respectively. Compare and determine the database A with the smallest read time as the target database.

Of course, it is also possible to comprehensively compare the test parameters corresponding to each of the databases to be stored according to the preset performance requirements to determine the target database, for example: the databases to be stored include database A, database B, database C, and database A The corresponding parameters are the read time of 1 second, the average CPU utilization rate of 10%, and the memory usage of 30M. The parameters corresponding to database B are the read time of 2 seconds, the average CPU utilization rate of 15%, and the memory usage of 30M. Database C corresponds to The parameters are the read time of 2.5 seconds, the average CPU utilization rate of 10%, and the memory footprint of 30M. The preset performance requirements are read time priority. Compare the products of the test parameters corresponding to database A, database B, and database C respectively. , and determine the database A with the smallest product of the test parameters as the target database.

Because in the embodiment of the present invention, a set amount of data is extracted from the data to be stored as test data, and according to the preset performance requirements, the corresponding test type is used to test each database to be stored, so that according to each The test parameters corresponding to the database to be stored determine the target storage database. Therefore, the selected target storage database is a database that meets the current performance requirements, thereby making data storage more reasonable and improving user experience.

Example 2:

In order to accurately determine the target database for data storage, on the basis of the above embodiments, in this embodiment of the present invention, before determining the test type to be tested on the database to be stored according to the preset performance requirements, the Methods also include:

According to the type of the data to be stored and the data type for each storage database, a database corresponding to the type of the data to be stored is selected as the database to be stored.

Depending on the architecture of the database, the database includes: key-value storage database, document database, column storage database and graph database. Among them, the key-value storage database mainly uses a hash table, which has a specific key in the hash table. and a pointer to specific data. The advantage of this type of database is that it is simple and easy to deploy. Typical key-value storage databases include: Redis, Memcached, Dynamo, Voldemort, and LevelDB. Configuration data, log data, etc. Column storage databases are usually used to deal with massive data stored in distributed storage. The keys still exist, but the characteristics of column storage databases are that they point to multiple columns. Typical column storage databases include: HBase, Cassandra, this type of database mainly targets data types including billing data, detailed data, etc. Graph databases are different from other row-column and rigid-structured databases. Graph databases use a flexible graph model and can be extended to multiple servers. On the other hand, typical graph databases include Neo4J, Infinite Graph, and OrientDB. The data types of this type of database mainly include social data, relational data, etc.; the data structure requirements of document databases are not strict, and the table structure is variable. Like relational databases, the table structure needs to be pre-defined. Typical document databases include MongoDB and CouchDB. This type of database mainly targets data types including website data and cached data.

Because each database targets different data types, the selected data type to be stored corresponds to the data type to be stored, so as to better meet the user's requirements for storage performance when storing the data to be stored, and give full play to the storage performance. Storage database performance. The identification of the type of data to be stored is in the prior art, and will not be described repeatedly in this embodiment of the present invention.

Specifically, according to the type of the data to be stored and the data type for each storage database, a database corresponding to the type of the data to be stored is selected as the database to be stored. For example, the type of data to be stored is billing data, and the databases HBase and Cassandra for billing data are selected as the databases to be stored.

2 is a schematic diagram of a data storage process provided by an embodiment of the present invention, and the process includes:

S201: For the current data to be stored, extract a set amount of data from the data to be stored as test data.

S202: According to the type of the data to be stored and the data type for each storage database, select a database corresponding to the type of the data to be stored as the database to be stored.

S203: Determine the test type to be tested for the database to be stored according to the preset performance requirements.

S204: Test each database to be stored according to the test type and the test data, and determine test parameters corresponding to each database to be stored.

S205: Determine a target database according to the test parameters corresponding to each to-be-stored database and the preset performance requirements, and store the to-be-stored data in the target database.

In the above-mentioned steps S202 and S201, the process of extracting a set amount of data from the data to be stored as the test data, the order before and after can be unlimited, and the database corresponding to the type of the data to be stored can be selected as the data to be stored. The database can also extract a set amount of data from the to-be-stored data as test data.

Example 3:

In this embodiment of the present invention, the performance requirements include:

Load performance priority, storage efficiency priority, query performance priority.

Loading performance priority means that when selecting the target database, the load performance of the database needs to be given priority; storage efficiency priority means that when selecting the target database, the storage efficiency of the database needs to be given priority; query performance priority means that when selecting the target database, the database storage efficiency needs to be considered query performance.

In order to understand the corresponding performance requirements of each database to be stored, it is necessary to perform corresponding tests on the database to be tested. Because each performance requirement is different, the types of tests performed on the database to be tested are also different. Specifically, in the embodiment of the present invention, determining the test type to be tested on the database to be stored according to the preset performance requirements includes:

If the preset performance requirement prioritizes loading performance or storage efficiency, determining to perform a load test on the database to be stored;

If the preset performance requirement prioritizes query performance, it is determined to perform a query test on the database to be stored.

Specifically, if the preset performance requirement is priority for loading performance or storage efficiency, it means that the user has higher requirements for loading performance or storage performance when loading the data to be stored in the database to be stored. In order to determine the loading performance and storage performance of the database to be stored, Need to load test the database to be stored.

If the preset performance requirement is query performance priority, it means that the user has higher query performance requirements for querying the to-be-stored data in the stored database, and it is necessary to perform a query test on the to-be-stored database.

Example 4:

In order to accurately select a target database that satisfies the preset performance requirements, on the basis of the above embodiments, in this embodiment of the present invention, if the preset performance requirements are loading performance priority, the determined test type is loading test , the test is performed on each database to be stored according to the test type and the test data, and the test parameters corresponding to each database to be stored are determined to include:

Control each of the to-be-stored databases to load the test data, and determine the load time, occupied storage space, and average CPU utilization of each of the to-be-stored databases to load the test data.

Specifically, each of the to-be-stored databases is controlled to load the test data, because the test data may contain multiple pieces of data, so for each to-be-stored database, the loading time of the to-be-stored database in the process of loading the test data is counted , the average utilization rate of the CPU, and determine the size of the storage space occupied by the test data in the database to be stored.

If the preset performance requirement is that the loading performance is the priority, use the test data to perform a load test on each database to be stored, and after obtaining the test data of the database to be tested, according to the test parameters corresponding to each database to be stored and The preset performance requirements and determining the target database include:

For each database to be stored, determine the first product result of the load time corresponding to the test data loaded into the database to be stored, the occupied storage space and the average utilization rate of the CPU;

Sort the first product results in descending order, and select a first set number of databases to be stored with smaller first product results as the first candidate database;

The database to be stored with the smallest loading time among the first candidate databases is determined as the target database.

Specifically, in the embodiment of the present invention, when determining the target database, for each database to be stored, determine the first of the load time, the storage space occupied, and the average utilization rate of the CPU corresponding to the database to be stored when loading the test data. Product results, sort the first product results in descending order, and select a first set number of databases to be stored with a smaller first product result as the first candidate database, that is, selecting and loading the test data consumes less resources. A small first set number of databases to be stored is used as the first candidate database, and the set number may be 2, 3, 5, and so on. Among the first candidate databases that consume less resources for loading the test data, a database to be stored with the shortest loading time is selected as the target database. For example: the result of the first product corresponding to the database A to be stored is 2, the result of the first product corresponding to the database B to be stored is 2.1, the result of the first product corresponding to the database C to be stored is 2.5, and the result of the first product corresponding to the database D to be stored is 3, The first set number is 2, and the first product results corresponding to the database A to be stored, the database B to be stored, the database C to be stored, and the database D to be stored are sorted in descending order, and the first product result is selected to be smaller The database A to be stored and database B to be stored are taken as the first candidate database, and the corresponding loading times of database A to be stored and database B to be stored are 0.1 seconds and 0.11 seconds respectively, and database A to be stored with the smallest loading time is selected as the target database .

3 is a schematic diagram of a data storage process provided by an embodiment of the present invention, and the process includes:

S301: For the current data to be stored, extract a set amount of data from the data to be stored as test data.

S302: According to the type of the data to be stored and the data type for each storage database, select a database corresponding to the type of the data to be stored as the database to be stored.

S303: If the preset performance requirement is that the loading performance is the priority, control each of the to-be-stored databases to load the test data, and determine the load time, occupied storage space and Average CPU utilization.

S304: For each database to be stored, determine a first product result of the loading time corresponding to loading the test data in the database to be stored, the occupied storage space and the average utilization rate of the CPU.

S305: Sort the first product results in descending order, and select a first set number of databases to be stored with smaller first product results as the first candidate database.

S306: Determine the to-be-stored database with the smallest loading time among the first candidate databases as the target database, and store the to-be-stored data in the target database.

In the above steps, in S302 and S301, the process of extracting a set amount of data from the data to be stored is used as the test data. The sequence before and after is not limited, and the database corresponding to the type of the data to be stored can be selected first as the data to be stored. The database can also extract a set amount of data from the to-be-stored data as test data.

Example 5:

In order to accurately select a target database that satisfies the preset performance requirements, on the basis of the above embodiments, in this embodiment of the present invention, if the preset performance requirement is storage efficiency, the test type is determined to be a load test, so According to the test type and the test data, each database to be stored is tested, and the test parameters corresponding to each database to be stored are determined to include:

Control each of the to-be-stored databases to load the test data, and determine the load time, occupied storage space, and average CPU utilization of each of the to-be-stored databases to load the test data.

Specifically, each of the to-be-stored databases is controlled to load the test data, because the test data may contain multiple pieces of data, so for each to-be-stored database, the loading time of the to-be-stored database in the process of loading the test data is counted , the average utilization rate of the CPU, and determine the size of the storage space occupied by the test data in the database to be stored.

If the preset performance requirement is that the loading performance is the priority, use the test data to perform a load test on each database to be stored, and after obtaining the test data of the database to be tested, according to the test parameters corresponding to each database to be stored and The preset performance requirements and determining the target database include:

For each database to be stored, determine the first product result of the load time corresponding to the test data loaded into the database to be stored, the occupied storage space and the average utilization rate of the CPU;

Sort the first product results in descending order, and select a second set number of databases to be stored with smaller first product results as the second candidate database;

The database to be stored with the smallest storage space occupied in the second candidate database is determined as the target database.

Specifically, in the embodiment of the present invention, when determining the target database, for each database to be stored, determine the first product of the load time corresponding to loading the test data in the database to be stored, the occupied storage space and the average utilization rate of the CPU As a result, the first product results are sorted in descending order, and the second set number of databases to be stored with smaller first product results are selected as the second candidate database, that is, the resource consumption of loading the test data is less. The second set number of databases to be stored is used as the second candidate database, and the set number can be 2, 3, 5, etc. In the second candidate database that consumes less resources for loading the test data, the database to be stored with the smallest occupied storage space is selected as the target database. For example: the result of the first product corresponding to the database A to be stored is 2, the result of the first product corresponding to the database B to be stored is 2.1, the result of the first product corresponding to the database C to be stored is 2.5, and the result of the first product corresponding to the database D to be stored is 3, The second set number is 2, the first product results corresponding to the database A to be stored, the database B to be stored, the database C to be stored, and the database D to be stored are sorted in descending order, and the first product result is selected to be smaller The to-be-stored database A and the to-be-stored database B are used as the second candidate database, and the occupied storage space corresponding to the to-be-stored database A and the to-be-stored database B are 600 bytes and 550 bytes respectively, and the database to be stored with the smallest occupied storage space is selected. B as the target database.

4 is a schematic diagram of a data storage process provided by an embodiment of the present invention, and the process includes:

S401: For the current data to be stored, extract a set amount of data from the data to be stored as test data.

S402: According to the type of the data to be stored and the data type for each storage database, select a database corresponding to the type of the data to be stored as the database to be stored.

S403: If the preset performance requirement is storage efficiency, control each of the to-be-stored databases to load the test data, and determine the load time, occupied storage space and CPU for loading the test data from each of the to-be-stored databases average utilization.

S404: For each database to be stored, determine a first product result of the loading time corresponding to loading the test data in the database to be stored, the occupied storage space, and the average utilization rate of the CPU.

S405: Sort the first product results in descending order, and select a second set number of databases to be stored with smaller first product results as second candidate databases.

S406: Determine the database to be stored with the smallest storage space occupied in the second candidate database as the target database, and store the data to be stored in the target database.

In the above steps S402 and S401, the process of extracting a set amount of data from the data to be stored as the test data, the order before and after can be unlimited, and the database corresponding to the type of the data to be stored can be selected as the data to be stored. The database can also extract a set amount of data from the to-be-stored data as test data.

Example 6:

In order to accurately select the target database that meets the preset performance requirements, on the basis of the above embodiments, in this embodiment of the present invention, if the preset performance requirements are query performance priority, the test type is determined to be query test, According to the test type and the test data, each database to be stored is tested, and the test parameters corresponding to each database to be stored are determined to include:

Query each piece of data in the test data in each of the to-be-stored databases, and determine the average response time, average network data transmission volume, and average CPU utilization of each to-be-stored database for querying each piece of data in the test data Rate.

Specifically, because the test data may contain multiple pieces of data, each piece of data in the test data is queried in each database to be stored, and the data in the process of querying each piece of data in the test data in the database to be stored is counted. Average response time, average network data transfer, and average CPU utilization.

If the preset performance requirement is query performance priority, use the test data to perform a query test on each database to be stored, and after obtaining the test data of the database to be tested, the test parameters corresponding to each database to be stored and The preset performance requirements and determining the target database include:

For each database to be stored, determine the second product result of the average response time, the average network data transmission volume and the average CPU utilization rate corresponding to the query of each piece of data in the test data in the database to be stored;

Sort the second product results in descending order, and select a third set number of databases to be stored with smaller second product results as the third candidate database;

The database to be stored with the smallest average response time among the third candidate databases is determined as the target database.

Specifically, in the embodiment of the present invention, when determining the target database, for each database to be stored, the average response time and the average network data transmission volume corresponding to the query for each piece of data in the test data in the database to be stored are determined. and the second product result of the average CPU utilization rate, sort the second product results in descending order, and select the third set number of databases to be stored with smaller second product results as the third candidate database, that is, select A third set number of databases to be stored with a smaller average resource consumption per piece of data in the test data is queried as a third candidate database, and the set number may be 2, 3, 5, or the like. In querying the third candidate database where the average resource consumption of each piece of data in the test data is smaller, the database with the smallest average response time is selected as the target database. For example: the result of the second product corresponding to the database A to be stored is 3, the result of the second product corresponding to the database B to be stored is 2.8, the result of the second product corresponding to the database C to be stored is 2.6, and the result of the second product corresponding to the database D to be stored is 2.5, The third set number is 2, the second product results corresponding to the to-be-stored database A, the to-be-stored database B, the to-be-stored database C, and the to-be-stored database D are sorted in descending order, and the second product result is selected to be smaller The database C to be stored and the database D to be stored are taken as the third candidate database, and the average response times corresponding to the database C to be stored and the database D to be stored are 0.11 seconds and 0.1 seconds respectively, and the database D to be stored with the smallest average response time is selected as target database.

5 is a schematic diagram of a data storage process provided by an embodiment of the present invention, and the process includes:

S501: For the current data to be stored, extract a set amount of data from the data to be stored as test data.

S502: According to the type of the data to be stored and the data type for each storage database, select a database corresponding to the type of the data to be stored as the database to be stored.

S503: If the preset performance requirement is query performance priority, query each piece of data in the test data in each of the databases to be stored, and determine that each database to be stored is queried for each piece of data in the test data average response time, average network data transfer volume, and average CPU utilization.

S504: For each database to be stored, determine the second product result of the average response time, the average network data transmission amount, and the average CPU utilization rate corresponding to the query for each piece of data in the test data in the database to be stored.

S505: Sort the second product results in descending order, and select a third set number of databases to be stored with smaller second product results as the third candidate database.

S506: Determine the database to be stored with the smallest average response time in the third candidate database as the target database, and store the data to be stored in the target database.

In the above steps, in S502 and S501, the process of extracting a set amount of data as the test data from the data to be stored, the order before and after is not limited, and the database corresponding to the type of the data to be stored can be selected as the data to be stored. The database can also extract a set amount of data from the to-be-stored data as test data.

Example 7:

6 is a schematic structural diagram of a data storage device according to an embodiment of the present invention, where the device includes:

The extraction module 61 is used for extracting a set amount of data as test data from the data to be stored for the current data to be stored;

The first determination module 63 is configured to determine the test type to be tested on the database to be stored according to the preset performance requirements;

The second determination module 64 is configured to test each database to be stored according to the test type and the test data, and determine the test parameters corresponding to each database to be stored;

The third determining module 65 is configured to determine a target database according to the test parameters corresponding to each database to be stored and the preset performance requirements, and store the data to be stored in the target database.

The device also includes:

The selecting module 62 is configured to select a database corresponding to the type of the data to be stored as the database to be stored according to the type of the data to be stored and the data type for each storage database.

The first determining module 63 is specifically configured to determine to perform a load test on the to-be-stored database if the preset performance requirement is the priority of loading performance or the priority of storage efficiency; if the preset performance requirement is Query performance is prioritized, and it is determined to perform a query test on the database to be stored.

The second determination module 64 is specifically configured to control each of the to-be-stored databases to load the test data if the test type is a load test, and to determine the loading time and occupation of each of the to-be-stored databases to load the test data. The average utilization of storage space and CPU.

The second determination module 64 is specifically configured to query each piece of data in the test data in each of the databases to be stored if the test type is a query test, and determine whether each database to be stored is responsible for each piece of the test data. The average response time, average network data transfer volume, and average CPU utilization for querying data from a single piece of data.

The third determining module 65 is specifically configured to determine, for each database to be stored, the loading time and storage occupied corresponding to loading the test data in the database to be stored if the preset performance requirement is priority for loading performance. The first product result of the average utilization rate of space and CPU; sort the first product results in descending order, and select the first set number of databases to be stored with smaller first product results as the first candidate database; determining the database to be stored with the smallest loading time among the first candidate databases as the target database.

The third determining module 65 is specifically configured to determine, for each database to be stored, the loading time and storage occupied corresponding to loading the test data in the database to be stored if the preset performance requirement is storage efficiency priority. The first product result of the average utilization rate of the space and the CPU; sort the first product results in descending order, and select a second set number of databases to be stored with smaller first product results as the second candidate database; determining the database to be stored with the smallest storage space occupied in the second candidate database as the target database.

The third determining module 65 is specifically configured to, for each database to be stored, determine the corresponding data in the database to be stored for querying each piece of data in the test data if the preset performance requirement is query performance priority. The second product result of the average response time, the average network data transmission volume and the average CPU utilization rate; sort the second product results in descending order, and select the third set number with the second product result smaller The database to be stored is used as the third candidate database; the database to be stored with the smallest average response time among the third candidate databases is determined as the target database.

The embodiment of the present invention discloses a data storage method and device, which are applied to electronic equipment. The method includes: for current data to be stored, extracting a set amount of data from the to-be-stored data as test data; According to the test type and the test data, each database to be stored is tested, and the corresponding test parameters of each database to be stored are determined; The test parameters corresponding to the database to be stored and the preset performance requirements are determined, a target database is determined, and the data to be stored is stored in the target database. Because in the embodiment of the present invention, a set amount of data is extracted from the data to be stored as test data, and according to the preset performance requirements, the corresponding test type is used to test each database to be stored, so that according to each The test parameters corresponding to the database to be stored determine the target database. Therefore, the selected target database is a database that meets the current performance requirements, thereby making data storage more reasonable and improving user experience.

It will be apparent to those skilled in the art that various modifications and variations can be made in the present invention without departing from the spirit and scope of the invention. Thus, provided that these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

Claims (11)

1. A data storage method is applied to an electronic device, and the method comprises the following steps:

aiming at the current data to be stored, extracting a set amount of data from the data to be stored as test data;

determining a test type for testing a database to be stored according to a preset performance requirement;

testing each database to be stored according to the test type and the test data, and determining a test parameter corresponding to each database to be stored;

determining a target database according to the test parameters corresponding to each database to be stored and the preset performance requirements, and storing the data to be stored into the target database;

wherein, if the preset performance requirement is that the loading performance is prior, the determining the target database according to the test parameters corresponding to each database to be stored and the preset performance requirement comprises:

determining a first product result of loading time corresponding to the test data loaded by the database to be stored, occupied storage space and average utilization rate of a CPU (central processing unit) for each database to be stored;

sorting the first product results in a descending order, and selecting a first set number of databases to be stored with smaller first product results as first candidate databases;

determining a database to be stored with the minimum loading time in the first candidate database as a target database;

if the preset performance requirement is that storage efficiency is first, determining a target database according to the test parameters corresponding to each database to be stored and the preset performance requirement comprises:

determining a first product result of loading time corresponding to the test data loaded by the database to be stored, occupied storage space and average utilization rate of a CPU (central processing unit) for each database to be stored;

sorting the first product results in a descending order, and selecting a second set number of databases to be stored with smaller first product results as second candidate databases;

determining the database to be stored with the minimum storage space occupied in the second candidate database as a target database;

if the preset performance requirement is that the query performance is prior, determining a target database according to the test parameters corresponding to each database to be stored and the preset performance requirement comprises:

determining a second product result of average response time, average network data transmission quantity and CPU average utilization rate corresponding to each piece of data in the test data queried in each to-be-stored database;

sorting the second product results in a descending order, and selecting a third set number of databases to be stored with smaller second product results as a third candidate database;

and determining the database to be stored with the minimum average response time in the third candidate databases as a target database.

2. The method of claim 1, wherein before determining the type of test to test the database to be stored according to the predetermined performance requirement, the method further comprises:

and selecting a database corresponding to the type of the data to be stored as the database to be stored according to the type of the data to be stored and the data type aimed at by each storage database.

3. The method of claim 1, wherein the performance requirements comprise:

the method has the advantages of priority of loading performance, priority of storage efficiency and priority of query performance.

4. The method of claim 3, wherein determining the type of test to test the database to be stored according to the predetermined performance requirement comprises:

if the preset performance requirement is that loading performance is prior or storage efficiency is prior, determining to carry out loading test on the database to be stored;

and if the preset performance requirement is that the query performance is prior, determining to perform query test on the database to be stored.

5. The method of claim 4, wherein if the test type is a load test, the testing each database to be stored according to the test type and the test data, and determining the test parameters corresponding to each database to be stored comprises:

and controlling each database to be stored to load the test data, and determining the loading time, occupied storage space and the average utilization rate of the CPU for each database to be stored to load the test data.

6. The method of claim 4, wherein if the test type is query test, the testing each database to be stored according to the test type and the test data, and determining the test parameters corresponding to each database to be stored comprises:

and inquiring each piece of data in the test data in each database to be stored, and determining the average response time, the average network data transmission quantity and the average CPU utilization rate of each database to be stored for inquiring each piece of data in the test data.

7. A data storage device, for use in an electronic device, the device comprising:

the extraction module is used for extracting a set amount of data from the data to be stored as test data aiming at the current data to be stored;

the first determining module is used for determining the test type for testing the database to be stored according to the preset performance requirement;

the second determining module is used for testing each database to be stored according to the test type and the test data and determining the test parameters corresponding to each database to be stored;

a third determining module, configured to determine a target database according to the test parameters corresponding to each to-be-stored database and the preset performance requirement, and store the to-be-stored data in the target database;

the third determining module is specifically configured to determine, for each to-be-stored database, a first product result of loading time, occupied storage space, and average utilization rate of the CPU, where the loading time, occupied storage space, and average utilization rate correspond to loading of the test data by the to-be-stored database, if the preset performance requirement is that loading performance is prioritized; sorting the first product results in a descending order, and selecting a first set number of databases to be stored with smaller first product results as first candidate databases; determining a database to be stored with the minimum loading time in the first candidate database as a target database;

if the preset performance requirement is that the storage efficiency is prior, determining a first product result of loading time corresponding to the test data loaded by the database to be stored, occupied storage space and the average utilization rate of the CPU for each database to be stored; sorting the first product results in a descending order, and selecting a second set number of databases to be stored with smaller first product results as second candidate databases; determining the database to be stored with the minimum storage space occupied in the second candidate database as a target database;

if the preset performance requirement is that the query performance is prior, determining a second product result of average response time, average network data transmission quantity and CPU average utilization rate corresponding to the query of each piece of data in the test data in each database to be stored aiming at each database to be stored; sorting the second product results in a descending order, and selecting a third set number of databases to be stored with smaller second product results as a third candidate database; and determining the database to be stored with the minimum average response time in the third candidate databases as a target database.

8. The apparatus of claim 7, wherein the apparatus further comprises:

and the selection module is used for selecting the database corresponding to the type of the data to be stored as the database to be stored according to the type of the data to be stored and the data type aimed at by each storage database.

9. The apparatus according to claim 7, wherein the first determining module is specifically configured to determine to perform a load test on the database to be stored if the preset performance requirement is a load performance priority or a storage efficiency priority; and if the preset performance requirement is that the query performance is prior, determining to perform query test on the database to be stored.

10. The apparatus according to claim 9, wherein the second determining module is specifically configured to, if the test type is a load test, control each of the databases to be stored to load the test data, and determine a load time, an occupied storage space, and an average utilization rate of the CPU for each of the databases to be stored to load the test data.

11. The apparatus according to claim 9, wherein the second determining module is specifically configured to, if the test type is a query test, query each piece of data in the test data in each of the databases to be stored, and determine an average response time, an average network data transmission amount, and an average CPU utilization rate of each database to be stored for querying each piece of data in the test data.

Images