CN114090366A - Method, device and system for monitoring data - Google Patents

Abstract

The invention discloses a method, device and system for monitoring data, and relates to the technical field of computers. A specific implementation of the method includes that the server receives the indicator data collected by the client, and uses a unified interface to perform read and write operations on a third-party time series database to monitor the storage and reading of the indicator data, which overcomes the need for existing systems to The defect of one-way writing of the third-party time series database improves the efficiency of monitoring data, and improves the liquidity and utilization of indicator data; by determining the global business label, it solves the correlation of the collected indicator data caused by the irregular business label. The problem of poor performance is solved; and by putting the received indicator data into the message queue, the problem of high concurrency of massive data is solved.

Description

A method, device and system for monitoring data

technical field

The present invention relates to the field of computer technology, and in particular, to a method, device and system for monitoring data.

Background technique

As a monitoring system, the Prometheus system provides a complete data monitoring solution. Because of its ecological openness and multi-component flexibility, it has been widely deployed and applied in the field of data monitoring.

In the process of realizing the present invention, the inventor found that there are at least the following problems in the prior art:

The indicator data received by the Prometheus system is stored in the local storage medium. Due to the limitation of storage space, mass data storage faces challenges. When the indicator data is stored in a third-party database, it only supports writing to the third-party database, but does not support reading the data of the third-party database directly. The complexity, at the same time, causes the problems of poor data mobility and low data utilization.

The Prometheus system acquires indicator data in complex and diverse ways, such as acquiring data from a target data source, or acquiring indicator data stored in a data gateway, or acquiring indicator data discovered through service discovery, which results in irregular monitoring data labels. The problem of lack of correlation between the data brings difficulties to the later data correlation analysis. When collecting massive data, the Prometheus system has the defect of high concurrent data collection, and there is a problem of data loss in the process of processing massive data collection. At the same time, the centralized processing of massive data makes the Prometheus system at risk of downtime.

SUMMARY OF THE INVENTION

In view of this, embodiments of the present invention provide a method, device, and system for monitoring data. The server receives the indicator data collected by the client, and uses a unified interface to perform read and write operations on a third-party time series database to monitor the indicator data. It overcomes the defect of one-way writing to the third-party time series database in the existing system, improves the efficiency of monitoring data, and improves the liquidity and utilization of indicator data; The problem of poor correlation of the collected indicator data due to irregular business labels; and by putting the received indicator data into the message queue, the problem of high concurrency of massive data is solved.

In order to achieve the above object, according to an aspect of the embodiments of the present invention, a method for monitoring data is provided, which is applied to the Prometheus system, characterized in that it includes: receiving indicator data, and determining the indicator according to the category of the indicator data The global service tag corresponding to the data, adding the global service tag and corresponding content to each of the indicator data; forming target indicator data and storing it in the message queue; using the Prometheus server to obtain the target indicator from the message queue data, according to the set monitoring strategy, monitor the target indicator data, and store the target indicator data in a time series database to monitor the storage of the indicator data; receive a query request, according to the target in the query request indicator data, determine the time series database corresponding to the target indicator data; obtain the target indicator data from the time series database through the Prometheus server to monitor the reading of the indicator data.

Optionally, the method for monitoring data is characterized in that:

According to the format rule of the first format, the data format of the indicator data based on the second format is converted into the first format.

Optionally, the method for monitoring data is characterized in that:

When the indicator data includes a non-numeric value, the non-numeric value is converted into a corresponding number according to a predefined correspondence between non-digits and numbers.

Optionally, the method for monitoring data is characterized in that:

For the query request for acquiring the target indicator data, the operators included in the query request are converted according to the grammatical rules of the time series database.

Optionally, the method for monitoring data is characterized in that:

Based on the remote process call model, the indicator data is stored in the time series database, and the indicator data in the time series database is read.

In order to achieve the above object, according to a second aspect of the embodiments of the present invention, a method for monitoring data is provided, which is characterized by comprising: collecting indicator data, and sending all data according to a set period according to the configured global service label and network address. The indicator data and the global service label are sent to the network address.

Optionally, the method for monitoring data is characterized in that:

The indicator data was collected using the data collection software package.

Optionally, the method for monitoring data is characterized in that:

Using the registration method included in the indicator data collection software package, a custom indicator is added, and the data collection software package is used to collect indicator data corresponding to the self-defined indicator.

Optionally, the method for monitoring data is characterized in that:

Use the indicator data collection script to collect indicator data.

In order to achieve the above object, according to a third aspect of the embodiments of the present invention, a device for monitoring data is provided, which is characterized in that, when applied to the Prometheus system, it includes: a data processing module and a data reading and writing module; wherein,

The data processing module is configured to receive indicator data, determine a global business label corresponding to the indicator data according to the category of the indicator data, and add the global business label and corresponding content to each of the indicator data; The target indicator data is formed and stored in the message queue;

The data reading and writing module is used to obtain the target indicator data from the message queue by using the Prometheus server, monitor the target indicator data according to the set monitoring strategy, and store the target indicator data in a time series database to monitor the storage of the indicator data; receive a query request, and determine the time series database corresponding to the target indicator data according to the target indicator data in the query request; obtain the time series database through the Prometheus server from the time series database Target metric data to monitor the reading of the metric data.

Optionally, the device for monitoring data is characterized in that:

According to the format rule of the first format, the data format of the indicator data based on the second format is converted into the first format.

Optionally, the device for monitoring data is characterized in that:

When the indicator data includes a non-numeric value, the non-numeric value is converted into a corresponding number according to a predefined correspondence between non-digits and numbers.

Optionally, the device for monitoring data is characterized in that:

For the query request for acquiring the target indicator data, the operators included in the query request are converted according to the grammatical rules of the time series database.

Optionally, the device for monitoring data is characterized in that:

Based on the remote process call model, the indicator data is stored in the time series database, and the indicator data in the time series database is read.

In order to achieve the above object, according to a fourth aspect of the embodiments of the present invention, a device for monitoring data is provided, which is characterized by comprising: a data collection module; wherein, the data collection module is used for collecting index data, and according to the configured A global service label and a network address, and the indicator data and the global service label are sent to the network address according to a set period.

Optionally, the device for monitoring data is characterized in that:

The indicator data was collected using the data collection software package.

Optionally, the device for monitoring data is characterized in that:

Using the registration method included in the indicator data collection software package, a custom indicator is added, and the data collection software package is used to collect indicator data corresponding to the self-defined indicator.

Optionally, the device for monitoring data is characterized in that:

Use the indicator data collection script to collect indicator data.

To achieve the above object, according to a fifth aspect of the embodiments of the present invention, a system 800 for monitoring data is provided, including the device 600 for monitoring data provided in the third aspect, and the device 700 for monitoring data provided in the fourth aspect. .

In order to achieve the above object, according to a sixth aspect of the embodiments of the present invention, an electronic device for monitoring data is provided, which is characterized by comprising: one or more processors; a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any one of the above methods for monitoring data.

In order to achieve the above object, according to a seventh aspect of the embodiments of the present invention, there is provided a computer-readable medium on which a computer program is stored, wherein when the program is executed by a processor, the above-mentioned method for monitoring data is implemented any of the methods described above.

An embodiment of the above invention has the following advantages or beneficial effects: the server receives the indicator data collected by the client, and uses a unified interface to perform read and write operations on a third-party time series database to monitor the storage and reading of the indicator data, It overcomes the defect of one-way writing to the third-party time series database in the existing system, improves the efficiency of monitoring data, and improves the liquidity and utilization of indicator data; by determining the global business label, it solves the problem caused by irregular business labels. The problem of poor correlation of the collected indicator data; and by putting the received indicator data into the message queue, the problem of high concurrency of massive data is solved.

Further effects of the above non-conventional alternatives will be described below in conjunction with specific embodiments.

Description of drawings

The accompanying drawings are used for better understanding of the present invention and do not constitute an improper limitation of the present invention. in:

1 is a schematic flowchart of a method for monitoring data provided by an embodiment of the present invention;

2 is a schematic flowchart of a method for collecting indicator data provided by an embodiment of the present invention;

3 is a schematic flowchart of a monitoring data provided by an embodiment of the present invention;

Fig. 4 is the schematic diagram of existing Prometheus system;

5 is a schematic diagram of an improved existing Prometheus system provided by an embodiment of the present invention;

6 is a schematic structural diagram of an apparatus for monitoring data provided by an embodiment of the present invention;

7 is a schematic structural diagram of a device for collecting index data provided by an embodiment of the present invention;

8 is a schematic structural diagram of a system for monitoring data provided by an embodiment of the present invention;

FIG. 9 is an exemplary system architecture diagram to which an embodiment of the present invention may be applied;

FIG. 10 is a schematic structural diagram of a computer system suitable for implementing a terminal device or a server according to an embodiment of the present invention.

Detailed ways

Exemplary embodiments of the present invention are described below with reference to the accompanying drawings, which include various details of the embodiments of the present invention to facilitate understanding and should be considered as exemplary only. Accordingly, those of ordinary skill in the art will recognize that various changes and modifications of the embodiments described herein can be made without departing from the scope and spirit of the invention. Also, descriptions of well-known functions and constructions are omitted from the following description for clarity and conciseness.

As shown in FIG. 1 , an embodiment of the present invention provides a method for monitoring data by a server, and the method may include the following steps:

Step S101: Receive indicator data, determine a global business label corresponding to the indicator data according to the category of the indicator data, and add the global business label to each of the indicator data; form target indicator data and store it in a message queue middle;

Specifically, the indicator data is the data monitored by the Prometheus monitoring system, for example: including indicator data related to physical machines (for example, the temperature of routers, switches, servers, hardware failure information, etc.; also including system indicator data running on physical machines, For example, CPU utilization, memory utilization, hard disk utilization, network card traffic, TCP status, number of processes, etc.), service indicator data (for example, operating indicator data of services such as Nginx, Tomcat, PHP, MySQL, Redis, etc. used by the system ); business indicator data (indicator data generated by specific business scenarios, such as: e-commerce website, how many orders are generated per minute, etc.), it can be understood that the present invention is an improvement based on the component framework of the Prometheus system, so The scope of the monitored indicator data is similar to that processed by the Prometheus system; the format of the indicator data contained in the Prometheus system is as follows:

metric{tagk=tagv,tagk1=tagv1,...}value;

Among them, metric is the identifier of the monitored indicator, tagk and tagk1 are the names of the parameters associated with the monitored indicator; tagv and tagv1 are the values corresponding to the parameters tagk and tagk1; value is the value corresponding to the metric.

Further, according to the category of the indicator data, determine the global business label corresponding to the indicator data; wherein the category of the indicator data is to distinguish the source of the indicator data, business type, etc., for example, the indicator data of different clusters is divided into Different categories; divide the index data from different network address ranges into different categories; divide the index data of different businesses into different categories (for example: divide the index data of logistics and e-commerce into different categories); further, according to the index data , determine the global service label corresponding to the indicator data, for example, determine that the global service label corresponding to the indicator data from cluster 1 is serviceId="cluster11111"; The specific content is not limited.

Further, the global business label is added to each of the indicator data; for example: the indicator data received is identified as finishJobAvgTime, and the identification may originate from different services, such as Hbase, Hadoop, Spark, etc., through the global business label according to Category, which can and if the metric data ID is the same, distinguish the source of the collected metric data. It can be seen from this that the use of the global service tag makes the received indicator data unique and makes the indicator data relevant, which overcomes the problem of low data correlation caused by the irregularity of the indicator data tag; for example: setting the global service tag serviceID= "123456abcdef" corresponds to the indicator data of the HBase service, the global business label serviceID="789010abcdef" corresponds to the indicator data of the Hadoop service, and the above global business label is added to each indicator data in the corresponding batch of indicator data to form a target index data; the problem of poor data correlation caused by inconsistent index data labels is solved through the above steps; preferably, according to the category of index data, according to the received global business label, determine the global business label; or according to the user-defined global business label The service label configuration rule is converted into a global service label matching the configuration rule based on the received global service label. At the same time, if the client sends other business labels while sending the global business label, the server can also add other business labels to each indicator data in a batch of indicator data.

Further, the target indicator data is stored in the message queue. Specifically, the existing Prometheus system obtains the data of the target network unit or data gateway according to the set period, and cannot complete the high concurrent data collection. Moreover, the following sample data is obtained through the stress test: the relevant data generated by 40,000 data processing units The amount of data can be 219GB/day, and the number of indicators generated: 15 billion per day, about 173,000 indicators per second. Unscheduled massive data may reduce the stability of the Prometheus server. Therefore, when massive indicator data is sent to the Prometheus server Previously, the use of message queues (for example: kafka) to process indicator data, and the use of message queue caching mechanisms, overcome the problem of concurrent processing of massive data. For example, in one embodiment of the present invention, the data gateway pushes the received indicator data to the message queue by setting topic partitions (for example, setting topic1 to indicate HBase-related indicator data, and setting topic2 to indicate Hadoop-related indicator data), which is understandable. What's more, by setting the topic partition (topic), the data reading and writing of different services in the message queue (kafka) can be isolated. Preferably, the Prometheus server can obtain the indicator data from the message queue according to the set period, or according to the set period. The indicator data is pushed to the Prometheus server periodically by the message queue; thus, the problem of task blocking or poor server stability caused by the direct push of massive data to the Prometheus server is partially solved.

Still further, according to the format rule of the first format, the data format of the indicator data based on the second format is converted into the first format. Specifically, the first format is a data format supported by the existing Prometheus system, and the second format is a data format inconsistent with the data format supported by the existing Prometheus system, for example: the second format is any format that can be converted into the first format Data format, the present invention does not limit the specific format of the second format.

The data format of the indicator data may be based on the data format (ie, the first format) defined by the Prometheus system. An example of the first format is as follows:

metric{tagk=tagv,tagk1=tagv1,...}value; wherein, metric is the identifier of the monitored indicator, tagk and tagk1 are the names of the parameters associated with the monitored indicator; tagv, tagv1 are corresponding to the parameters tagk, tagk1 value; value is the value corresponding to the metric; further, the data format of the accepted indicator data can also be the second format. An example is as follows:

Among them, metric is the indicator of the monitored indicator. In the above example, metric is "query_info_12345", and tags contains the name of the parameter associated with the indicator. For example, parameter 1 is "status" and the value is "failed"; according to the format of the first format The rule is to convert the indicator data format based on the second format into the first format; for example, converting the above-mentioned indicator data example based on the second format into the first format is:

query_info_12345{status="failed", cluster="AAAA"}200

It can be understood that, according to the format rules of the first format, the received indicator data in the second format is correspondingly converted into the first format according to the corresponding label or content, and then the indicator data in the first format is further monitored and analyzed. Data analysis expands the scope of monitored indicator data, and reduces the complexity of data processing due to the unification of data formats.

Step S102: Use the Prometheus server to obtain the target indicator data from the message queue, monitor the target indicator data according to the set monitoring strategy, and store the target indicator data in a time series database to monitor the indicator Data storage; receive a query request, and determine a time series database corresponding to the target indicator data according to the target indicator data in the query request; obtain the target indicator data from the time series database through the Prometheus server to monitor The reading of the indicator data.

Specifically, the target indicator data is obtained from the message queue by using the Prometheus server (that is, the monitoring server). Preferably, the indicator data can be obtained from the message queue according to a set period, for example, the set period is set to 30 seconds; it can be understood that, The setting period is determined according to the specific business or business scenario, and the frequency and granularity of user monitoring data. For example, for indicator data with high real-time requirements, a shorter period can be set, for example, it can be set to 1 second. According to the set monitoring strategy, the target indicator data is monitored, and the Prometheus server monitors the received indicator data according to the set monitoring strategy. It can be understood that the monitoring strategy is set according to the monitored business scenarios and indicators, including setting Trigger alarm rules, set thresholds, set monitoring periods, set monitored index data, etc.; monitor the received index data according to the set monitoring strategy as the functions of the existing Prometheus server, and the present invention does not further the functions already possessed by the Prometheus server Explore.

Further, according to the description of step S101, it can be known that the received data is processed to form target indicator data with a consistent format, and the Prometheus server stores the target indicator data in a time series database to monitor the storage of the indicator data; wherein, Using HBase-based time series database OpenTSDB as the time series database, it can be understood that the indicator data can be stored in multiple time series databases according to the category of indicator data, and each time series database can be a third-party database. The data is stored in multiple third-party time series databases, which solves the problem of capacity limitation of local storage, and through this step, the storage of indicator data in multiple time series databases can be monitored.

Further, according to the rules of the time series database (OpenTSDB), it is not supported to store indicator data whose values are non-numeric, and then the value of the indicator data is converted into a corresponding number according to the predefined correspondence between non-numerics and numbers. For example, the value "active" indicating the service status is corresponding to the number 1, and "standby" is corresponding to the number 0; that is, when the indicator data contains a non-numeric value, according to the predefined non-numeric and numeric corresponding relationship, Convert the non-numeric value to the corresponding number.

Further, a query request is received, and according to the target indicator data in the query request, a time series database corresponding to the target indicator data is determined; the target indicator data is obtained from the time series database to monitor the indicator data read. Specifically, a query request for target indicator data is received, a time series database corresponding to the target indicator data is determined, and the target indicator data is acquired, so as to monitor the reading of the indicator data; compared with the existing Prometheus system, by It supports using the Prometheus server to monitor the reading of the target time series database, and obtains the indicator data in the time series database according to the request, which increases the liquidity and utilization rate of massive data, and can analyze the indicator data more conveniently to generate indicator data analysis information. At the same time, the indicator data in the time series database is directly obtained by the Prometheus server, which improves the monitoring and management efficiency of the third-party time series database, and overcomes the defect that the Prometheus server cannot read data from the third-party database in the prior art.

Due to the inconsistency between the query syntax of the time series database (OpenTSDB) and the query syntax of the data format of the Prometheus system, for example: the data format of the Prometheus system contains "=", "!=", "=～", "! ~" four operation symbols, and the data format of OpenTSDB does not contain the above operation symbols; the query request to obtain the target indicator data is converted according to the grammar rules of the time series database, for example, the "=" in the query request can be converted. " and "!=" can be converted to the filter condition operator "literal_or" in the grammar rules in OpenTSDB format; while fuzzy matching operators "=～" and "!～" do not have matching filter condition operators in OpenTSDB format, Preferably, the return result corresponding to the query request containing the fuzzy matching operation symbol can be obtained first, and then based on the return result, the OR operator is used to convert it into the grammar rules of the time series database defined by OpenTSDB, and further query operations are performed. It is understandable that after the Prometheus server obtains the target indicator data from OpenTSDB, further data calculation and processing are performed in the Prometheus server. That is, for the query request for acquiring the target indicator data, the operators included in the query request are converted according to the grammatical rules of the time series database.

Further, based on a remote process call model, the indicator data is stored in the target database, and the indicator data in the target database is read. Specifically, the monitoring server (Prometheus server) performs read and write operations on the time series database (for example: OpenTSDB) through the remote process call model (for example: gRPC); and monitors the read and write operations on the time series database, where gRPC can pass the structure The data serialization method (for example: protobuf) is used to define the interface, and the structured data serialization method can serialize the data into binary code and compress the data, thereby reducing the amount of data transmitted and improving the performance of data transmission.

As shown in FIG. 2, an embodiment of the present invention provides a method for a client to collect indicator data, and the method may include the following steps:

Step S201: Collect indicator data, and send the indicator data and the global service label to the network address according to a set period according to the configured global service label and network address.

There are two methods for the client to collect the index data as follows: wherein, the client is a server or computer used for collecting the index data; the present invention does not limit the specific equipment to which the client belongs.

The first method is to use the data collection software package to collect indicator data.

Specifically, an embodiment of the present invention is that the client uses an application developed in the java language to collect indicator data, the client uses a data collection software package (for example: javaagent.jar) to complete, and the client can use the following commands Add the following parameters to the parameters, -javaagent:{javaagent.jar}={IP:Port},labels={serviceId:abcd},file={a.yml},jobName={abc}; The parameter description is shown in Table 1;

Table 1 Java application collection indicator data parameters

Specifically, the data collection software package includes the following functions: supporting adding a global service label through startup parameters; and directly pushing indicator data to the data gateway by configuring the network address of the data gateway. Further, the client scans and collects data through the probe technology possessed by Java after the application is started by configuring the global service label and dynamically configuring the network address of the data gateway. The collected indicator data is sent to the configured data gateway through the hypertext transfer protocol. Further, the data gateway pushes the received indicator data to the message queue (kafka) through the topic information, and the Prometheus server can obtain the indicator data from the message queue. .

Further, when the client needs to send the indicator data of the custom indicator based on its own business logic, it can use the registration method included in the data collection software package (javaagent.jar) to report the data to complete the data collection. For example, you can use the method shown below to register a custom indicator metricRegistry.register(name, metricType), where metricRegistry.register is the method used for registration, and metricType is the indicator type. For example, it contains five types, namely: Gauge, Counter, Meter, Timer, Histogram; among them, Gauge: record the instantaneous value of the indicator, such as the usage of the current Java virtual machine of the service, including memory utilization, CPU utilization, thread usage status, etc.; Counter: is a counter, through Increase and decrease operations to form cumulative indicators, such as: the sum of the number of tasks submitted in a cluster, etc.; Meter: used to count the frequency of events, such as: count the network traffic in the last 1 minute, 5 minutes, and 15 minutes, for Aggregate calculation of indicators; Timer: used for statistical distribution, such as: statistics of request frequency and time-consuming data of an interface; Histogram: used to count numerical distribution of indicator data, such as: minimum, maximum, and average statistical values Value, median, 75th percentile, 90th percentile, etc. The client can define the indicator and put it in the position where the logic code throws the exception. When the exception occurs, the logic code that triggers the exception throw sends the indicator data corresponding to the indicator to the data gateway. It can be seen that the client can throw the exception in the logic code part of the exception. , call the registration method included in the data collection software package and then collect the indicator data corresponding to the custom indicator, that is, use the registration method included in the indicator data collection software package to add a custom indicator, and use the data collection software The package collects the indicator data corresponding to the custom indicator.

The second method: use the indicator data collection script to collect indicator data.

Specifically, for applications developed in non-java languages, the client can use the indicator data collection script to collect indicator data, according to the configured global service label and network address, and customize the indicator according to the data format set by the data gateway. Use hypertext transfer protocol to send the indicator data and global service label to the configured network address (for example: the network address of the data gateway), and the language used to develop the indicator data collection script can be Phython, Perl, etc. The specific content and implementation method are not limited.

As shown in FIG. 3, an embodiment of the present invention provides a flowchart of monitoring data, and the method may include the following steps:

Step S301: The client collects indicator data.

Specifically, the description that the client uses a data collection software package or a data collection script to collect indicator data is consistent with step S201, and details are not repeated here. That is, the indicator data is collected, and according to the configured global service label and network address, the indicator data and the global service label are sent to the network address according to a set period. Further, index data is collected by using a data collection software package. Using the registration method included in the indicator data collection software package, a custom indicator is added, and the data collection software package is used to collect indicator data corresponding to the self-defined indicator. In addition, the indicator data corresponding to the service identifier described in the script is collected by using the indicator data.

Step S302: The data gateway receives the indicator data from the client.

Specifically, the data gateway receives the indicator data, receives the indicator data, determines the global service label corresponding to the indicator data according to the category of the indicator data, and adds the global service label to each of the indicator data; forms a target indicator data and stored in the message queue;

The description about receiving index data and processing index data is consistent with step S101, and details are not repeated here.

Step S303: The data gateway puts the target indicator data into the message queue.

Specifically, the description about the data gateway putting the target indicator data into the message queue is consistent with step S101, and details are not repeated here.

Step S304-Step S305: Use the Prometheus server to obtain the target indicator data from the message queue, monitor the target indicator data according to the set monitoring strategy, and store the target indicator data in a time series database for monitoring. Storage of the indicator data; receiving a query request, and determining a time series database corresponding to the target indicator data according to the target indicator data in the query request; obtaining the target indicator data from the time series database through a Prometheus server , to monitor the reading of the indicator data.

The description about the Prometheus server acquiring data from the message queue and monitoring the storage and reading of the time series database is consistent with step S102, and will not be repeated here.

Figure 4 shows a schematic diagram of an existing Prometheus system;

5 shows a schematic diagram of an improved Prometheus system provided by an embodiment of the present invention;

Embodiments of the present invention will be described below by comparing FIGS. 4 and 5 .

1) In the existing Prometheus system, the indicator data is stored locally, and there is a capacity limitation in mass data storage. Although the indicator data can be stored in a third-party database (TSDB as shown in Figure 4), the interaction with the third-party database is one-way, and only one-way writing is supported, that is, the indicator data stored in the third-party database is static and flowing. It has poor performance and low data utilization, resulting in the lack of data correlation and the possible incompatibility of data formats.

Compared with Fig. 4, as shown in Fig. 5, the present invention realizes the two-way operation of the storage and reading of the time series database by the Prometheus server and monitors the storage and reading, which is used for monitoring and indicators based on multiple third-party time series databases. Data storage and reading. For example, the Prometheus server used for monitoring stores and reads the indicator data of a third-party database (for example, the TSDB shown in FIG. 5, namely the OpenTSDB described in the present invention) and monitors the storage and reading of the third-party database, Using the unified interface of the Prometheus server to read and write operations to third-party databases improves the efficiency of monitoring indicator data, improves the liquidity of indicator data and the utilization rate of indicator data, and also solves the problem of Prometheus caused by the storage of massive data on local hard disks. performance issues.

That is, use the Prometheus server to obtain the target indicator data from the message queue, monitor the target indicator data according to the set monitoring strategy, and store the target indicator data in a time series database to monitor the indicator data storage; receive a query request, and determine the time series database corresponding to the target indicator data according to the target indicator data in the query request; obtain the target indicator data from the time series database through the Prometheus server to monitor all Read the indicator data.

2) As shown in Figure 4, in the existing Prometheus system, Prometheus periodically obtains the indicator data of static configuration monitoring targets (targets) or data gateway (Pushgateway), and can also obtain data container kubernetes (k8s) through service discovery (Servicediscovery) ) indicator data. It can be seen that the methods of data collection are complex and diverse, and the index data labels used for collection may be irregular, resulting in data islands (that is, data lack of correlation), which makes the later data correlation analysis problematic; 4 It can be seen that the k8s machine is highly coupled with the monitoring service.

As shown in Figure 5, the Prometheus server can periodically obtain the data in the message queue (that is, the target indicator data). The target indicator data of the message queue (kafka) is obtained by processing the received indicator data through the data gateway (Pushgateway). The data gateway determines the global business label according to the received indicator data category and adds a global business table to fill in the indicator data. , Through this technical solution, the indicator data can be obtained uniformly and the correlation and uniqueness of the indicator data can be determined, and it can also be decoupled from k8s, that is, the indicator data from k8s is not directly collected.

That is, receiving the indicator data, determining the global service label corresponding to the indicator data according to the category of the indicator data, adding the global business label to each of the indicator data; forming target indicator data and storing it in the message queue ;

3) As can be seen from the description of 2), in an embodiment of the present invention, a message queue (kafka) is used to solve the problem of collecting high-concurrency and massive data, and the technical solution is that the present invention is based on the existing Prometheus system. Using the mechanism of message queue caching, the problems caused by the high throughput and low latency of the monitoring server can be partially solved, and at the same time, it can have good fault tolerance in the case of high concurrency.

As shown in FIG. 6, an embodiment of the present invention provides an apparatus 600 for monitoring data, including: a data processing module 601 and a data reading and writing module 601; wherein,

The data processing module 601 is configured to receive indicator data, determine a global business label corresponding to the indicator data according to the category of the indicator data, and add the global business label to each of the indicator data; form a target indicator data and stored in the message queue;

The data reading and writing module 602 is used to obtain the target indicator data from the message queue by using the Prometheus server, monitor the target indicator data according to a set monitoring strategy, and store the target indicator data in a time series database , to monitor the storage of the indicator data; receive a query request, and determine the time series database corresponding to the target indicator data according to the target indicator data in the query request; obtain the data from the time series database through the Prometheus server the target indicator data to monitor the reading of the indicator data.

Optionally, the data processing module 601 is further configured to convert the data format of the indicator data based on the second format into the first format according to the format rule of the first format.

Optionally, the data processing module 601 is further configured to convert the non-numeric value into a corresponding number according to a predefined correspondence between non-numeric values and numbers when the indicator data contains non-numeric values. .

Optionally, the data reading and writing module 602 is further configured to obtain a query request for the target indicator data, and convert the operators included in the query request according to the grammatical rules of the time series database.

Optionally, the data reading and writing module 602 is further configured to store the indicator data in the time series database based on a remote process call model, and read the indicator data in the time series database.

As shown in FIG. 7 , an embodiment of the present invention provides an apparatus 700 for monitoring data, including: a data collection module 701; wherein,

The data collection module 701 is configured to collect indicator data, and according to the configured global service label and network address, send the indicator data and the global service label to the network address according to a set period.

Optionally, the data collection module 701 is further configured to collect indicator data by using a data collection software package.

Optionally, the data collection module 701 is further configured to use the registration method included in the indicator data collection software package to add a custom indicator, and use the data collection software package to collect the corresponding custom indicators. indicator data.

Optionally, the data collection module 701 is further configured to collect indicator data by using an indicator data collection script.

As shown in FIG. 8 , an embodiment of the present invention provides a system for monitoring data, including: an apparatus for monitoring data as shown in FIG. 6 , and an apparatus for monitoring data as shown in FIG. 7 .

An embodiment of the present invention also provides an electronic device for monitoring data, including: one or more processors; a storage device for storing one or more programs, when the one or more programs are stored by the one or more programs Each processor executes, so that the one or more processors implement the method provided by any one of the foregoing embodiments.

An embodiment of the present invention further provides a computer-readable medium, on which a computer program is stored, and when the program is executed by a processor, implements the method provided by any of the foregoing embodiments.

FIG. 9 shows an exemplary system architecture 900 of a method for monitoring data or an apparatus for monitoring data to which embodiments of the present invention may be applied.

As shown in FIG. 9 , the system architecture 900 may include terminal devices 901 , 902 , and 903 , a network 904 and a server 905 . The network 904 is a medium used to provide a communication link between the terminal devices 901 , 902 , 903 and the server 905 . Network 904 may include various connection types, such as wired, wireless communication links, or fiber optic cables, among others.

The user can use the terminal devices 901, 902, 903 to interact with the server 905 through the network 904 to receive or send messages and the like. Various client applications may be installed on the terminal devices 901 , 902 and 903 , such as web browser applications, search applications, instant messaging tools, and email clients.

The terminal devices 901, 902, 903 may be various electronic devices having a display screen and supporting web browsing, including but not limited to servers, smart phones, tablet computers, laptop computers, desktop computers, and the like.

The server 905 may be a server that provides various services, for example, a background management server that provides support for data monitoring requests made by users using the terminal devices 901 , 902 , and 903 . The background management server can process the received data monitoring requests and other data, store the received indicator data, and feed back the indicator data analysis results to the terminal device.

It should be noted that the method for monitoring data provided in the embodiment of the present invention is generally executed by the server 905 , and accordingly, the apparatus for monitoring data is generally set in the server 905 .

It should be understood that the numbers of terminal devices, networks and servers in FIG. 9 are only illustrative. There can be any number of terminal devices, networks and servers according to implementation needs.

Referring to FIG. 10 below, it shows a schematic structural diagram of a computer system 1000 suitable for implementing a terminal device according to an embodiment of the present invention. The terminal device shown in FIG. 10 is only an example, and should not impose any limitations on the functions and scope of use of the embodiments of the present invention.

As shown in FIG. 10, a computer system 1000 includes a central processing unit (CPU) 1001, which can be loaded into a random access memory (RAM) 1003 according to a program stored in a read only memory (ROM) 1002 or a program from a storage section 1008 Instead, various appropriate actions and processes are performed. In the RAM 1003, various programs and data necessary for the operation of the system 1000 are also stored. The CPU 1001 , the ROM 1002 , and the RAM 1003 are connected to each other through a bus 1004 . An input/output (I/O) interface 1005 is also connected to the bus 1004 .

The following components are connected to the I/O interface 1005: an input section 1006 including a keyboard, a mouse, etc.; an output section 1007 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 1008 including a hard disk, etc. ; and a communication section 1009 including a network interface card such as a LAN card, a modem, and the like. The communication section 1009 performs communication processing via a network such as the Internet. A drive 1010 is also connected to the I/O interface 1005 as needed. A removable medium 1011, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is mounted on the drive 1010 as needed so that a computer program read therefrom is installed into the storage section 1008 as needed.

In particular, the processes described above with reference to the flowcharts may be implemented as computer software programs in accordance with the disclosed embodiments of the present invention. For example, embodiments disclosed herein include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the method illustrated in the flowchart. In such an embodiment, the computer program may be downloaded and installed from the network via the communication portion 1009, and/or installed from the removable medium 1011. When the computer program is executed by the central processing unit (CPU) 1001, the above-described functions defined in the system of the present invention are executed.

It should be noted that the computer-readable medium shown in the present invention may be a computer-readable signal medium or a computer-readable storage medium, or any combination of the above two. The computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus or device, or a combination of any of the above. More specific examples of computer readable storage media may include, but are not limited to, electrical connections with one or more wires, portable computer disks, hard disks, random access memory (RAM), read only memory (ROM), erasable Programmable read only memory (EPROM or flash memory), fiber optics, portable compact disk read only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing. In the present invention, a computer-readable storage medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In the present invention, however, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code therein. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the foregoing. A computer-readable signal medium can also be any computer-readable medium other than a computer-readable storage medium that can transmit, propagate, or transport the program for use by or in connection with the instruction execution system, apparatus, or device . Program code embodied on a computer readable medium may be transmitted using any suitable medium including, but not limited to, wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

The flowchart and block diagrams in the Figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present invention. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code that contains one or more logical functions for implementing the specified functions executable instructions. It should also be noted that, in some alternative implementations, the functions noted in the blocks may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It is also noted that each block of the block diagrams or flowchart illustrations, and combinations of blocks in the block diagrams or flowchart illustrations, can be implemented in special purpose hardware-based systems that perform the specified functions or operations, or can be implemented using A combination of dedicated hardware and computer instructions is implemented.

The modules and/or units involved in the embodiments of the present invention may be implemented in a software manner, and may also be implemented in a hardware manner. The described modules and/or units can also be provided in a processor, for example, it can be described as: a processor includes a data processing module, a data reading and writing module, and a data acquisition module. Among them, the names of these modules do not constitute a limitation of the module itself under certain circumstances. For example, the data processing module can also be described as "receiving indicator data, converting the data format of the indicator data into the first format, and creating A module for adding global business labels to indicator data".

As another aspect, the present invention also provides a computer-readable medium, which may be included in the device described in the above embodiments; or may exist alone without being assembled into the device. The above-mentioned computer-readable medium carries one or more programs, and when the above-mentioned one or more programs are executed by a device, the device includes: receiving indicator data, and determining the corresponding indicator data according to the category of the indicator data. The global business tag is added to each of the indicator data; the target indicator data is formed and stored in the message queue; the Prometheus server is used to obtain the target indicator data from the message queue, and monitor according to the settings strategy, monitor the target indicator data, and store the target indicator data in a time series database to monitor the storage of the indicator data; receive a query request, and determine the target indicator data according to the target indicator data in the query request The time series database corresponding to the target indicator data; the target indicator data is obtained from the time series database through the Prometheus server to monitor the reading of the indicator data. Collect indicator data, and send the indicator data and the global service label to the network address according to a set period according to the configured global service label and network address.

According to the technical solution of the embodiment of the present invention, the server receives the indicator data collected by the client, and uses a unified interface to perform read and write operations on the third-party time series database, so as to monitor the storage and reading of the indicator data, which overcomes the problem of the existing system. The defect of one-way storage of third-party time series databases improves the efficiency of monitoring data, and improves the liquidity and utilization of indicator data; by determining the global business label, it solves the problem of the indicator data collected due to irregular business labels. The problem of poor correlation; and by putting the received indicator data into the message queue, the problem of high concurrency of massive data is solved.

The above-mentioned specific embodiments do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions may occur depending on design requirements and other factors. Any modifications, equivalent replacements and improvements made within the spirit and principle of the present invention shall be included within the protection scope of the present invention.

Claims (14)

1. A method for monitoring data, which is applied to a Prometous system, comprises the following steps:

receiving index data, determining a global service label corresponding to the index data according to the category of the index data, and adding the global service label to each index data; forming target index data and storing the target index data in a message queue;

acquiring the target index data from the message queue by using a Prometous server, monitoring the target index data according to a set monitoring strategy, and storing the target index data in a time sequence database so as to monitor the storage of the index data;

receiving a query request, and determining a time sequence database corresponding to target index data according to the target index data in the query request; and acquiring the target index data from the time sequence database through a Prometheus server to monitor the reading of the index data.

2. The method of claim 1,

and converting the data format of the index data based on the second format into the first format according to the format rule of the first format.

3. The method of claim 1,

and when the index data contains non-numerical values, converting the non-numerical values into corresponding numbers according to a predefined corresponding relation between the non-numerical values and the numerical values.

4. The method of claim 1,

and acquiring a query request of the target index data, and converting operators contained in the query request according to grammar rules of a time sequence database.

5. The method of claim 1,

and storing the index data to the time sequence database based on a remote process calling model, and reading the index data of the time sequence database.

6. A method of monitoring data, comprising:

acquiring index data, and sending the index data and the global service label to the network address according to the configured global service label and the network address and a set period.

7. The method of claim 6,

and acquiring index data by using a data acquisition software package.

8. The method of claim 7,

and adding a user-defined index by using a registration method contained in the index data acquisition software package, and acquiring index data corresponding to the user-defined index by using the data acquisition software package.

9. The method of claim 6,

and acquiring the index data by using the index data acquisition script.

10. An apparatus for monitoring data, applied to a Prometheus system, comprising: the data processing module and the data reading and writing module; wherein,

the data processing module is used for receiving index data, determining a global service tag corresponding to the index data according to the category of the index data, and adding the global service tag and corresponding content to each index data; forming target index data and storing the target index data in a message queue;

the data reading and writing module is used for acquiring the target index data from the message queue by using a Prometous server, monitoring the target index data according to a set monitoring strategy, and storing the target index data in a time sequence database so as to monitor the storage of the index data; receiving a query request, and determining a time sequence database corresponding to target index data according to the target index data in the query request; and acquiring the target index data from the time sequence database through a Prometheus server to monitor the reading of the index data.

11. An apparatus for monitoring data, comprising: a data acquisition module; the data acquisition module is used for acquiring index data and sending the index data and the global service label to the network address according to a set period and the configured global service label and the network address.

12. A system for monitoring data, comprising: the apparatus for sharing files of claim 10, and the apparatus for sharing files of claim 11.

13. An electronic device, comprising:

one or more processors;

a storage device for storing one or more programs,

when executed by the one or more processors, cause the one or more processors to implement the method of any one of claims 1-5 or 6-9.

14. A computer-readable medium, on which a computer program is stored, which program, when being executed by a processor, is adapted to carry out the method of any one of claims 1-5 or 6-9.

Images