CN112115019A - Application log monitoring method and system for application - Google Patents

Abstract

The invention discloses an application log monitoring method and system for an application program, wherein the application log monitoring method comprises the following steps: checking the effectiveness of the collected application logs to screen out effective application logs; judging whether the screened effective application logs belong to a stack or not; according to configuration items of a preset rule index, performing predictive anomaly and non-predictive anomaly analysis on the effective application logs which do not belong to the stack; and sending alarm information related to the analysis result according to the configuration item. The invention aims at analyzing and alarming defects possibly occurring in business logic in logs in a targeted manner in real time at least.

Description

Application log monitoring method and system for application

technical field

The present invention relates to the field of computer technology, and in particular, to an application log monitoring method and system for application programs.

Background technique

Logs are a widely used concept in computer systems. Operating system kernels, various application servers, etc., programs may output logs. Logs also vary in content, size, and purpose, making it difficult to generalize. With the development of software technology, the number of lines of code in the software has become very large, and the work of maintaining the software has also become very complicated and difficult. Usually, in the process of writing the code, you can add the log recording code of the key business, so that during the software running process, the program processing of the key business is recorded in the log file. Once the software runs abnormally, you can analyze the log information by , and make corresponding processing.

The existing log analysis system is usually composed of a log collection agent and a log analysis management system, which can analyze and process a small number of log files. However, the existing log analysis system is not capable of analyzing and processing massive log files, and lacks real-time analysis, query and early warning capabilities. There is no specific function implementation on the market based on real-time retrieval and monitoring of business indicators and exception stacks based on logs.

On the other hand, with the maturity of computer technology, although the application programs running on smart devices are becoming more and more perfect, errors will inevitably occur in the actual operation process. Therefore, engineers and technicians usually need to record the running status and operation content of the application program through the log system, so that the personnel can view it and use it as a basis for debugging equipment. The log records various running status and operation information of the application, and generates log files.

However, viewing and analyzing logs are based on Linux scripts using traditional tools to detect logs. These tools and methods have the following disadvantages: they are not timely, and the problem cannot be found at the first time; they are not intuitive and not easy to expand, and their scope of application is limited to a small number of hosts and log file types. In the existing method for monitoring application logs, the application server records the application logs, regularly uploads the application log files to the remote monitoring server, and then the relevant technical maintenance personnel regularly obtain the log files from the monitoring server for analysis, so that the logs cannot be analyzed in real time and accurately. Processing to meet the real-time requirements for alarm monitoring at the application level, making log monitoring less efficient.

SUMMARY OF THE INVENTION

Aiming at the problems in the related art, the present invention proposes an application log monitoring method and system for an application program, which can analyze and alarm the possible defects of the business logic in the log in real time and pertinently.

The technical scheme of the present invention is realized as follows:

According to an aspect of the present invention, an application log monitoring method for an application is provided, comprising:

Check the validity of the collected application logs to filter out valid application logs;

judging whether the filtered valid application log belongs to the stack;

According to the configuration item of the predetermined rule indicator, perform predictive exception and non-predictive exception analysis on the valid application logs that do not belong to the stack;

Alarm information related to the analysis result is sent according to the configuration item.

According to an embodiment of the present invention, the application log monitoring method further includes: monitoring at least one of an error code type and a statistical indicator type through a visual console according to a monitoring indicator, where the monitoring indicator includes a monitoring level, whether to upgrade an alarm, a notification At least one of frequency and notification person.

According to an embodiment of the present invention, sending the alarm information includes: using a minute-level timed task to determine whether an alarm condition is satisfied.

According to an embodiment of the present invention, sending the alarm information related to the analysis result according to the configuration item includes: acquiring the result of the corresponding statistical item based on the indicator, and comparing the result of the statistical item with the threshold of the indicator ; If the result of the statistical item reaches the threshold of the indicator, send the alarm information.

According to an embodiment of the present invention, the application log monitoring method further includes: installing a collection tool locally on the application program for capturing the collection application log.

According to another aspect of the present invention, an application log monitoring system for an application program is provided, characterized in that it includes:

a collection tool, installed locally in the application program, for collecting application logs to a distributed message queue;

a log routing module, configured to check the validity of the application logs in the distributed message queue, so as to filter out valid application logs;

A log processing module, configured to judge whether the screened valid application logs belong to the stack, and perform predictive and unforeseen exception analysis on the valid application logs that do not belong to the stack according to the configuration items of the predetermined rule indicators ;

An alarm notification module, configured to send alarm information related to the analysis result according to the configuration item.

According to an embodiment of the present invention, the alarm notification module is further configured to receive a socket request or a microservice request sent by the no-log system, and the no-log application is converted into the effective application log.

According to the embodiment of the present invention, the application log monitoring system further includes a batch running module, which is used for regularly loading the configuration items and related indicators.

According to an embodiment of the present invention, the application log monitoring system further includes a visual console for monitoring at least one of error code types and statistical index types according to monitoring indicators, where the monitoring indicators include monitoring levels, whether to upgrade alarms, At least one of notification frequency and notification person.

According to an embodiment of the present invention, the application log monitoring system further includes a distributed cache or database for storing the analysis results of the log processing module.

Description of drawings

In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the accompanying drawings required in the embodiments will be briefly introduced below. Obviously, the drawings in the following description are only some of the present invention. In the embodiments, for those of ordinary skill in the art, other drawings can also be obtained according to these drawings without any creative effort.

1 is a flowchart of an application log monitoring method for an application according to an embodiment of the present invention;

2 is a flowchart of an alarming step of an application log monitoring method according to an embodiment of the present invention;

3 is a schematic diagram of the architecture of an application log monitoring system according to an embodiment of the present invention;

4 is a schematic diagram of the configuration of the error code class of the visual console of the application log monitoring system according to an embodiment of the present invention;

FIG. 5 is a schematic diagram of judging a configuration item of stack storage according to an embodiment of the present invention.

Detailed ways

The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, but not all of the embodiments. Based on the embodiments in the present invention, all other embodiments obtained by those of ordinary skill in the art fall within the protection scope of the present invention.

In view of the shortcomings in the prior art, the present invention provides an application log monitoring method and an application log monitoring system for an application program. The application log monitoring method includes the following steps S1 to S4. The application log monitoring method of the present invention will be described below with reference to FIG. 1 .

(1) Step S1

At this step S1, firstly, the validity of the collected application logs is checked to filter out valid application logs. Specifically, in step S1, the application log can be collected before judging the validity of the application log, and the regular verification of filebeat (a log data collector for local files) is performed in step S11; in step S12, it is judged that the application log is correct If the verification is passed, if the application log verification is passed, go to step S13, and upload the collected application log to the cluster, for example, to the kafka cluster. Then at step S14 in FIG. 1 , the application log of the cluster can be consumed through the log routing module, and it is determined whether the application log is valid. In the case that the reference log is judged to be valid, the method proceeds to step S2.

(2) Step S2

At this step S2, it is judged whether the filtered valid application logs belong to the stack. As shown in FIG. 1 , in the illustrated method flow example, it is determined at step S15 whether the filtered valid application logs belong to a stack. The rules for judging whether it belongs to a stack can be configured according to actual needs. When it is determined that the valid application log belongs to the stack, step S16 is performed, and the stack application log is stored; when it is determined that the valid application log does not belong to the stack, step S17 is performed to process the valid application log that needs to be monitored. The logging of the stack log at step S16 in FIG. 1 is the same as the logging of the valid log that does not belong to the stack. FIG. 5 is a schematic diagram of judging a configuration item of stack storage according to an embodiment of the present invention.

(3) Step S3

At this step S3, according to the configuration items of predetermined rule indicators, predictive exception and non-predictive exception analysis are performed on the valid application logs that do not belong to the stack. As shown in FIG. 1 , in the illustrated method flow example, it is determined whether there is a configuration item at step S18 . If there is a configuration item, go to step S19, further analyze the valid logs that have been screened and perform logical operations, match the configuration items of the established rule indicators, perform asynchronous operations, and analyze the unforeseen anomalies and foreseeable anomalies respectively, and persist the final result. According to the embodiment of the present invention, the configuration item of stack storage (stack exception) may be the same as the configuration item of the established rule indicator for monitoring effective logs.

(4) Step S4

Alarm information related to the analysis result is sent according to the configuration item. The final result that has been persisted is notified to each application responsible person through certain rules according to the indicator configuration items and related events. In some embodiments, the generated alarm information may be sent by means of email, short message, DingTalk, or voice call.

The above-mentioned application log monitoring method of the present invention can monitor the updated content in the application log file in real time and establish a corresponding processing method, so as to solve the problem that the existing method for monitoring the application log cannot detect the possible occurrence of business logic in the log in a real-time and targeted manner. It can process the system and application in real time and accurately, and meet the real-time requirements of alarm monitoring at the application level, so as to improve the monitoring efficiency and the immediacy of problem discovery. The present invention does not need to intervene in the application code, and completes the monitoring of corresponding indicators and stack exceptions. Effectively improve monitoring efficiency and save labor costs.

The above solution of the present invention can be applied to a variety of applications. For example, applications related to various payment success rates of card swiping business, bank channel deposit and withdrawal success rates, and risk control system interception.

In some embodiments, the present invention applies a minute-level timed task of the log monitoring method to determine whether an alarm condition is satisfied. As shown in Figure 2, through the minute-level timing task, when it is judged that the alarm condition is met, the task is added to the to-be-sent queue for sending. When it is judged that the alarm conditions are not met, the statistical results are cleared.

FIG. 3 is a schematic structural diagram of an application log monitoring system according to an embodiment of the present invention. The above application log monitoring method can be applied to the application log monitoring system shown in FIG. 3 . As shown in FIG. 3 , the application log monitoring system of the present invention includes a collection tool 21 , a log routing module 22 , a log processing module 23 and an alarm notification module 24 .

The collection tool 21 can be installed locally in the application, and is used to collect application logs to a distributed message queue. The log routing module 22 is configured to check the validity of the application logs in the distributed message queue, so as to filter out valid application logs. The log processing module 23 is configured to judge whether the filtered valid application log belongs to the stack, and according to the configuration item of the predetermined rule index. The log processing module 23 also performs predictive and unpredictable exception analysis on the valid application logs that do not belong to the stack. The alarm notification module 24 is configured to send alarm information related to the analysis result according to the configuration item.

In the embodiment shown in FIG. 3, when the objects are Niginx (an HTTP and reverse proxy web server) system logs and application logs in various languages (such as Java, Python, etc.), the collection tool 21 can use Logstash (a HTTP and reverse proxy web server) An application log management platform) or Filebeat components, Logstash or Filebeat components collect log file information to the distributed message queue Kafka (a distributed publish-subscribe messaging system). When the object is a log-free system, the collection tool 21 can actively send a socket (socket) request or a micro-service request to the alarm notification module 24 for processing. The log routing module 22 consumes messages from the distributed message queue Kafka to the routing application, converts them into byte objects, and verifies the validity of the logs. Valid logs can be filtered out through established rules for further analysis and processing. The log processing module 23 performs further analysis, statistics and logical operations on the selected valid logs, matches the configuration items of the established rule indicators, performs asynchronous operations, analyzes unforeseen exceptions and foreseeable exceptions respectively, and persists the final results.

In a specific example, the foreseeable exception can be, for example: a channel service is unavailable due to publishing or some other reasons, and the log can be printed without throwing an exception to obtain it through the system; in this example, the established rule indicator can be Yes: the above foreseen exceptions, if they are different occasionally, they will be alarmed according to stack exceptions, and statistics can also be made by defining error codes, for example, an alarm will be issued when a specific number of times occurs within 1 minute; in this example, the configuration items can include: The service connection fails Number of times, transaction success rate, etc.

The alarm notification module 24 notifies the person in charge of each application of the persistent final result according to the indicator configuration item and related events through certain rules. The alarm notification module 24 is further configured to receive a socket request or a microservice request sent by the no-log system, and the no-log application is converted into the effective application log.

Continuing to refer to FIG. 3 , the application log monitoring system of the present invention further includes a batch running module 25 and a visualization console 26 . The batch running module 25 is used to load the configuration items and related indicators regularly. The visual console 26 is configured to monitor at least one of error code types and statistical index types according to monitoring indicators, where the monitoring indicators include at least one of monitoring level, alarm escalation, notification frequency, and notification person. As shown in FIG. 4 , the present invention provides a visual console 26 to implement visual monitoring. The visual console 26 can monitor error code types, statistical index types, etc. Status, whether to automatically escalate the alarm, whether there is a plan, who to notify, description, and notification frequency, etc.

The application log monitoring system of the present invention further includes a database 27, and the database 27 includes a distributed cache redis or a database polardb for storing the analysis results of the log processing module.

The application log monitoring system of the present invention captures the application log of the application program by locally installing the logstash/filebeat collection tool in the application program or resource to be monitored. By configuring some specific codes on the collection tool, you can perform operations such as format splicing of application logs. If it is middleware such as redis, you can also use specific scripts. After that, the application logs are sent to the Kafka cluster. Among them, the filebeat collection tool does not perform any filtering on the local log, and all filtering is performed by the method of the present invention.

The monitoring system first obtains log files from the distributed message queue Kafka, and then distributes them to the back-end log processing core nodes based on the system. The core nodes convert the logs into internal objects, and then analyze the logs based on the configured statistics items. , the statistical results are placed in the distributed cache Redis or the database polardb. The alarm system will periodically collect and calculate the results of the corresponding statistical items based on the indicators, and finally compare it with the threshold of the indicator. If the threshold is touched, an alarm will be issued.

The traditional log monitoring technology only records statistics for logs, and only monitors hosts such as Cpu and memory, but does not monitor the predictive and unpredictable errors of the specific logic of the application itself. The result is abnormal discovery. It is a lag, waiting for the problem to be discovered and then logging in to the machine to view the log to trace the cause of the problem. In this way, what is often checked is not the first scene but a snapshot.

In the present invention, no intrusive code implantation is performed on the system and the application program, and the system application prints a specification log for the foreseeable errors, and notifies the person in charge in real time when the problem occurs, so that the immediate action can be taken. decision (rollback or other remedial action), especially for applications with low fault tolerance such as payments.

The above descriptions are only preferred embodiments of the present invention, and are not intended to limit the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the scope of the present invention. within the scope of protection.

Claims (10)

1. An application log monitoring method for an application program, comprising:

checking the effectiveness of the collected application logs to screen out effective application logs;

judging whether the screened effective application logs belong to a stack or not;

according to configuration items of a preset rule index, performing predictive anomaly and non-predictive anomaly analysis on the effective application logs which do not belong to the stack;

and sending alarm information related to the analysis result according to the configuration item.

2. The application log monitoring method for an application program according to claim 1, further comprising:

and monitoring at least one of an error code type and a statistical index type through a visual console according to a monitoring index, wherein the monitoring index comprises at least one of a monitoring grade, whether to upgrade an alarm, a notification frequency and a notifier.

3. The application log monitoring method for the application program according to claim 1, wherein sending the alarm information comprises:

and judging whether the alarm condition is met by utilizing the minute-level timing task.

4. The application log monitoring method for the application program according to claim 1, wherein sending alarm information related to the analysis result according to the configuration item comprises:

acquiring a result of a corresponding statistical item based on an index, and comparing the result of the statistical item with a threshold value of the index;

and if the result of the statistical item reaches the threshold value of the index, sending the alarm information.

5. The application log monitoring method for an application program according to claim 1, further comprising:

and installing a collection tool for grabbing the collected application log locally in the application program.

6. An application log monitoring system for an application program, comprising:

the collection tool is installed locally in the application program and used for collecting the application logs to the distributed message queue;

the log routing module is used for checking the validity of the application logs in the distributed message queue so as to screen out valid application logs;

the log processing module is used for judging whether the screened effective application logs belong to a stack or not, and analyzing foreseeable exception and unpredicted exception of the effective application logs which do not belong to the stack according to configuration items of preset rule indexes;

and the alarm notification module is used for sending alarm information related to the analysis result according to the configuration item.

7. The application log monitoring system for an application program of claim 6,

the alarm notification module is further used for receiving a socket request or a micro-service request sent by the no-log system, and the no-log application is converted into the effective application log.

8. The application log monitoring system for an application program of claim 6, further comprising:

and the batch running module is used for loading the configuration items and the related indexes at fixed time.

9. The application log monitoring system for an application program of claim 6, further comprising:

and the visual control console is used for monitoring at least one of the error code type and the statistical index type according to monitoring indexes, wherein the monitoring indexes comprise at least one of monitoring level, alarm upgrading or not, notification frequency and notifier.

10. The application log monitoring system for an application program of claim 6, further comprising:

and the distributed cache or the database is used for storing the analysis result of the log processing module.

Images