CN115098378A - Method and device for classifying and aggregating log segments based on abnormal breakpoints - Google Patents

Abstract

The application provides a method for classifying and aggregating log segments based on abnormal breakpoints. The method comprises the following steps: responding to an access request of a worker to execute flow playback; monitoring the flow playback event; when a flow playback failure event is monitored, extracting playback thread information of current execution playback from a playback context acquired by the monitoring event; acquiring a preset log path, wherein the log path is used for storing logs; acquiring a log according to the preset log path; performing content matching and interception from the log according to the playback thread information to obtain a plurality of corresponding abnormal information sections; processing the plurality of abnormal information segments to obtain specific abnormal log segments; and storing the specific abnormal log segments in a memory. The positioning of workers is assisted, the problem solving is more efficient, and the working efficiency is improved.

Description

Method and device for classifying and aggregating log segments based on abnormal breakpoints

technical field

The present invention relates to the field of computer technology, in particular to a method for classifying and aggregating log segments based on abnormal breakpoints, and a device for classifying and aggregating log segments based on abnormal breakpoints.

Background technique

Logs often play an important role in enterprise IT services, and the log information generated is of great value in the development, testing and operation phases of software products. Many Internet companies will output logs through page embedding to obtain user operation information to assist operations. At the same time, the log information also has other more important functions, especially in the troubleshooting stage, when the system encounters an unexplained abnormality during the operation process, the log can help the development and operation and maintenance colleagues to see the current system abnormal error report. In this way, the complete stack information of the exception can be quickly found, and the abnormal code fragment can be located.

Usually, the log files of the application are usually recorded in a specific directory of the server where the application is deployed. When we query the log, we need to log in to the server to find the relevant log files for query, and we also need to master the relevant vi operations for accurate query. shell script. If the application is deployed in a production environment, it is necessary to apply for server permissions and go through layers of approval. Even if the application is connected to other log collection tools, such as ELK, FLUME, etc., although it can be queried by accessing the visual interface, the real-time and abnormality of the log Considering the positioning accuracy, it is difficult to locate the abnormal information in time and accurately. Therefore, when an exception occurs, these factors are not conducive to us quickly locating the problem through the log, thereby affecting work efficiency.

SUMMARY OF THE INVENTION

The present application provides a method for classifying and aggregating log fragments based on abnormal breakpoints, and a device for classifying and aggregating log fragments based on abnormal breakpoints, which help staff locate problems and improve work efficiency.

In a first aspect, an embodiment of the present application provides a method for classifying and aggregating log segments based on abnormal breakpoints. The method for classifying and aggregating log segments based on abnormal breakpoints includes: performing traffic playback in response to an access request of a staff member; monitoring the traffic playback event; Playback thread information currently performing playback; obtain a preset log path, where the log path is used to store logs; obtain logs according to the preset log path; perform content matching and interception from the log according to the playback thread information Obtaining several corresponding abnormal information segments; processing the several abnormal information segments to obtain specific abnormal log segments; and storing the specific abnormal log segments in the memory.

In a second aspect, an embodiment of the present application further provides an apparatus for classifying and aggregating log segments based on abnormal breakpoints, wherein the main playback module in the apparatus includes:

A playback unit, used to perform traffic playback in response to a worker's access request;

a monitoring unit, configured to monitor the traffic playback event and call the playback comparison failure event module when monitoring the traffic playback failure event;

The playback comparison failure event module includes:

a path obtaining unit, configured to obtain a preset log path, where the log path is used to store logs;

a log obtaining unit, configured to obtain a log according to the preset log path;

an extraction unit, which performs content matching and interception from the log according to the playback thread information to obtain several corresponding abnormal information segments;

a processing unit, configured to process the several exception information segments to obtain each specific exception log segment; and

The storage unit is configured to store the specific exception log segments in the memory.

In a third aspect, an embodiment of the present application further provides a server, wherein the server includes:

memory for storing computer program instructions;

A processor for executing computer program instructions to implement the method for classifying and aggregating log segments based on exception breakpoints.

The above method for classifying and aggregating log segments based on exception breakpoints obtains data information such as log data, exception information, program playback threads, and error stacks generated during the operation by performing a traffic playback operation. Obtain abnormal information fragments in the log according to the source program and the generated information, and perform processing such as interception and aggregation on the log fragments to generate processed abnormal fragments, and store the processed abnormal fragments in the cloud storage. to be displayed to the user for the user to locate the problem based on the anomaly fragment. At the same time, log interception, aggregation, and matching logic can also be optimized. This solution can check the log without logging in to the server when an exception occurs, eliminating the need to log in to the server to perform permission approval operations for related springboard machines. And after the original log is accurately matched and aggregated through the information data of the source code, it can be stored in the cloud storage in real time, and the user can immediately click to view it directly on the comparison failure details page. Through the organized log, users can quickly find the exception stack from the log, help analyze the cause of abnormal failure, locate and solve problems, and improve work efficiency.

Description of drawings

In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the accompanying drawings required for the description of the embodiments or the prior art. Obviously, the drawings in the following description are only These are some embodiments of the present application. For those of ordinary skill in the art, other drawings can also be obtained according to the structures shown in these drawings without any creative effort.

FIG. 1 is a flowchart of a method for classifying and aggregating log segments based on abnormal breakpoints provided by an embodiment of the present application.

FIG. 2 is a schematic structural diagram of an apparatus for classifying and aggregating log segments based on abnormal breakpoints according to an embodiment of the present invention.

FIG. 3 is another flowchart of a method for classifying and aggregating log segments based on abnormal breakpoints provided by an embodiment of the present application.

FIG. 4 is a schematic diagram of an internal structure of a computer device for a method for classifying and aggregating log segments based on abnormal breakpoints provided by an embodiment of the present application.

The realization, functional characteristics and advantages of the purpose of the present application will be further described with reference to the accompanying drawings in conjunction with the embodiments.

Detailed ways

In order to make the purpose, technical solutions and advantages of the present application more clearly understood, the present application will be described in further detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application, but not to limit the present application. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present application.

The terms "first", "second", "third", "fourth", etc. (if any) in the description and claims of the present application and the above-mentioned drawings are used to distinguish similar planning objects and do not necessarily Used to describe a specific order or sequence. It is to be understood that the data so used are interchangeable under appropriate circumstances, in other words, the described embodiments are implemented in accordance with sequences other than those illustrated or described herein. Furthermore, the terms "comprising" and "having" and any variations thereof may also encompass other content, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to only those expressly listed steps or units, but may include other steps or units not expressly listed or inherent to these processes, methods, products or devices.

It should be noted that the descriptions involving "first", "second", etc. in this application are only for the purpose of description, and should not be construed as indicating or implying their relative importance or implying the number of indicated technical features . Thus, a feature defined as "first" or "second" may expressly or implicitly include one or more of that feature. In addition, the technical solutions between the various embodiments can be combined with each other, but must be based on the realization by those of ordinary skill in the art. When the combination of technical solutions is contradictory or cannot be realized, it should be considered that the combination of such technical solutions does not exist. , is not within the scope of protection claimed in this application.

The present application provides a method for classifying and aggregating log fragments based on abnormal breakpoints. The method is applied to a system for classifying and aggregating log fragments based on abnormal breakpoints. The system includes a client and a server. The client communicates with the server over the network. Wherein, the client can be, but not limited to, various personal computers, notebook computers, and tablet computers. The server can be implemented as an independent server or a server cluster composed of multiple servers. The worker sends a request for performing a traffic playback operation to the server through the client, and the server processes the request according to a preset algorithm and sends the processing result to the client for the worker to view. During the development, testing and operation of software products, network equipment, etc., log data is generated when the products or equipment operate. The log data records data generated during running, and each line of log data includes descriptions of related operations such as date, time, user, and action. Before executing the above method, a storage path is preconfigured for the log generated by the server during the operation, so as to be called when needed. And the log interception plug-in is loaded in the server to capture abnormal events. When an exception occurs, the log interception plug-in can intercept and aggregate specific log fragments.

Please refer to FIG. 1 , which is a flowchart of a method for classifying and aggregating log segments based on abnormal breakpoints provided by the first embodiment of the present application. The method for classifying and aggregating log segments based on abnormal breakpoints includes steps S101-S108.

Step S101, performing traffic playback in response to a staff member's access request.

After development is complete, the code needs to be tested. In the process of code testing, new requirements will continue to be added. In the process of adding new requirements, there will be refactoring and splitting of the code, and the source code will be constantly changed. During the process of changing, the original functions may be affected. Traffic playback is to store all requests sent to a certain application A within a certain period of time, and send the requests to another different application B in a unified manner, so that the request parameters received by A and B are consistent, thus realizing A. The received request is re-requested to B. Every time a new requirement is added in the code test, the request for the operation of the source code will be re-run on the new code to make sure that the new function code does not affect the function implemented by the source code. The staff can select the traffic segment to be played back on the client side, which can be the traffic of a certain case set, such as a large number of requests initiated by users on the hour during Double Eleven shopping, or a single traffic, for example, a user purchases a ticket from Buy tickets on the system. The client sends the worker's request to the server so that the server executes the traffic playback event.

Step S102, monitor the traffic playback event.

Monitoring is to monitor the network running status and data flow, and can obtain all the information generated during the monitoring process. The server side responds to the client's request to run the traffic in a new complete function code, and records the data generated during the running process and the involved codes, including the thread name Main, the main calling function and the subordinate code of the counterparty's code. Calling functions, etc., the generated data is recorded in the log, including: the origination time, the Exception information generated when the traffic playback is abnormal, the Exception information is judged by the program itself during the running process of the program, and the error is irreversible and will be thrown. exception, the error stack in the program code, the exception process information printed in the SQL computer language during the execution process, and the info/warn information corresponding to the exception, where the info/warn information is a log category.

Step S103, when a traffic playback failure event is monitored, the playback thread information currently performing playback is extracted from the playback context obtained from the monitoring event.

When the server detects abnormal information such as Exception information generated by the running program, abnormal process information printed in SQL computer language, etc., it is determined as a traffic playback failure event. Get the code snippets involved in traffic playback, the thread name Main of the code, the main calling function, and the sub-calling function. Get exception information generated by the program during playback.

Step S104: Obtain a preset log path, where the log path is used for storing logs.

Obtain a preconfigured log path of the playback process, and obtain a log of the playback process according to the path. Use RandomAccessFile operation file to read log content in reverse order. RandomAccessFile is the most feature-rich file content access class that can be used in the process of Java input or output data. It can not only read file content, but also output data to the file. RandomAccessFile operation Files can be accessed from anywhere. Since the log file is written from the back at work, and when viewing the log, you need to start with the latest information. When the file is too large, it is impossible to read and view the entire file, so the log is read in reverse order. The log records all code fragments involved in the playback process, the calling process, and exception information generated during the calling process.

Step S105, obtaining a log according to the preset log path.

Step S106: Perform content matching and interception from the log according to the playback thread information to obtain several corresponding abnormal information segments.

The thread information is the abnormal code involved in the code running process, and the log interception plug-in is used to match and obtain several abnormal code fragments corresponding to the thread information in the log.

Step S107, processing the several abnormal information segments to obtain specific abnormal log segments.

The processing includes interception, aggregation, and the like. Each acquired abnormal code segment is divided into several code information segments, and the several code information segments acquired by segments are aggregated according to preset conditions to form the several specific abnormal information segments. The preset conditions include the start time of the code running, the thread name of the code, the main call information and the sub-call information contained in the code.

Step S108: Store the specific exception log segments in a memory.

The storage is cloud storage, and specifically, the cloud storage may be Amazon S3 cloud storage. The log fragments obtained after processing are stored in the Amazon S3 cloud storage. When the traffic playback fails, the server sends a playback failure page to the client, and the playback failure page has several playback options for the user to choose. The several options include: playback result options, source code snippets involved, log content generated, and the like. If the customer selects the return visit result query option on the page, the exception code snippet obtained after processing will be displayed.

Understandably, if the traffic playback operation is successful, the next playback request from the client is received and the playback operation is performed.

The above embodiment also provides an apparatus 1 for classifying and aggregating log segments based on abnormal breakpoints, characterized in that the playback main module 2 and the playback comparison failure event module 3 are described.

The playback main module 2 includes:

The playback unit 3 is used to perform traffic playback in response to the access request of the staff; the staff can select the playback traffic segment on the client side, which can be the traffic of a certain case set, such as that initiated by the user on the hour during Double Eleven shopping A large number of requests can also be a single piece of traffic. For example, a user buys a ticket from a ticketing system. The client sends the worker's request to the server so that the server executes the traffic playback event.

The monitoring unit 4 is used to monitor the traffic playback event and when monitoring the traffic playback failure event, call the playback comparison failure event module; the server side responds to the client's request to run the traffic in a new complete function code, And record the data and code involved in the running process

The playback comparison failure event module 2 includes:

The path obtaining unit 5 is configured to obtain a preset log path, where the log path is used to store the log; obtain a preconfigured log path of the playback process, and obtain the log of the recording playback process according to the path.

a log obtaining unit 6, configured to obtain a log according to the preset log path;

Extraction unit 7, to perform content matching and interception from the log according to the playback thread information to obtain a number of corresponding abnormal information segments; the thread information is the abnormal code involved in the code running process, and the log interception plug-in is applied to match And acquire several abnormal code fragments corresponding to the thread information in the log.

The processing unit 8 is used to process the several abnormal information segments to obtain each specific abnormal log segment; and divide each acquired abnormal code segment into several code information segments, and divide the acquired code information segments according to preset Conditions are aggregated to form the specific pieces of exception information.

The storage unit 9 is configured to store the specific exception log segments in the memory. The log fragments obtained after processing are stored in the Amazon S3 cloud storage.

As shown in FIG. 3 , the method for classifying and aggregating log segments based on abnormal breakpoints provided by the present application includes: a worker initiates a request from a client, and a server performs traffic playback in response to the worker's access request. The server listens for the traffic playback event. When monitoring the traffic playback failure event, the server extracts the playback thread information currently performing playback from the playback context acquired by the monitoring event. When the playback operation is successful, the server responds to the access request of the next worker and performs the traffic playback operation. The server obtains a preset log path, where the log path is used for storing logs. The server obtains the log according to the preset log path. The server performs content matching and interception from the log according to the playback thread information to obtain several corresponding abnormal information segments, and the server processes the several abnormal information segments to obtain each specific abnormal log segment. and the server stores the specific exception log segments in the memory. Staff can obtain exception information from the memory page.

The present application also provides a computer device 900. Please refer to FIG. 4, which is a schematic diagram of the internal structure of the computer device 900 provided by the first embodiment of the present application. Further, the computer embedded device is a HUD. The computer device 900 includes at least a memory 901 and a processor 902 . Specifically, the memory 901 is used to store program instructions of a method for classifying and aggregating log segments based on exception breakpoints. The processor 902 is configured to execute program instructions to enable the computer device 900 to implement the above-mentioned method for classifying and aggregating log segments based on exception breakpoints.

The memory 901 includes at least one type of readable storage medium, including flash memory, hard disk, multimedia card, card-type memory (eg, SD or DX memory, etc.), magnetic memory, magnetic disk, optical disk, and the like. The memory 901 can not only be used to store the application software and various data installed in the computer device 900, for example, the control instructions of the method of classifying and aggregating log fragments based on exception breakpoints, etc., but also can be used to temporarily store the outputted or to be outputted data. data.

The processor 902 may be a central processing unit (Central Processing Unit, CPU), a controller, a microcontroller, a microprocessor or other data processing chips in some embodiments, and is used to execute program instructions or process data stored in the memory 901 . Specifically, the processor 902 executes the program instructions of the method of classifying and aggregating log segments based on exception breakpoints to control the computer device 900 to implement the method of classifying and aggregating log segments based on exception breakpoints.

Further, the bus 903 may be a peripheral component interconnect standard bus (peripheral component interconnect, referred to as PCI) or an extended industry standard architecture bus (extended industry standard architecture, referred to as EISA) or the like. The bus can be divided into address bus, data bus, control bus and so on. For ease of presentation, only one thick line is used in FIG. 4, but it does not mean that there is only one bus or one type of bus.

Further, computer device 900 may also include display component 904 . The display component 904 may be an LED (Light Emitting Diode, light-emitting diode) display, a liquid crystal display, a touch-sensitive liquid crystal display, an OLED (Organic Light-Emitting Diode, organic light-emitting diode) touch device, and the like. The display component 904 may also be appropriately referred to as a display device or a display unit for displaying information processed in the computer device 900 and for displaying a visual user interface.

Further, the computer device 900 may also include a communication component 905, and the communication component 905 may optionally include a wired communication component and/or a wireless communication component (such as a WI-FI communication component, a Bluetooth communication component, etc.), which are generally used in computer equipment A communication connection is established between 900 and other computer devices.

FIG. 4 only shows a computer device 900 having components 901-905 and program instructions for implementing the method for classifying and aggregating log segments based on exception breakpoints. Those skilled in the art can understand that the structure shown in FIG. 4 does not constitute a The definition of computer device 900 may include fewer or more components than shown, or a combination of certain components, or a different arrangement of components.

Those skilled in the art can clearly understand that, for the convenience and brevity of the description, the specific working process of the above-described systems, devices and units, specifically, the processor 902 of the computer device 900 executes the classification and aggregation of log segments based on exception breakpoints. The program instructions of the method are used to control the computer device 900 to implement the detailed process of the method for predicting the motion trajectory of the movable object. Reference may be made to the corresponding processes in the foregoing method embodiments, and details are not described herein again.

Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, and these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

The above enumeration is only the preferred embodiment of the present application, of course, the scope of the right of the present application cannot be limited by this, so the equivalent changes made according to the claims of the present application still belong to the scope covered by the present application.

Claims (10)

1. A method for classifying and aggregating log fragments based on abnormal breakpoints, wherein the method comprises: Perform traffic replay in response to worker's access request; monitor the traffic playback event; When the traffic playback failure event is monitored, the playback thread information currently performing playback is extracted from the playback context obtained by the monitoring event; Obtain a preset log path, where the log path is used to store logs; Obtain the log according to the preset log path; Perform content matching and interception from the log according to the playback thread information to obtain a number of corresponding abnormal information segments; processing the several exception information segments to obtain each specific exception log segment; and The specific exception log segments are stored in the memory. 2. The method according to claim 1, wherein the method further comprises: If the playback operation is successful, respond to the access request of the next worker and perform the traffic playback operation. 3. The method according to claim 1, wherein the method further comprises: providing a playback failure page that provides several playback options for the user to select; In response to the playback result option selected by the user, the corresponding abnormal log segment is queried and displayed to the user. 4. The method for classifying and aggregating log fragments based on abnormal breakpoints according to claim 1, wherein processing the several abnormal information fragments to obtain specific abnormal log fragments comprises: Segment each abnormal information segment to form several abnormal information segments; The specific abnormal log fragments are formed by aggregating several abnormal information fragments according to preset conditions. 5 . The method for classifying and aggregating log segments based on exception breakpoints according to claim 4 , wherein the preset conditions include initiation time, thread name, main call information, and sub-call information. 6 . 6. The method for classifying and aggregating log fragments based on abnormal breakpoints as claimed in claim 4, wherein acquiring the log according to the preset log path specifically comprises: Use RandomAccessFile to operate the file to read the log content in reverse order. 7 . The method for classifying and aggregating log fragments based on abnormal breakpoints according to claim 4 , wherein the storage is a cloud storage. 8 . 8. a device based on abnormal breakpoint classification aggregation log fragment, it is characterized in that, described device is made up of playback main module and playback comparison failure event module, The main playback module includes: A playback unit, used to perform traffic playback in response to a worker's access request; a monitoring unit, configured to monitor the traffic playback event and call the playback comparison failure event module when monitoring the traffic playback failure event; The playback comparison failure event module includes: a path obtaining unit, configured to obtain a preset log path, where the log path is used to store logs; a log obtaining unit, configured to obtain a log according to the preset log path; an extraction unit, which performs content matching and interception from the log according to the playback thread information to obtain several corresponding abnormal information segments; a processing unit, configured to process the several exception information segments to obtain each specific exception log segment; and The storage unit is configured to store the specific exception log segments in the memory. 9. A computer-readable storage medium, wherein the computer-readable storage medium is used to store program instructions, the program instructions can be executed by a processor to Anomaly breakpoint classification method for aggregated log fragments. 10. A server, characterized in that the server comprises: memory for storing computer program instructions; A processor for executing computer program instructions to implement the method for classifying and aggregating log segments based on exception breakpoints as claimed in any one of claims 1 to 8.

Images