CN109992476B - Log analysis method, server and storage medium - Google Patents

Abstract

The embodiment of the application relates to the field of data processing, and discloses a log analysis method, a server and a storage medium. In some embodiments of the present application, a method for analyzing a log includes: acquiring a first log to be processed; processing the first log to obtain a word bag of the first log; determining similarity of a word bag of the first log and a word bag of a reference log in a mapping file, wherein the mapping file comprises the word bag of the reference log and a fault class of the reference log and/or a fault level of the reference log; and determining the fault class of the first log and/or the fault level of the first log according to the similarity between the word bags of the first log and the word bags of the reference log. In the implementation, the server can analyze the first log by using the mapping file, determine the fault class of the first log and/or the fault level of the first log, so that the intelligence of the server is improved, and the pressure of maintenance personnel from analyzing the log is reduced.

Description

A log analysis method, server and storage medium

technical field

The embodiments of the present invention relate to the field of data processing, and in particular to a log analysis method, server and storage medium.

Background technique

The kernel log is the main means used to record the server itself, as well as the performance status of the running processes and modules during the running process. However, for some kernel messages, the kernel log cannot record them. For example, when the system crashes (panic), some information is directly displayed on the screen, and this part of information cannot be recorded in the kernel log due to the crash. When the system is restarted, this part of the information will disappear. Currently, some transmission tools, such as netconsoles, solve the problem that this part of the kernel log cannot be collected. It sends this part of the kernel log to another server for storage through the network, so that the kernel log that the system can keep is as complete as possible.

However, the inventor has found that there are at least the following problems in the prior art: the amount of logs generated every day is huge, especially for the number of enterprise-level servers, the amount of logs is extremely large, and it will be wasteful to manually process the logs of each server. A lot of time and energy.

It should be noted that the information disclosed in the above background section is only for enhancing the understanding of the background of the present disclosure, and therefore may include information that does not constitute the prior art known to those of ordinary skill in the art.

Contents of the invention

The purpose of the embodiments of the present invention is to provide a log analysis method, server and storage medium, so that the number of recorded logs is reduced, and the time and energy spent on manual processing of logs are reduced.

In order to solve the above technical problems, the embodiment of the present invention also provides a log analysis method, including the following steps: obtaining the first log to be processed; processing the first log to obtain the bag of words of the first log; determining the first log The similarity between the bag of words of a log and the bag of words of the reference log in the mapping file, wherein the mapping file includes the bag of words of the reference log, and the fault category of the reference log, and/or, the fault level of the reference log; according to the first The similarity between the bag of words of the log and the bag of words of the reference log determines the fault category of the first log, and/or, the fault level of the first log.

Embodiments of the present invention also provide a server, including: at least one processor; and a memory connected in communication with at least one processor; wherein, the memory stores instructions that can be executed by at least one processor, and the instructions are executed by at least one processor. The processor executes, so that at least one processor can execute the log analysis method mentioned in the above implementation manner.

Embodiments of the present invention also provide a computer-readable storage medium storing a computer program, and implementing the log analysis method mentioned in the above embodiment when the computer program is executed by a processor.

Compared with the prior art, the embodiment of the present invention compares the bag of words of the log to be processed with the bag of words of the historical log, and can determine the relationship between the log to be processed and the bag of words of the historical log, which reflects the The relationship between pending logs and historical logs. Since the server can determine the relationship between the log to be processed and the historical log, it can selectively retain the recorded log according to the relationship, reducing the number of recorded logs and reducing the burden of manual log processing.

In addition, according to the similarity between the word bag of the first log and the word bag of the reference log, determine the fault category of the first log, and/or, the fault level of the first log, specifically include: The failure category of the reference log with the highest similarity is used as the failure category of the first log, and/or the failure level of the reference log with the highest similarity to the bag of words of the first log is used as the failure level of the first log. In this implementation, the fault type and/or fault level of the log can be automatically determined, which improves intelligence and facilitates maintenance personnel to know the fault type and fault level that occur on the server.

In addition, according to the similarity between the bag of words of the first log and the bag of words of the reference log, determining the fault category of the first log, and/or, the fault level of the first log, specifically includes: judging whether there is a The similarity of the bag of words of the log is greater than the bag of words of the reference log of the second preset value; if it is determined to be yes, the fault category of the reference log with the highest similarity to the bag of words of the first log is used as the fault category of the first log , and/or, use the fault level of the reference log with the highest similarity with the bag of words of the first log as the fault level of the first log; otherwise, determine the fault category of the bag of words of the first log as an unknown category, and/or Or, determine that the failure level of the first log is an unknown level. In this implementation, automatic identification of new faults is enabled.

In addition, the mapping file includes the word bag of the reference log, the fault category of the reference log, and the fault level of the reference log; after determining that the fault category of the word bag of the first log is an unknown category, and determining that the fault level of the first log is an unknown level, The log analysis method also includes: reporting the first log; determining the fault category of the first log and the fault level of the first log according to the fault category and fault level specified by the user; Category and failure level of the first log, update the mapping file. In this implementation, the mapping file can be updated according to the identified log of the new fault category, and the mapping file can be continuously expanded to facilitate more accurate analysis of subsequent logs.

In addition, the mapping file includes the bag of words of the reference log, the fault category of the reference log and the fault level of the reference log; according to the similarity between the bag of words of the first log and the bag of words of the reference log, the fault category of the first log and the fault category of the second log are determined. After the failure level of a log, the log analysis method also includes: judging whether there is a second log in the recorded log, wherein, the second log is a log belonging to the same fault category as the first log; if it is determined to exist, compare the first log According to the comparison result, update the recorded log; if it is determined that it does not exist, record the first log.

In addition, according to the comparison result, updating the recorded log specifically includes: if it is determined that the comparison result indicates that the failure level of the first log is higher than that of the second log, overwriting the second log with the first log; The fault level of the log is not higher than that of the second log, and the second log is not overwritten by the first log. In this implementation, the log with a higher fault level in the same fault category is recorded to ensure that the importance of the reference log is continuously improved, so as to achieve the effect of continuous escalation of alarms.

In addition, determining the similarity between the bag of words of the first log and the bag of words of the reference log in the mapping file specifically includes: calculating the similarity according to the constraint relationship between the bag of words of the first log, the bag of words of the reference log, and the similarity; Wherein, constraint relation is: similarity=appear in the word bag of the first log and the word quantity of the word bag of reference log/(the word quantity of the word bag of the first log+the word quantity of reference log-appear in the first log at the same time number of words in bag-of-words log and bag-of-words reference log).

In addition, before calculating the similarity according to the bag of words of the first log, the bag of words of the reference log and the constraint relationship of the similarity, the analysis method of the log also includes: removing the bag of words of the first log and the bag of words of the reference log Invalid word; where invalid word is a pre-specified word. In this implementation, the impact of invalid words on the similarity between the bag of words of the first log and the bag of words of the reference log in the mapping file can be avoided.

In addition, the first log is processed to obtain the bag of words of the first log, which specifically includes: deleting the variable in the first log, where the variable is a preset parameter; splitting the first log after the variable is deleted into N words, Generate bag-of-words for logs to be processed, N is a positive integer.

In addition, the preset parameters include at least any one of bad track location information, bad track number information, bad block location information, and bad block number information.

In addition, deleting the variable in the first log specifically includes: identifying the number of the text part of the first log; and deleting the number of the text part of the first log.

Description of drawings

One or more embodiments are exemplified by the pictures in the corresponding drawings, and these exemplifications do not constitute a limitation to the embodiments. Elements with the same reference numerals in the drawings represent similar elements. Unless otherwise stated, the drawings in the drawings are not limited to scale.

Fig. 1 is the flowchart of the log processing method of the first embodiment of the present invention;

Fig. 2 is the flowchart of the log processing method of the second embodiment of the present invention;

Fig. 3 is the flow chart of the log analysis method of the third embodiment of the present invention;

Fig. 4 is a flowchart of a log analysis method according to a fourth embodiment of the present invention;

FIG. 5 is a schematic structural diagram of a server according to a fifth embodiment of the present invention;

FIG. 6 is a schematic structural diagram of a server according to a sixth embodiment of the present invention.

Detailed ways

In order to make the purpose, technical solutions and advantages of the embodiments of the present invention more clear, various implementation modes of the present invention will be described in detail below in conjunction with the accompanying drawings. However, those of ordinary skill in the art can understand that, in each implementation manner of the present invention, many technical details are provided for readers to better understand the present application. However, even without these technical details and various changes and modifications based on the following implementation modes, the technical solution claimed in this application can also be realized.

The first embodiment of the present invention relates to a log processing method, which is applied to a server. As shown in Figure 1, the log processing methods include:

Step 101: Obtain logs to be processed.

Specifically, the logs to be processed may be logs generated by the server itself, or logs of other servers stored on the server. Among them, other servers can transfer their own logs to this server through netconsoles, or copy logs of other servers to this server through other methods.

It should be noted that those skilled in the art can understand that the log processing method can be applied to the process of the server processing multiple recorded logs, and can also be applied to the server to determine whether to record the log after the log is generated. During the process, this embodiment does not limit the application scenarios of the log processing method.

For clarification, in this embodiment, it is assumed that after receiving the first log file, the server uses the log processing method mentioned in this embodiment to process each log in order from old to new. Those skilled in the art can understand that, in practical applications, the process of processing the logs generated by the server itself can refer to the relevant content of this embodiment, and will not be repeated here.

Step 102: Process the log to be processed to obtain the bag of words of the log to be processed.

Specifically, the log is mainly composed of words. A log is converted into a bag of words consisting of several words. The bag of words does not contain repeated words. The relationship between the logs can be determined through the relationship between the bags of words in the log.

In an example, the server first deletes the variable in the log to be processed; splits the log to be processed after the variable is deleted into N words, and generates a bag of words of the log to be processed, and N is a positive integer. That is, the server compresses the logs to be processed before converting the logs into word bags. Since there may be some overly detailed information in the logs to be processed, or information that maintenance personnel do not value, this information is not very useful for analyzing the running status of the server, so maintenance personnel can set these information as variables so that the server can When processing the log, delete this information and compress the log.

The process of deleting the variable in the log to be processed by the server is illustrated as an example below.

In one example, the variables are preset parameters. Wherein, the preset parameters include at least any one of bad track location information, bad track number information, bad block location information, and bad block number information. In this case, the method for the server to delete the above variable may be: identifying the number of the body part of the log to be processed; and deleting the number of the body part of the log to be processed.

For each log, it can be split into two parts, namely the timestamp part and the body part. For the text part, the pre-set variables are removed, and the quantification is retained. Among them, quantitative is more important information in the log. The determination of variables and quantification can be set according to experience and needs.

In one example, variables may include, but are not limited to, the following information:

1. Too detailed information, such as bad track location information (such as address), bad block location information, bad track number information, and bad block number information. The above information is composed of a series of numbers, or numbers plus English letters .

2. Relatively unimportant information, such as 1 in sda1 indicates the first partition of the disk named sda, wherein sda is important information, and 1 is unimportant information.

Since the above information includes numbers, the server can determine the variables in the log by identifying the numbers. Of course, in this embodiment, the method of identifying numbers to determine the variables in the log does not mean that the numbers are all unimportant information. For example, in the log whose content is Nov 26 00:24:04 CPU27:Package power limit notification(total events=173318), the importance of 27 in CPU27 is the same as that of a in sda disk, and should not be deleted. Among them, Nov 26 00:24:04 CPU27:Package power limit notification(total events=173318) indicates that at 0:24:4 on November 26, the performance power limit notification of the No. 27 central processing unit (Central Processing Unit, CPU) (total events = 173318). In this case, the deletion rule can be modified to avoid the situation that the important information is deleted by mistake. For example, before deleting the number in the text part of the log to be processed, the server judges whether the word before the number is CPU, and if it is determined that it is not, then deletes the number.

It should be noted that those skilled in the art can understand that when the preset variable changes, the method for identifying the variable by the server may also change. In practical applications, the method for identifying the variable by the server can be set as required.

The following describes the process of compressing the logs to be processed in combination with the actual situation.

For example, the pending log is: Nov 20 00:01:02 I/O error on device sdc1, logicalblock 1057. The text part of the log is: I/O error on device sdc1, logical block 1057. Among them, the variable sdc1 refers to the first partition of the sdc disk, and the variable 1057 represents the 1057th logical block, which is "I/O error on device" and "logical block". Therefore, the log tells us that an error occurred when reading and writing logical block No. 1057 of the first partition of the sdc disk. Compress the log, that is, modify the information in the current log, or discard some less important information to achieve the purpose of compression. For the log records in the example, the information of 1057 and the information of "first partition" of sdc can be discarded. Therefore, after compressing the log of the example, the information obtained is I/O error on device sdc, logical block. This information indicates that a logical block read/write error occurred on the disk named sdc.

From the above content, it can be known that the compression processing of the log to be processed retains more important information in the log to be processed, discards less important information, and reduces the storage space occupied by the log.

It should be noted that those skilled in the art can understand that in practical applications, the server can also perform other processing on the logs to be processed. In this embodiment, the compression processing is taken as an example, but the compression processing is not in the process of processing the logs to be processed. You can selectively execute this part of the necessary steps.

Step 103: Compare the bag of words of the log to be processed with the bag of words of the historical log, and determine the relationship between the bag of words of the log to be processed and the bag of words of the historical log.

Specifically, the relationship between the bag of words of the log to be processed (hereinafter referred to as bag of words 1) and the bag of words of the historical log (hereinafter referred to as bag of words 2) includes but is not limited to: first relationship, second relationship, third relationship relationship, fourth relationship and fifth relationship. Among them, the first relationship is that the bag of words of the historical log contains the bag of words of the log to be processed, the second relationship is that the bag of words of the historical log is equal to the bag of words of the log to be processed, and the third relationship is the bag of words of the log to be processed It contains the bag of words of the historical log, the fourth relationship is that the bag of words of the log to be processed intersects the bag of words of the historical log, and the fifth relationship is that the bag of words of the log to be processed is independent of the bag of words of the historical log.

Various relationships between the bag of words of the log to be processed (hereinafter referred to as bag of words 1) and the bag of words of the historical log (hereinafter referred to as bag of words 2) are explained below.

First, explain the first relationship and the third relationship, that is, explain the containment relationship represented by the bag of words 1 containing the bag of words or the bag of words 2 containing the representation of the bag of words 1 . Bag of words 1 contains bag of words 2 means that all words of bag of words 2 appear in bag of words 1, while bag of words 1 has some words that are not in bag of words 2. Bag of words 2 contains bag of words 1 means that all words of bag of words 1 appear in bag of words 2, and bag of words 2 has some words that are not in bag of words 1. When there is a problem in the log transmission process, the bag-of-words relationship of the same two logs may be the inclusion relationship. For example, for two identical log sentences, during the process of netconsoles transmitting logs, some elements are lost in one sentence, causing the server to receive a complete log sentence and an incomplete log sentence. In this case, the bag of words of the complete log contains the bag of words of the incomplete log, and there is an inclusion relationship between the bags of words.

Then, explain the second relationship, that is, explain the equality relationship represented by the bag-of-words 1 and the bag-of-words 2 being equal. Word bag 1 is equal to word bag 2, indicating that the words in word bag 1 and word bag 2 are exactly the same. For example, the bags of words generated by two identical logs are equal, or the bags of words of logs with the same important information are equal.

Next, the fourth relationship is explained, that is, the intersection relationship represented by the intersection of bag of words 1 and bag of words 2 is explained. The intersection of word bag 1 and word bag 2 means that some words in word bag 1 and word bag 2 have one-to-one correspondence, but some words do not appear in each other's word bags. When some important information in the log to be processed is the same as in the historical log, and some important information is different, the bag of words 1 and bag of words 2 are intersecting. For example, the pending log is Nov 25 18:09:11 Kernelpanic-not syncing: Fatal hardware error! The historical log is Nov 20 00:01:02 I/O error ondevice sdc1, logical block 1057. Both bags of words have the word "error", but they are two completely different logs. Among them, "Nov 25 18:09:11 Kernel panic-not syncing: Fatal hardware error!" means that at 18:09:11 on November 25, the system experienced a kernel panic-not syncing: fatal hardware error! "Nov 20 00:01:02I/O error on device sdc1, logical block 1057" indicates that at 00:01:02 on November 20, a read occurred on logical block 1057 of the first partition of the disk named sdc Written wrong.

Finally, explain the fifth relationship, that is, explain bag-of-words 1 and bag-of-words 2 independently. Bag of words 1 and bag of words 2 are independent, that is, bag of words 1 and bag of words 2 do not have the same words, and there is no connection between the corresponding logs.

In one example, a masked bag of words is set in the server, and before step 103 is executed, the server determines that the bag of words of the log to be processed does not include all the words in the masked bag of words. The blocked word bag contains blocked words, and when the word bag of the log to be processed includes all the words in the blocked word bag, the server deletes the log.

In one example, 16 blocked word bags are set in the server. The shielded word in the first shielded word bag is audit (audit), the shielded word in the second shielded word bag is inode (metadata node), and the shielded word in the third shielded word bag is hook (hook). The shielded words in the fourth shielded word bag are hung (hang), task (task), timeout (timeout) and secs (seconds), and the shielded words in the fifth shielded word bag are CAP (permission), NET ( Network) and ADMIN (administrator), the shielded word in the 6th shielded word bag is filesystem (file system), the shielded word in the 7th shielded word bag is IPVS (IP Virtual Server, IP virtual server), the 8th The shielded words in the first shielded word bag are the (this), kdump (kernel), crash (downtime) and info (information), and the shielded words in the ninth shielded word bag are USB (UniversalSerial Bus, Universal Serial Bus ), the shielded words in the 10th shielded word bag are bitmap (bitmap), the shielded words in the 11th shielded word bag are connect (connection), debounce (debounce) and failed (failure), the 12th shielded word The shielded words in the word bag are eth (network card), Reset (restart) and adapter (adapter), and the shielded words in the 13th shielded word bag are loading (loading), buddy (partner) and information (information), the 14th The shielded words in the first shielded word bag are license (certificate) and expired (expired), the shielded words in the 15th shielded word bag are bus (bus) and error (error), and the shielded words in the 16th shielded word bag For error (error) and device (device). When the log to be processed includes all the words in any one of the masked word bags, delete the log to be processed.

It should be noted that those skilled in the art can understand that in practical applications, the number of masked word bags can be set as required, and this embodiment does not limit the number of masked word bags.

It should be noted that those skilled in the art can understand that the masked words in each masked word bag can be set as required, and are not listed here.

It is worth mentioning that the server directly removes part of the logs according to the set blocked word bag, which reduces the processing pressure on the server and further reduces the number of logs.

Step 104: According to the relationship between the bag of words of the log to be processed and the bag of words of the historical log, determine whether to keep the log to be processed.

Specifically, during the processing of the first log file, the processed logs in the first log file that are determined to be referred to are saved in the second log file. Wherein, the historical log refers to the log in the second log file.

It should be noted that those skilled in the art can understand that if the server processes the log immediately after the log is generated, and then records the log after confirming to keep the log, the historical log refers to the recorded log. There is no limit to the meaning of the history log.

In one example, if the server determines that the relationship between the bag of words of the log to be processed and the bag of words of the historical log is the first relationship or the second relationship, the server deletes the log to be processed; The relationship between the bag of words of the log is the third relationship, and the time stamp part of the historical log and the bag of words of the log to be processed are reserved; if the relationship between the bag of words of the log to be processed and the bag of words of the historical log is determined to be the fourth or fifth relationship relation, keeping the timestamp portion of the pending log and the bag-of-words of the pending log. When the word bag of the log to be processed is the first relationship with the bag of words of the history log, it means that the log to be processed may be incomplete, or the information recorded in the history log is more detailed than the information recorded in the log to be processed. When the bag of words of the log and the bag of words of the history log have a third relationship, it means that the history log may be incomplete, or the information recorded in the log to be processed is more detailed than the information recorded in the history log. In this case, the log with the larger bag of words is reserved, and the timestamp of the log with the earlier timestamp is selected as the timestamp of the log. When the bag-of-words of the log to be processed is in the second relationship with the bag-of-words of the historical log, it means that the log to be processed may be exactly the same as the historical log, so the log to be processed may be deleted. When the relationship between the bag of words of the log to be processed and the bag of words of the history log is the fourth relationship, it means that some parameters of the history log and the log to be processed are the same, and some parameters are different. These two logs may be recorded on the same disk Logs of different fault types may also be logs of the same fault type recorded on different disks, or logs with the same descriptive words but completely different essences. Therefore, pending logs and historical logs need to be kept. When the bag-of-words of the log to be processed and the bag-of-words of the history log have the fifth relationship, it means that the log to be processed and the history log are completely irrelevant logs, so it is necessary to keep the log to be processed and the history log.

It can be seen from the above that the log processing method mentioned in this embodiment focuses on discovering the internal relationship between logs, so as to ensure that key logs can be found as accurately as possible. After the first log file is processed by the log processing method provided in this embodiment, the second log file can be obtained. Compared with the first log file, the second log file has greatly reduced the number of repeated records and damaged logs. Some overly detailed information is also eliminated, each log is a unique record, and is based on the first time it appears, so that the storage space occupied by the log can be reduced, irrelevant or error logs can be removed, and duplicate logs can be merged, thereby Speed up your analysis. It has been verified that the log files processed by the log processing method provided in this embodiment can reduce the storage space by 90%, and the analysis efficiency is doubled.

It should be noted that the above is only for illustration and does not limit the technical solution of the present invention.

Compared with the prior art, in the log processing method provided in this embodiment, the server compares the bag-of-words of the log to be processed with the bag-of-words of the historical log, and can determine the relationship between the log to be processed and the bag-of-words of the historical log The relationship, which reflects the relationship between the logs to be processed and the historical logs. Since the server can determine the relationship between the log to be processed and the historical log, it can selectively retain the recorded log according to the relationship, reducing the number of recorded logs and reducing the burden of manual log processing.

The second embodiment of the present invention relates to a log processing method. This embodiment is a further improvement on the first embodiment. The specific improvement is: after all pending logs are processed, a mapping is generated according to the reference log file for easy analysis of subsequent received logs.

Specifically, as shown in FIG. 2 , in this embodiment, steps 201 to 208 are included, wherein, steps 201 to 204 are substantially the same as steps 101 to 104 in the first embodiment, and are not repeated here. repeat. The following mainly introduces the differences between the second embodiment and the first embodiment:

Execute step 201 to step 204.

After processing all pending logs, perform the following steps:

Step 205: Obtain the retained logs, use the retained logs as reference logs, and determine the similarity between the reference logs.

Specifically, in the process of determining the similarity between any two reference logs, the server performs the following operations respectively: determine the similarity between the bags of words of the two reference logs; degree, as the similarity between two reference logs. For example, the reference log includes log 1 and log 2, the bag of words of log 1 is bag of words 3, the bag of words of log 2 is bag of words 4, the similarity between log 1 and log 2 = the difference between bag of words 3 and bag of words 4 similarity between.

In the first example, similarity between bag-of-words3 and bag-of-words4 = number of words that appear in bag-of-words3 and bag-of-words4/(number of words in bag-of-words3 + words in bag-of-words4- The number of words that appear in both bag-of-words 3 and bag-of-words 4) * 100%.

In the second example, the server removes meaningless words such as prepositions and conjunctions in word bag 3 to obtain word bag 5, removes meaningless words such as prepositions and conjunctions in word bag 4 to obtain word bag 6, word bag The similarity between 3 and bag of words 4 = the number of words that appear in bag of words 5 and bag of words at the same time / (the number of words in bag of words 5 + the number of words in bag of words 6 - the number of words that appear in bag of words 5 and bag of words at the same time number of words in 6)*100%.

Step 206: Classify the reference logs according to the similarity between the reference logs.

Specifically, the similarity of logs of the same category is greater than a first preset value. Wherein, the first preset value may be any percentage greater than 0 and smaller than 1, for example, the first preset value is a percentage of 30% to 60%, such as 40%.

For example, the first preset value is 40%, and the reference log includes 5 logs, numbered 1 to 5 respectively. Table 1 shows the pre-processed information, processed information and bag-of-words size of each log.

Table 1

Among them, mce:[Hardware Error]:Machine check:Processor context corrupt indicates that the machine check is abnormal: [hardware error]: machine check: the processor context is damaged; Kernel panic-not syncing:Timeout:Not all CPU entered broadcast exception handler indicates the kernel Crash - out of sync: timeout: not all CPUs entered broadcast exception handler; sbridge:Lost 47 memory errors means lost 47 memory errors; sbridge:HANDLING MCE MEMORY ERROR means handle MCE memory errors; mce:[Hardware Error]: CPU 17:Machine Check Exception: 5 Bank 12:be00003f001000c3 indicates that an exception was found when checking CPU No. 17, and the abnormal location is in 5 Bank 12:be00003f001000c3. Use the method provided in the second example to calculate the similarity between each log, and the similarity between each log is shown in Table 2.

Form 2

It can be seen from the above table that the log numbered 1 and the log numbered 5 belong to the log reflecting the same fault category, the log numbered 3 and the log numbered 4 belong to the log reflecting the same fault category, and the log numbered 2 is independently A type of log. By continuously learning the existing logs, the fault categories of the reference logs can be continuously enriched.

Step 207: Determine the fault category of each type of log, and the fault level of each reference log.

Specifically, the server displays each type of log to the maintenance personnel, the maintenance personnel determines and inputs the fault type of the log of this type, and the server determines the fault type of each type of log according to the fault type input by the user. The server can display each reference log to the maintenance personnel, the maintenance personnel determines and inputs the failure level of each reference log, and the server determines the failure level of each reference log according to the failure level of retained logs input by the maintenance personnel.

It should be noted that those skilled in the art can understand that in practical applications, the server can also automatically identify the words in the bag of words of each log to determine the fault category and fault level of the log. This embodiment does not limit the determination of each type of word The failure category of the log, and the failure level of each referenced log.

In one example, the bags of words of the same fault category are divided into five fault levels of A, B, C, D, and E in descending order of importance.

Step 208: Generate a mapping file according to the reference log, the fault category of the reference log, and the fault level of the reference log.

Specifically, the mapping file is a mapping from a reference log to a category of a reference log, and a mapping from a reference log to a fault level of a reference log, and is used to analyze the subsequently received log and determine the fault type and fault level of the subsequently received log.

In an example, after the mapping file is generated, the server uses the mapping file to analyze subsequent logs received. The process of analyzing logs by the server is as follows: the server obtains the logs to be analyzed; processes the logs to be analyzed to obtain the bag of words of the logs to be analyzed; determines the similarity between the bag of words of the logs to be analyzed and the bag of words of the reference log in the mapping file; The similarity between the bag of words of the analysis log and the bag of words of the reference log is used to determine the fault category of the log to be analyzed and the fault level of the log to be analyzed.

In one example, the server determines the fault category of the log to be analyzed and the fault level of the log to be analyzed according to the similarity between the bag of words of the log to be analyzed and the bag of words of the reference log, including but not limited to the following two:

Method 1: The server uses the fault category of the reference log with the highest similarity with the bag of words of the log to be analyzed as the fault category of the log to be analyzed; uses the fault level of the reference log with the highest similarity as the fault level of the log to be analyzed.

Method 2: The server judges whether there is a word bag of the reference log whose similarity with the word bag of the log to be analyzed is greater than the second preset value in the mapping file; The fault category of the reference log is used as the fault category of the first log, and the fault level of the reference log with the highest similarity is used as the fault level of the log to be analyzed; The failure level of the analysis log is unknown level.

In an example, the method for the server to determine the similarity between the bag of words of the log to be analyzed and the bag of words of the reference log in the mapping file is: according to the constraint relationship between the bag of words of the log to be processed, the bag of words of the reference log, and the similarity, Calculate similarity; Wherein, constraint relation is: similarity=occur in the word quantity of the word bag of the first log and the word bag of reference log/(the word quantity of the word bag of the first log+reference log word quantity-simultaneously The number of words that appear in the bag-of-words of the first log and the bag-of-words of the reference log).

It should be noted that, the process of the server analyzing the log to be analyzed can refer to the process of the server analyzing the first log in the third embodiment and the fourth embodiment, which will not be described in detail here, and those skilled in the art can refer to the third embodiment and The content of the fourth embodiment is to analyze the log to be analyzed.

It should be noted that the above is only for illustration and does not limit the technical solution of the present invention.

Compared with the prior art, the log processing method provided in this embodiment, since the server can determine the relationship between the log to be processed and the historical log, it can selectively retain the recorded log according to the relationship, reducing the number of records The number of logs reduces the burden of manual log processing. The server generates a mapping file based on the processed logs, so that the server can automatically analyze the subsequent logs received, which improves the intelligence of the server, reduces the workload of maintenance personnel, and reduces the pressure of manual log analysis.

The third embodiment of the present invention relates to a log analysis method, which is applied to a server. As shown in Figure 3, the following steps are included:

Step 301: Obtain the first log to be processed.

Step 302: Process the first log to obtain a bag of words of the first log.

Specifically, the log is mainly composed of words. A log is converted into a bag of words consisting of several words. The bag of words does not contain repeated words. The relationship between the logs can be determined through the relationship between the bags of words in the log.

In an example, the server deletes the variable in the first log, and the variable is a preset parameter; splits the first log after the variable is deleted into N words to generate a bag of words of the log to be processed, and N is a positive integer. Wherein, the preset parameters include at least any one of bad track location information, bad track number information, bad block location information, and bad block number information.

In an example, the server deletes the variable in the first log by: identifying the number in the body part of the first log; and deleting the number in the body part of the first log.

It should be noted that the process of the server processing the first log to obtain the bag of words of the first log is roughly the same as the process of processing the log to be processed and obtaining the bag of words of the log to be processed in the first embodiment. A skilled person may perform this step with reference to relevant content of the first embodiment.

Step 303: Determine the similarity between the bag-of-words of the first log and the bag-of-words of the reference log in the mapping file.

Specifically, the mapping file includes the bag of words of the reference log, and the fault category of the reference log, and/or, the fault level of the reference log. For the creation method of the mapping file, reference may be made to the relevant content of the log processing method mentioned in the second embodiment, and details are not repeated here.

The method for the server to determine the similarity between the bag of words of the first log and the bag of words of the reference log in the mapping file includes but not limited to the following two:

Method 1: The server calculates the similarity according to the bag of words in the first log, the bag of words in the reference log, and the constraint relationship of similarity; where the constraint relationship is: similarity = the bag of words that appears in the first log and the word bag of the reference log at the same time The number of words in the bag of words / (the number of words in the bag of words in the first log + the number of words in the reference log - the number of words that appear in both the bag of words in the first log and the word bag in the reference log).

Method 2: The server removes invalid words in the bag of words of the first log and the bag of words of the reference log. The invalid words are pre-specified words, for example, various prepositions, conjunctions and other meaningless words. After removing invalid words in the bag of words of the first log and the bag of words of the reference log, the similarity is calculated according to the constraint relationship of the bag of words of the first log, the bag of words of the reference log and the similarity.

It is worth mentioning that since the same invalid word does not represent the fault category of the two logs, and/or, the fault level is the same, therefore, removing invalid words in the bag of words of the first log and the bag of words of the reference log can avoid The effect of invalid words on the similarity between the first log's bag-of-words and the reference log's bag-of-words.

In one example, the server is provided with a shielded word bag. Before determining the word bag of the first log and the word bag of the reference log, it is judged whether the word bag of the first log contains all the words in the shielded word bag. If it is determined to be, Then ignore the first log, otherwise, execute the next steps.

Step 304: According to the similarity between the bag-of-words of the first log and the bag-of-words of the reference log, determine the fault category of the first log, and/or, the fault level of the first log.

Specifically, since the mapping file includes the bag of words of the reference log, the fault category of the reference log, and/or the fault level of the first log, the server can use the mapping file to analyze the first log.

The method for analyzing the first log by the server using the mapping file is described below with an example.

Method a, the server uses the fault category of the reference log with the highest similarity with the bag of words of the first log in the mapping file as the fault category of the first log, and/or, sets the fault category with the highest similarity with the bag of words of the first log The failure level of the reference log is used as the failure level of the first log.

Specifically, if the mapping file includes the bag of words of the reference log and the fault category of the reference log, the server will use the fault category of the reference log that has the highest similarity with the bag of words of the first log in the mapping file as the fault of the first log. category. If the mapping file includes the bag of words of the reference log and the failure level of the reference log, the server uses the failure level of the reference log in the mapping file with the highest similarity to the bag of words of the first log as the failure level of the first log. If the mapping file includes the bag of words of the reference log, the fault category of the reference log, and the fault level of the reference log, the server will use the fault category of the reference log with the highest similarity to the bag of words of the first log in the mapping file as the first log The fault category of the first log will be the fault level of the reference log with the highest similarity with the bag of words of the first log as the fault level of the first log.

Method b, the server judges whether there is a bag of words in the reference log whose similarity with the bag of words of the first log is greater than the second preset value in the mapping file; The failure category of the reference log is used as the failure category of the first log, and/or, the failure level of the reference log with the highest similarity with the word bag of the first log is used as the failure level of the first log; otherwise, the first log is determined. The failure category of the bag-of-words log is an unknown category, and/or, the failure level of the first log is determined to be an unknown level. Wherein, the second preset value can be set to a value greater than 0 and less than 1 as required, for example, set to a value of 30% to 60%, such as 40%.

In an example, logs of the same fault category may be divided into five fault levels A, B, C, D and E, wherein E is an unknown level. For the logs of the same fault category and the same fault level, there may be some differences in their importance. In this case, M sub-levels can be derived under each fault level. For example, for fault level A, sub-levels A1, A2, A3, A4, A5, A6, A7, A8, A9, and A10, so that the bag-of-words of logs with the same fault level still have room for differentiation.

It is worth mentioning that if there is no bag of words in the reference log whose similarity with the bag of words of the first log is greater than the second preset value in the mapping file, it means that the first log and the reference log in the mapping file do not belong to the same The fault category, the server marks the fault category of the bag of words in the first log as an unknown category, so that the maintenance personnel can find out the undiscovered new fault category in time.

In one example, the mapping file includes a bag of words for the reference logs, a failure category for the reference logs, and a failure level for the reference logs. After the server determines that the fault category of the bag of words in the first log is an unknown category, and determines that the fault level of the first log is an unknown level, the server reports the first log; according to the fault category and fault level specified by the user, determine the fault of the first log category and the failure level of the first log; according to the bag of words of the first log, the failure category of the first log and the failure level of the first log, the mapping file is updated.

It is worth mentioning that the server reports logs of unknown categories and levels in a timely manner, and updates the mapping file according to the fault category and fault level evaluated by the user, so that the mapping file can be continuously expanded and improved, and the accuracy of the server's log analysis can be improved.

It should be noted that, in extreme cases, there are multiple reference logs with the highest similarity, that is, the bags of words of multiple reference logs have the same degree of similarity as the bag of words of the first log, and the similarity is the highest. The fault category of the first log is set to the unknown category, and the fault level of the first log is set to the unknown level.

Compared with the prior art, in the log analysis method provided by this embodiment, the server can use the mapping file to analyze the first log, determine the fault type of the first log, and/or, improve the fault level of the first log This improves the intelligence of the server and reduces the pressure on the maintenance staff from analyzing logs. In addition, in the process of analyzing logs, the logs of unknown categories can be fed back in time, and the mapping files can be updated in time according to the fault categories and fault levels specified for the logs of unknown categories, and the mapping files can be continuously improved. Use this mapping file to analyze the log to get more accurate conclusions.

The fourth embodiment of the present invention relates to a log analysis method. This embodiment is a further improvement on the third embodiment. The specific improvement is that after step 304, other related steps are added.

Specifically, as shown in FIG. 4 , in this embodiment, steps 401 to 408 are included, wherein, steps 401 to 403 are substantially the same as steps 301 to 303 in the first embodiment, and are not repeated here. repeat. The following mainly introduces the differences between the fourth embodiment and the third embodiment:

Execute step 401 to step 403.

Step 404: According to the similarity between the bag-of-words of the first log and the bag-of-words of the reference log, determine the fault category of the first log and the fault level of the first log.

Specifically, the mapping file includes the bag-of-words of the reference logs, the fault categories of the reference logs, and the fault levels of the reference logs. According to the similarity between the bag of words of the first log and the bag of words of the reference log, the server determines the reference log with the highest similarity to the bag of words of the first log, and uses the fault category of the reference log with the highest similarity as the fault category of the first log , taking the fault level of the reference log with the highest similarity as the fault level of the first log.

Step 405: Determine whether there is a second log in the recorded logs.

Specifically, the second log is a log belonging to the same fault category as the first log. If the server determines that there is a second log in the recorded logs, step 406 is performed; otherwise, step 407 is performed.

Step 406: Compare the fault level of the first log with the fault level of the second log, and update the recorded log according to the comparison result.

Specifically, if the server determines that the comparison result indicates that the failure level of the first log is higher than that of the second log, the server overwrites the second log with the first log; if it determines that the comparison result indicates that the failure level of the first log is not higher than that of the second log The failure level of the log, do not cover the second log with the first log, so that the log with a high failure level covers the log with a low failure level.

It is worth mentioning that the log with a high fault level covers the low fault level log, which reduces the number of recorded logs and reduces the time and energy wasted by maintenance personnel on analyzing logs. Maintenance personnel can more intuitively know the key logs with the highest fault level in each fault category, so that maintenance personnel can repair more serious faults in time.

It should be noted that those skilled in the art can understand that in practical applications, the recorded logs can also be updated in other ways, for example, the first log and the second log are stored in the server in the form of a table. If the fault level of the first log is higher than the second log, the first log is recorded before the second log, and if the fault level of the first log is lower than the first log, the first log is recorded after the second log, This embodiment does not limit the method of updating logs.

Step 407: Record the first log.

Specifically, since no log of the fault category is recorded, the server may record the first log in a log file, so that maintenance personnel can obtain information about the log.

Compared with the prior art, in the log analysis method provided by this embodiment, after the fault level of the first log is higher than that of the recorded log of the same fault category, the recorded log is replaced with the first log, It ensures that the importance of the recorded logs is continuously improved, so as to achieve the effect of continuous escalation of alarms.

The step division of the above various methods is only for the sake of clarity of description. During implementation, it can be combined into one step or some steps can be split and decomposed into multiple steps. As long as they include the same logical relationship, they are all within the scope of protection of this patent. ; Adding insignificant modifications or introducing insignificant designs to the algorithm or process, but not changing the core design of the algorithm and process are all within the scope of protection of this patent.

The fifth embodiment of the present invention relates to a server, as shown in FIG. 5 , including: at least one processor 501; and a memory 502 communicatively connected to at least one processor 501; The instructions executed by the processor 501 are executed by at least one processor 501, so that the at least one processor 501 can execute the log processing method mentioned in the foregoing implementation manner.

The sixth embodiment of the present invention relates to a server, as shown in FIG. 6 , including: at least one processor 601; and a memory 602 communicatively connected to at least one processor 601; The instructions executed by the processor 601 are executed by at least one processor 601, so that the at least one processor 601 can execute the log analysis method mentioned in the above implementation manner.

In the fifth embodiment and the sixth embodiment, the server includes: one or more processors and a memory, and one processor is taken as an example in FIG. 5 and FIG. 6 . The processor and the memory may be connected through a bus or in other ways. In FIG. 5 and FIG. 6, connection through a bus is used as an example. The memory, as a non-volatile computer-readable storage medium, can be used to store non-volatile software programs, non-volatile computer-executable programs, and modules. The processor executes various functional applications and data processing of the device by running non-volatile software programs, instructions and modules stored in the memory.

The memory may include a program storage area and a data storage area, wherein the program storage area may store an operating system and an application program required by at least one function; the data storage area may store an option list and the like. In addition, the memory may include high-speed random access memory, and may also include non-volatile memory, such as at least one magnetic disk storage device, flash memory device, or other non-volatile solid-state storage devices. In some embodiments, the memory may optionally include a memory that is remotely located relative to the processor, and these remote memories may be connected to external devices through a network. Examples of the aforementioned networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

One or more modules are stored in the memory, and when executed by one or more processors, execute the log processing method or the log analysis method in any method implementation above.

The above-mentioned products can execute the methods provided in the embodiments of this application, and have the corresponding functional modules and beneficial effects for executing the methods. For technical details not described in detail in this embodiment, please refer to the methods provided in the embodiments of this application.

A seventh embodiment of the present invention relates to a computer-readable storage medium storing a computer program. When the computer program is executed by the processor, the embodiment of the above log processing method is realized.

An eighth embodiment of the present invention relates to a computer-readable storage medium storing a computer program. When the computer program is executed by the processor, the embodiment of the above log analysis method is realized.

That is, those skilled in the art can understand that all or part of the steps in the method of the above-mentioned embodiments can be completed by instructing related hardware through a program, the program is stored in a storage medium, and includes several instructions to make a device ( It may be a single-chip microcomputer, a chip, etc.) or a processor (processor) to execute all or part of the steps of the methods described in the various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), magnetic disk or optical disk, and other media that can store program codes.

Those of ordinary skill in the art can understand that the above-mentioned embodiments are specific examples for realizing the present invention, and in practical applications, various changes can be made to it in form and details without departing from the spirit and spirit of the present invention. scope.

Claims (12)

1. A method of analyzing a log, comprising:

acquiring a first log to be processed;

processing the first log to obtain a word bag of the first log;

determining similarity of a word bag of the first log and a word bag of a reference log in a mapping file, wherein the mapping file comprises the word bag of the reference log and a fault class of the reference log and/or a fault level of the reference log;

determining a fault class of the first log and/or a fault level of the first log according to the similarity of the word bags of the first log and the reference log;

after determining the fault class of the first log and the fault level of the first log according to the similarity between the word bag of the first log and the word bag of the reference log, the method for analyzing the log further comprises:

judging whether a second log exists in the recorded logs, wherein the second log is a log belonging to the same fault type as the first log;

If the fault level of the first log and the fault level of the second log are determined to exist, updating the recorded log according to a comparison result;

and if the first log does not exist, recording the first log.

2. The method for analyzing the log according to claim 1, wherein the determining the fault class of the first log and/or the fault level of the first log according to the similarity between the word bag of the first log and the word bag of the reference log specifically comprises:

and taking the fault class of the reference log with the highest similarity with the word bag of the first log as the fault class of the first log, and/or taking the fault level of the reference log with the highest similarity with the word bag of the first log as the fault level of the first log.

3. The method for analyzing the log according to claim 1, wherein the determining the fault class of the first log and/or the fault level of the first log according to the similarity between the word bag of the first log and the word bag of the reference log specifically comprises:

judging whether a word bag of a reference log with similarity to the word bag of the first log larger than a second preset value exists in the mapping file or not;

If so, taking the fault class of the reference log with the highest similarity with the word bag of the first log as the fault class of the first log, and/or taking the fault level of the reference log with the highest similarity with the word bag of the first log as the fault level of the first log;

otherwise, determining the fault class of the bag of words of the first log as an unknown class, and/or determining the fault level of the first log as an unknown level.

4. A method of analysing a log according to claim 3, wherein the map file comprises a bag of words for the reference log, a fault class for the reference log and a fault level for the reference log;

after determining that the fault class of the bag of words of the first log is an unknown class and determining that the fault level of the first log is an unknown level, the log analysis method further comprises:

reporting the first log;

determining the fault class of the first log and the fault level of the first log according to the fault class and the fault level designated by the user;

and updating the mapping file according to the word bag of the first log, the fault class of the first log and the fault level of the first log.

5. The method for analyzing a log according to claim 1, wherein updating the recorded log according to the comparison result specifically comprises:

if the comparison result is determined to indicate that the fault level of the first log is higher than that of the second log, covering the second log by the first log;

and if the comparison result is determined to indicate that the fault level of the first log is not higher than the fault level of the second log, the second log is not covered by the first log.

6. A method of analyzing logs according to any of claims 1 to 3, wherein determining the similarity of the bag of words of the first log to the bag of words of the reference log in the mapping file comprises:

calculating the similarity according to the constraint relation of the word bags of the first log, the word bags of the reference log and the similarity; wherein, the constraint relation is: the similarity = the number of words of the bag of words that occur simultaneously in the first log and the bag of words of the reference log/(the number of words of the bag of words of the first log + the number of words of the reference log-the number of words of the bag of words that occur simultaneously in the first log and the bag of words of the reference log).

7. The method according to claim 6, wherein before the calculation of the similarity according to the constraint relation of the bag of words of the first log, the bag of words of the reference log, and the similarity, the method further comprises:

removing invalid words in the word bags of the first log and the reference log; wherein the invalid word is a pre-specified word.

8. The method for analyzing a log according to claim 1, wherein the processing the first log to obtain a bag of words of the first log specifically includes:

deleting a variable in the first log, wherein the variable is a preset parameter;

splitting the first log after deleting the variables into N words, generating a word bag of the log to be processed, wherein N is a positive integer.

9. The method according to claim 8, wherein the preset parameters include at least any one of position information of a bad track, number information of a bad track, position information of a bad block, and number information of a bad block.

10. The method for analyzing a log according to claim 9, wherein deleting the variable in the first log specifically includes:

Identifying a number of a body portion of the first log;

and deleting the number of the text part of the first log.

11. A server, comprising: at least one processor; the method comprises the steps of,

a memory communicatively coupled to the at least one processor; wherein the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the method of log analysis of any one of claims 1 to 10.

12. A computer readable storage medium storing a computer program, wherein the computer program when executed by a processor implements the method of analyzing logs according to any of claims 1 to 10.