CN107315756B - A log processing method and device - Google Patents

Abstract

The embodiment of the invention discloses a log processing method, which is applied to processing nodes and comprises the following steps: acquiring log data corresponding to a first service log to be processed from a data cache; acquiring first processed log data according to the first service log to be processed and the log data; determining whether other processing nodes access log data corresponding to the first service log to be processed in the data cache or not; if no other processing node accesses the log data corresponding to the first service log to be processed in the data cache, storing the first processed log data in the data cache; and if other processing nodes access the log data corresponding to the first service log to be processed in the data cache, storing the first processed log data in a temporary cache. Furthermore, the embodiment of the invention also discloses a log processing device.

Description

A log processing method and device

technical field

The present invention relates to data processing technology in the communication field, and in particular, to a log processing method and device.

Background technique

With the development of communication technology, more and more users are using mobile phone, fixed line, tablet and computer broadband. Operators provide users with various services, and users can choose different services according to their own needs.

In the prior art, the channels for users to handle business include physical business halls and online business halls. When the user handles business, a business log is generated in real time, and the business log records information such as the user ID, the business ID handled by the user, and the processing time. . Through real-time analysis of business logs, real-time monitoring of business processing time, the number of business processing failures, and business processing failure links and other indicators, the operating status of the business processing system can be obtained in real time, system faults can be located in time, system fault resolution time can be shortened, and business processing efficiency and efficiency can be improved. user experience.

Due to the large number of users who handle the business, the amount of business log data generated every day is large. When analyzing business logs, the processing performance of a single server is difficult to meet the demand. Therefore, a cluster distributed architecture needs to be used for processing, that is, a large amount of log data is allocated. Go to multiple processing nodes for processing to improve processing speed. Since the business logs of the same business need to be processed in context, the business logs of the same business need to be assigned to the same processing node. However, there are various business types handled by users. Some businesses generate more business logs, and some businesses generate The number of business logs allocated to different processing nodes is likely to vary greatly, which in turn leads to unbalanced loads on multiple processing nodes and affects system stability.

SUMMARY OF THE INVENTION

In order to solve the above technical problems, the embodiments of the present invention are expected to provide a log processing method and device, which can use multiple nodes to process the service log of the same service, so as to balance the load of multiple processing nodes and improve system stability.

The technical scheme of the present invention is realized as follows:

On the one hand, an embodiment of the present invention provides a log processing method, which is applied to a processing node, including:

Obtain log data corresponding to the first pending business log from the data cache;

According to the first to-be-processed business log and the log data, obtain first processed log data;

determining whether there are currently other processing nodes accessing log data corresponding to the first pending business log in the data cache;

If no other processing node currently accesses the log data corresponding to the first to-be-processed business log in the data cache, the first processed log data is stored in the data cache; if other processing nodes currently access the data cache For the log data corresponding to the first service log to be processed, the first processed log data is stored in a temporary cache.

Optionally, after storing the first processed log data in the temporary cache, the method further includes:

Obtain log data corresponding to the second business log to be processed from the data cache;

obtaining the first processed log data from the temporary cache;

The second processed log data is acquired according to the second to-be-processed business log, log data corresponding to the second to-be-processed business log, and the first processed log data.

Optionally, the obtaining the first processed log data from the temporary cache includes:

According to the key value of the second to-be-processed business log, the first processed log data corresponding to the key value is acquired from the temporary cache.

Optionally, before acquiring the first processed log data according to the first service log to be processed and the log data, the method further includes:

performing information enhancement processing on the first to-be-processed business log to obtain a first enhanced to-be-processed business log;

The obtaining of the first processed log data according to the first to-be-processed business log and the log data includes:

Obtain first processed log data according to the first enhanced service log to be processed and the log data.

Optionally, before the information enhancement processing is performed on the first service log to be processed and the first enhanced service log to be processed is obtained, the method further includes:

The first business log to be processed is preprocessed, and the preprocessing includes log information format normalization, null value, and abnormal value checking.

On the other hand, an embodiment of the present invention provides a log processing device, including:

a first obtaining unit, configured to obtain log data corresponding to the first business log to be processed from the data cache;

a second obtaining unit, configured to obtain the first processed log data according to the first to-be-processed business log and the log data;

a determining unit, configured to determine whether there are currently other processing nodes accessing log data corresponding to the first to-be-processed business log in the data cache;

A storage unit, configured to store the first processed log data in the data cache if no other processing node currently accesses the log data corresponding to the first to-be-processed business log in the data cache; if other processing nodes currently exist The log data corresponding to the first to-be-processed business log in the data cache is accessed, and the first processed log data is stored in the temporary cache.

Optionally, the device further includes:

a third obtaining unit, configured to obtain log data corresponding to the second business log to be processed from the data cache;

a fourth obtaining unit, configured to obtain the first processed log data from the temporary cache;

A fifth obtaining unit, configured to obtain second processed log data according to the second to-be-processed business log, log data corresponding to the second to-be-processed business log, and the first processed log data.

Optionally, the fourth obtaining unit is specifically used for:

According to the key value of the second to-be-processed business log, the first processed log data corresponding to the key value is acquired from the temporary cache.

Optionally, the device further includes:

an enhancement unit, configured to perform information enhancement processing on the first to-be-processed business log, and obtain the first enhanced to-be-processed business log;

The second obtaining unit is specifically used for:

Obtain first processed log data according to the first enhanced service log to be processed and the log data.

Optionally, the device further includes:

A preprocessing unit, configured to perform preprocessing on the first to-be-processed business log, where the preprocessing includes log information format normalization, null value, and abnormal value checking.

Embodiments of the present invention provide a log processing method and device. The log processing method is applied to a processing node, and includes: acquiring log data corresponding to a first service log to be processed from a data cache; log and the log data, obtain the log data after the first processing; determine whether there are other processing nodes currently accessing the log data corresponding to the first pending business log in the data cache; if no other processing nodes currently access the log data; log data corresponding to the first service log to be processed in the data cache, and store the first processed log data in the data cache; if other processing nodes currently access the first service to be processed in the data cache For log data corresponding to the log, the first processed log data is stored in a temporary cache. Compared with the prior art, since the first post-processing log data can be stored in a data cache or a temporary cache according to different situations, information sharing between different processing nodes is realized, so that the business logs generated by the same business can be stored in the data cache. Different processing nodes perform processing, so that the same number of business logs can be allocated to each processing node, so that the load of each processing node is balanced and the system stability is improved.

Description of drawings

FIG. 1 is a schematic flowchart 1 of a log processing method provided by an embodiment of the present invention;

2 is a schematic structural diagram of a log processing system according to an embodiment of the present invention;

3 is a schematic flowchart 2 of a log processing method provided by an embodiment of the present invention;

FIG. 4 is a schematic structural diagram 1 of a log processing apparatus provided by an embodiment of the present invention;

FIG. 5 is a schematic structural diagram 2 of a log processing apparatus provided by an embodiment of the present invention;

FIG. 6 is a schematic structural diagram 3 of a log processing apparatus provided by an embodiment of the present invention;

FIG. 7 is a schematic structural diagram 4 of a log processing apparatus according to an embodiment of the present invention.

Detailed ways

The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention.

Example 1

An embodiment of the present invention provides a log processing method, which is applied to a processing node. The processing node may be a hardware facility such as a processor and a single-chip microcomputer, or may be a hardware-dependent software program, which is not limited in the embodiment of the present invention. As shown in Figure 1, the log processing method includes:

Step 101: Acquire log data corresponding to a first pending business log from a data cache.

For example, it is assumed that the user is currently in the first business in class, and the first business log to be processed is a business log generated by the first business. After receiving the allocated first service to be processed, the processing node may first obtain the key value of the first service to be processed, and then send the key value to the data cache. The data cache stores multiple log data according to different key values. After receiving the key value sent by the processing node, the data cache queries the log data corresponding to the key value, and then sends the log data to the processing node. The key value may include information such as a service identifier corresponding to the first to-be-processed service log, and a user identifier corresponding to the first to-be-processed service log.

Step 102: Acquire first processed log data according to the first service log to be processed and the log data.

For example, since the processing of a business will last for a long time, generally about half an hour, some businesses will take up to two hours, that is, multiple business logs generated in a long period of time are related and need to be contextualized. Associated or aggregated processing. After each processing, the latest processing result of the corresponding business is stored in the data cache according to the key value of the business log processed this time, and the processing result is the log data. When processing business logs, you can obtain log data corresponding to the key value from the data cache according to the key value of the business log to be processed, and then obtain new log data according to the log data and the business log to be processed. For example, the log data corresponding to the first to-be-processed business log obtained in the data cache is the processing result of multiple business logs generated by the first business before the first to-be-processed business log is generated. After the business log, according to the first pending business log and the log data corresponding to the first pending business log obtained from the data cache, new log data, that is, the first processed log data, can be acquired.

Step 103: Determine whether there are other processing nodes currently accessing log data corresponding to the first to-be-processed service log in the data cache.

For example, since there may be multiple processing nodes that simultaneously process the business log generated by the first service, there may be multiple processing nodes that simultaneously access the log data of the first service in the data cache, and the log data is the first service to be processed. The log data corresponding to the log. If the processing node directly stores the first processed log data in the data cache after obtaining the first processed log data, and replaces the log data corresponding to the first pending business log stored before the data cache, other processing nodes will be affected. In the case where the processing cannot be completed, it can be first determined whether there are other nodes accessing the log data corresponding to the first service log to be processed in the data cache.

Step 104: If there is currently no other processing node accessing the log data corresponding to the first pending business log in the data cache, store the first processed log data in the data cache; if there are currently other processing nodes accessing the log data. log data corresponding to the first service log to be processed in the data cache, and store the first processed log data in the temporary cache.

Exemplarily, if no other processing node currently accesses the log data corresponding to the first to-be-processed business log in the data cache, it means that replacing the log data corresponding to the first to-be-processed business log will not affect the processing process of other processing nodes, Therefore, the first processed log data can be directly stored in the data cache.

If other processing nodes currently access the log data corresponding to the first service log to be processed in the data cache, it means that replacing the log data corresponding to the first service log to be processed may affect the processing process of other processing nodes, causing other processing The node cannot obtain relevant information, so the log data after the first processing can be stored in a temporary cache, which is different from the data cache, so that the latest log data of the first service is saved without affecting other The processing process of the processing node.

In this way, since the first processed log data can be stored with the data cache or temporary cache according to different situations, information sharing between different processing nodes is realized, so that the business logs generated by the same business can be processed in different The nodes are processed, so that the same number of business logs can be allocated to each processing node, so that the load of each processing node is balanced and the system stability is improved.

Further, after the first processed log data is stored in the temporary cache, log data corresponding to the second business log to be processed may be obtained from the data cache, and the first log data may be obtained from the temporary cache. After processing the log data, and then obtaining the second processed log data according to the second pending business log, the log data corresponding to the second pending business log, and the first processed log data.

For example, in the process of handling the first business, it is possible to continue to generate a new business log, that is, a second business log to be processed, and the second business log to be processed has the same key value as the first business log to be processed. , indicating that both are business logs generated by the first business. When processing the second to-be-processed business log, the processing result of the business log generated before the second to-be-processed business log needs to be combined, but it is impossible to determine whether the latest processing result is the log data stored in the data cache or the log stored in the temporary cache. Therefore, the log data corresponding to the second business log to be processed can be obtained from the data cache, the first processed log data can be obtained from the temporary cache, and then the log data corresponding to the second business log to be processed can be obtained according to the second business log to be processed. The log data and the first processed log data are obtained, and the second processed log data is obtained.

Optionally, when acquiring the first processed log data from the temporary cache, the data corresponding to the key value may be acquired from the temporary cache according to the key value of the second to-be-processed business log. the first processed log data.

For example, the first processed log data can be stored in the temporary cache according to the key value of the first pending business log, and when the processing node receives the second pending business log, it can first obtain the second pending business log. The key value, since the second business log to be processed and the first business log to be processed are both business logs generated by the first business, the key value of the second business log to be processed and the first business log to be processed are the same, so according to the second For the key value of the business log to be processed, the first processed log data corresponding to the key value of the second business log to be processed can be obtained from the temporary cache.

Further, before obtaining the first post-processing log data according to the first business log to be processed and the log data, information enhancement processing may also be performed on the first business log to be processed to obtain the first enhanced business log to be processed. business log, and then obtain the first post-processing log data according to the first enhanced business log to be processed and the log data.

For example, when a user handles a business, in order to improve the business processing time, the business log is composed of brief information. Assuming that the user handles the broadband service, the generated first pending service log only includes the broadband account, but the user's ID card information, home address information, etc. may also be used during the processing of the first pending service log. It is stored corresponding to the broadband account. After receiving the first pending business log, according to the broadband account, the user's ID card information, home address information, etc. can be supplemented into the first pending business log to form the first enhanced pending business log. .

Further, before the information enhancement processing is performed on the first to-be-processed business log and the first enhanced to-be-processed business log is obtained, the first to-be-processed business log may also be preprocessed, and the preprocessing includes: Log information format normalization, null value, outlier checking.

As an example, suppose a user handles broadband services and generates multiple service logs in various formats. Some service logs include 5 fields, and adjacent fields are separated by "/", and some service logs include 4 fields, and adjacent fields are The first field of some business logs is the broadband account number, and the first field of some business logs is garbled, but the log processing system can only process business logs in a preset format. The format indicates that the service log needs to include 5 fields, and adjacent fields are separated by "/", and the first field is the broadband account number, and the last field is the service identifier. After receiving the first business log to be processed, it may be first determined whether the first business log to be processed satisfies the preset format, and if the first business log to be processed does not meet the preset format, the first business log to be processed may be preprocessed , the preprocessing includes log information format normalization, null value, and abnormal value checking, so that the first business log to be processed can be identified and processed by the log processing system. The normalization of the log information format, null value, abnormal value checking, etc. are all in the prior art, which are not repeated in this embodiment of the present invention.

An embodiment of the present invention provides a log processing method, which is applied to a processing node, including: acquiring log data corresponding to a first pending business log from a data cache; and according to the first pending business log and the log data, Obtain the first processed log data; determine whether other processing nodes currently access the log data corresponding to the first pending business log in the data cache; if no other processing nodes currently access the first log data in the data cache log data corresponding to the business log to be processed, store the first processed log data in the data cache; if other processing nodes currently access the log data corresponding to the first business log to be processed in the data cache, store the log data corresponding to the first business log to be processed in the data cache. The first processed log data is stored in a temporary cache. Compared with the prior art, since the first post-processing log data can be stored in a data cache or a temporary cache according to different situations, information sharing between different processing nodes is realized, so that the business logs generated by the same business can be stored in the data cache. Different processing nodes perform processing, so that the same number of business logs can be allocated to each processing node, so that the load of each processing node is balanced and the system stability is improved.

Embodiment 2

An embodiment of the present invention provides a log processing method, which is applied to a log processing system. As shown in FIG. 2 , the log processing system at least includes a transceiver node, a first processing node, a second processing node, a data cache node, and a state cache node. , temporary cache nodes and output nodes. Specifically, as shown in Figure 3, the log processing system includes:

Step 301: The transceiver node acquires multiple service logs generated when multiple users handle services, and executes step 302.

For example, the transceiver node may receive multiple business logs generated when multiple users handle services, that is, multiple business logs may be aggregated to the transceiver node, and distributed through the transceiver node to ensure load balance of each processing node.

Step 302: The transceiver node preprocesses the plurality of service logs, and the preprocessing includes log information format normalization, null value, and abnormal value checking, and step 303 is executed.

For example, since each business log has a different form, but the log processing system can only process business logs in a preset format, after receiving multiple business logs, each business log can be preprocessed first, and the preprocessing includes: Log information format normalization, null value, outlier checking. Specifically, the processing time window of the transceiver node includes two types: the number window and the time window. If the transceiver node uses the number window to process business logs, assuming that the preset number of the number window is 100, each time the transceiver node can obtain 100 business logs are preprocessed; if the sending and receiving nodes use a time window to process business logs, assuming that the preset time period of the time window is 10 seconds (s), the sending and receiving nodes can summarize the business logs obtained every 10s and pre-process them uniformly. deal with.

Step 303: The sending and receiving nodes evenly distribute the multiple service logs to each processing node, and step 304 is executed.

For example, in order to ensure the load balance of each processing node, the sending and receiving nodes can evenly allocate multiple business logs to each processing node, or can also allocate services with the same amount of calculation to each processing node according to the estimated amount of calculation. log, which is not limited in this embodiment of the present invention.

Step 304 , the first processing node performs information enhancement processing on the assigned first service log to be processed, obtains the first enhanced service log to be processed, and executes step 305 .

For example, since the first business log to be processed may only consist of brief information, more specific information may be used when processing the first business log to be processed. Information augmentation processing. For example, when a user handles a broadband service, the generated first pending service log only includes the broadband account, but the user's ID card information, home address information, etc. may also be used in the processing of the first pending service log. Both are stored corresponding to the broadband account. After receiving the first pending business log, the first processing node can supplement the user's ID card information, home address information, etc. into the first pending business log according to the broadband account to form the first pending business log. Enhanced pending business logs.

Step 305 , the first processing node obtains the key value of the first enhanced service log to be processed, and executes step 306 .

For example, the key value of the first enhanced service log to be processed may include information such as a service identifier corresponding to the first service log to be processed, a user identifier corresponding to the first service log to be processed, and the like.

Step 306: The first processing node sends the key value of the first enhanced service log to be processed to the data cache node, and step 307 is executed.

For example, the data cache node may be a memory in the log processing system, used to store the latest log data of multiple services according to different key values.

Step 307: The data cache node sends the log data corresponding to the key value to the first processing node, and step 308 is executed. Send the log data corresponding to the key value

Exemplarily, the data cache node queries locally stored information for log data corresponding to the key value, and then sends the log data to the first processing node.

Step 308: The data cache node sends the key value to the state cache node, and step 309 is executed.

For example, the state cache node may be a memory in the log processing system, which is used to store the access count of each log data. From the access count, it can be seen whether the log data is currently accessed by multiple processing nodes at the same time. After the data cache node sends the log data corresponding to the key value of the first enhanced service log to be processed to the first processing node, it indicates that the first processing node is currently accessing the log corresponding to the key value of the first enhanced service log to be processed Therefore, the data cache node sends the key value to the state cache node, so that the state cache node can update the access number of the log data corresponding to the key value of the first enhanced service log to be processed.

Step 309: The state cache node updates the log data access state corresponding to the key value, so that the access number of the log data is incremented by 1, and step 310 is executed.

For example, if currently only the first processing node accesses the log data corresponding to the key value of the first enhanced service log to be processed, the number of accesses to the log data after the update is 1.

Step 310: The first processing node performs context association processing on the first enhanced service log to be processed and the log data corresponding to the key value, obtains the first processed log data, and executes step 311.

For example, the summary information of the first enhanced service log to be processed and the log data corresponding to the key value may be calculated, including the number of calculated logs, the latest record, the maximum value, the minimum value, the union interval, etc. Not limited.

Step 311: The first processing node sends the key value to the state cache node, and step 312 is executed.

For example, after the first processing node completes the processing, it indicates that the first processing node no longer needs to access the log data corresponding to the first enhanced service log to be processed in the data cache, so the key value of the first enhanced service log to be processed can be sent. To the state cache node, so that the state cache node can update the access number of the log data corresponding to the key value of the first enhanced service log to be processed.

Step 312 , the state cache node updates the log data access state corresponding to the key value, so that the access number of the log data is decremented by 1, and step 313 is executed.

For example, if currently only the first processing node accesses the log data, the number of accesses to the log data after the update is 0. If there are currently other processing nodes accessing the log data, the number of accesses to the log data after the update is not 0.

Step 313: The state cache node sends the access number of the log data corresponding to the key value to the first processing node, and step 314 is executed.

For example, the state cache node sends the updated access number of the log data to the first processing node, so that the first processing node can determine whether other processing nodes access the log data.

Step 314: The first processing node determines whether the access number of the log data corresponding to the key value is 0; if the access number of the log data corresponding to the key value is 0, perform step 315; If the access count is not 0, step 316 is executed.

For example, if the access number of the log data corresponding to the key value is 0, it means that currently only the first processing node accesses the log data; if the access number of the log data corresponding to the key value is not 0, it means that there are Other processing nodes access this log data.

Step 315: The first processing node stores the first processed log data in the data cache node, and step 325 is executed.

For example, when only the first processing node accesses the log data, it means that replacing the log data corresponding to the first pending business log will not affect the processing process of other processing nodes, so the first processed log data can be directly Stored in the data cache.

Step 316 , the first processing node stores the first processed log data in the temporary cache node, and executes step 317 .

For example, when there are other processing nodes accessing the log data, it means that replacing the log data corresponding to the first pending business log may affect the processing process of other processing nodes, so that other processing nodes cannot obtain relevant information. The log data after the first processing is stored in a temporary cache.

Step 317 , the second processing node performs information enhancement processing on the assigned second service log to be processed, obtains the second enhanced service log to be processed, and executes step 318 .

For example, since the service log of the same service may be simultaneously allocated to multiple processing nodes for processing, the second processing node and the first processing node may both process the service log of the same service. Assuming that the second processing node performs information enhancement processing on another service log of the same service assigned, that is, the second service log to be processed, similarly, the second service log to be processed can be enhanced according to the above method.

Step 318 , the second processing node acquires the key value of the second enhanced service log to be processed, and executes step 319 .

For example, since the second business log to be processed and the first business log to be processed are business logs generated by the same business, the key value of the second business log to be processed is the same as the key value of the first business log to be processed.

Step 319: The second processing node sends the key value to the data cache node and the temporary cache node, and step 320 is executed.

For example, in order to obtain the latest log data of the corresponding service, the key value of the second service log to be processed may be sent to the data cache node and the temporary cache node at the same time, so as to obtain the required information.

Step 320: The data cache node sends the log data corresponding to the key value to the second processing node, and step 321 is executed.

For example, the data cache node queries locally stored information according to the key value of the second service log to be processed, obtains log data corresponding to the key value, and sends it to the second processing node.

Step 321: The temporary cache node sends the first processed log data corresponding to the key value to the second processing node, and step 322 is executed.

For example, the temporary cache node stores multiple log data according to the key value. After receiving the key value of the second pending business log, the information stored locally can be queried according to the key value, and the log corresponding to the key value can be obtained. The data is sent to the second processing node.

Step 322: The second processing node performs contextual association processing on the second enhanced service log to be processed, the log data corresponding to the key value, and the first processed log data, obtains the second processed log data, and executes the steps 323.

For example, the second enhanced service log to be processed, the log data corresponding to the key value, and the summary information of the first processed log data can be calculated, including the number of calculated logs, the latest record, the maximum value, the minimum value, and the union. interval, etc., which are not limited in this embodiment of the present invention.

Step 323: The second processing node sends the second processed log data to the data cache node, replaces the log data corresponding to the key value in the data cache node, and executes step 324.

Exemplarily, the second processed log data is the current latest log data of the corresponding service, so the second processing node may send the second processed log data to the data cache node, so that the data cache node can store it.

Step 324: The data cache node replaces the log data corresponding to the key value in the data cache node with the second processed log data.

Exemplarily, after the data cache node replaces the log data corresponding to the key value in the data cache node with the second processed log data, the output node may also output the latest log processing result in the data cache node, It may be output to a log file, a log database, or other systems, which is not limited in this embodiment of the present invention.

It should be noted that the order of steps of the log processing method provided by the embodiment of the present invention can be appropriately adjusted, and the steps can also be increased or decreased according to the situation. Variations of methods that can be easily imagined should all be included within the protection scope of the present invention, and thus will not be described again.

The embodiment of the present invention provides a log processing method. Compared with the prior art, since the first processed log data can be stored in a data cache or a temporary cache according to different situations, information sharing between different processing nodes is realized. , so that the business logs generated by the same business can be processed on different processing nodes, so that the same number of business logs can be allocated to each processing node, so that the load of each processing node is balanced and the system stability is improved.

Embodiment 3

An embodiment of the present invention provides a log processing apparatus 40, as shown in FIG. 4, including:

The first obtaining unit 401 is configured to obtain log data corresponding to the first service log to be processed from the data cache.

The second obtaining unit 402 is configured to obtain first processed log data according to the first service log to be processed and the log data.

The determining unit 403 is configured to determine whether there are other processing nodes currently accessing log data corresponding to the first to-be-processed service log in the data cache.

The storage unit 404 is configured to store the first processed log data in the data cache if no other processing node currently accesses the log data corresponding to the first pending business log in the data cache; if other processing nodes currently exist The node accesses log data corresponding to the first service log to be processed in the data cache, and stores the first processed log data in the temporary cache.

In this way, since the first processed log data can be stored with the data cache or temporary cache according to different situations, information sharing between different processing nodes is realized, so that the business logs generated by the same business can be processed in different The nodes are processed, so that the same number of business logs can be allocated to each processing node, so that the load of each processing node is balanced and the system stability is improved.

Optionally, as shown in FIG. 5 , the apparatus 40 further includes: a third obtaining unit 405, configured to obtain log data corresponding to the second service log to be processed from the data cache; a fourth obtaining unit 406, using to obtain the log data after the first processing from the temporary cache; the fifth obtaining unit 407 is configured to obtain the log data corresponding to the second business log to be processed, the log data corresponding to the second business log to be processed, and the The first processed log data is obtained, and the second processed log data is obtained.

Optionally, the fourth obtaining unit 406 is specifically configured to: obtain the first processed log data corresponding to the key value from the temporary cache according to the key value of the second to-be-processed business log .

Optionally, as shown in FIG. 6 , the apparatus 40 further includes: an enhancement unit 408, configured to perform information enhancement processing on the first to-be-processed service log to obtain the first enhanced to-be-processed service log; the second The obtaining unit 402 is specifically configured to: obtain the first processed log data according to the first enhanced service log to be processed and the log data.

Optionally, as shown in FIG. 7 , the apparatus 40 further includes: a preprocessing unit 409, configured to preprocess the first to-be-processed business log, and the preprocessing includes log information format normalization, null value, Outlier checking.

It should be noted that, first, those skilled in the art can clearly understand that, for the convenience and brevity of the description, the specific working process of the above-described devices and units may refer to the corresponding processes in the foregoing method embodiments, which are not described here. Repeat.

Second, the first obtaining unit 401 , the second obtaining unit 402 , the determining unit 403 , the third obtaining unit 405 , the fourth obtaining unit 406 , the fifth obtaining unit 407 , the enhancing unit 408 and the preprocessing unit 409 can all be located in A central processing unit (Central Processing Unit, CPU), a microprocessor (Micro Processor Unit, MPU), a digital signal processor (Digital Signal Processor, DSP), or a field programmable gate array (Field Programmable Gate Array) in the log processing device 40 , FPGA) etc. The storage unit 404 may be implemented by a memory located in the log processing device 40 .

An embodiment of the present invention provides a log processing apparatus, including: a first obtaining unit, configured to obtain log data corresponding to a first service log to be processed from a data cache. A second obtaining unit, configured to obtain first processed log data according to the first to-be-processed service log and the log data. A determination unit, configured to determine whether there are other processing nodes currently accessing log data corresponding to the first service log to be processed in the data cache. A storage unit, configured to store the first processed log data in the data cache if no other processing node currently accesses the log data corresponding to the first to-be-processed business log in the data cache; if other processing nodes currently exist The log data corresponding to the first to-be-processed business log in the data cache is accessed, and the first processed log data is stored in the temporary cache. Compared with the prior art, since the first post-processing log data can be stored in a data cache or a temporary cache according to different situations, information sharing between different processing nodes is realized, so that the business logs generated by the same business can be stored in the data cache. Different processing nodes perform processing, so that the same number of business logs can be allocated to each processing node, so that the load of each processing node is balanced and the system stability is improved.

As will be appreciated by one skilled in the art, embodiments of the present invention may be provided as a method, system, or computer program product. Accordingly, the invention may take the form of a hardware embodiment, a software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product embodied on one or more computer-usable storage media having computer-usable program code embodied therein, including but not limited to disk storage, optical storage, and the like.

The present invention is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each flow and/or block in the flowchart illustrations and/or block diagrams, and combinations of flows and/or blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to the processor of a general purpose computer, special purpose computer, embedded processor or other programmable data processing device to produce a machine such that the instructions executed by the processor of the computer or other programmable data processing device produce Means for implementing the functions specified in a flow or flow of a flowchart and/or a block or blocks of a block diagram.

These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory result in an article of manufacture comprising instruction means, the instructions The apparatus implements the functions specified in the flow or flows of the flowcharts and/or the block or blocks of the block diagrams.

These computer program instructions can also be loaded on a computer or other programmable data processing device to cause a series of operational steps to be performed on the computer or other programmable device to produce a computer-implemented process such that The instructions provide steps for implementing the functions specified in the flow or blocks of the flowcharts and/or the block or blocks of the block diagrams.

The above descriptions are only preferred embodiments of the present invention, and are not intended to limit the protection scope of the present invention.

Claims (10)

1. A log processing method is applied to a processing node and comprises the following steps:

acquiring log data corresponding to a first service log to be processed from a data cache;

acquiring first processed log data according to the first service log to be processed and the log data;

determining whether other processing nodes access log data corresponding to the first service log to be processed in the data cache or not;

if no other processing node accesses the log data corresponding to the first service log to be processed in the data cache, storing the first processed log data in the data cache; and if other processing nodes access the log data corresponding to the first service log to be processed in the data cache, storing the first processed log data in a temporary cache.

2. The method of claim 1, wherein after storing the first processed log data in a temporary cache, the method further comprises:

acquiring log data corresponding to a second service log to be processed from the data cache;

obtaining the first processed log data from the temporary cache;

and acquiring second processed log data according to the second service log to be processed, the log data corresponding to the second service log to be processed and the first processed log data.

3. The method of claim 2, wherein the retrieving the first processed log data from the temporary cache comprises:

and acquiring the first processed log data corresponding to the key value from the temporary cache according to the key value of the second service log to be processed.

4. The method according to claim 1 or 2, wherein before obtaining first processed log data according to the first service log to be processed and the log data, the method further comprises:

performing information enhancement processing on the first service log to be processed to obtain a first enhanced service log to be processed;

the obtaining of the first processed log data according to the first to-be-processed service log and the log data includes:

and acquiring first processed log data according to the first enhanced service log to be processed and the log data.

5. The method according to claim 4, wherein before the performing information enhancement processing on the first service log to be processed and obtaining the first enhanced service log to be processed, the method further comprises:

and preprocessing the first service log to be processed, wherein the preprocessing comprises log information format standardization, null value and abnormal value detection.

6. A log processing apparatus, comprising:

the first acquisition unit is used for acquiring log data corresponding to a first service log to be processed from the data cache;

a second obtaining unit, configured to obtain first processed log data according to the first to-be-processed service log and the log data;

a determining unit, configured to determine whether there are other processing nodes accessing log data corresponding to the first to-be-processed service log in the data cache currently;

the storage unit is used for storing the first processed log data in the data cache if no other processing node accesses the log data corresponding to the first to-be-processed service log in the data cache currently; and if other processing nodes access the log data corresponding to the first service log to be processed in the data cache, storing the first processed log data in a temporary cache.

7. The apparatus of claim 6, further comprising:

a third obtaining unit, configured to obtain log data corresponding to a second service log to be processed from the data cache;

a fourth obtaining unit, configured to obtain the first processed log data from the temporary cache;

and a fifth obtaining unit, configured to obtain second processed log data according to the second service log to be processed, the log data corresponding to the second service log to be processed, and the first processed log data.

8. The apparatus according to claim 7, wherein the fourth obtaining unit is specifically configured to:

and acquiring the first processed log data corresponding to the key value from the temporary cache according to the key value of the second service log to be processed.

9. The apparatus of claim 6 or 7, further comprising:

the enhancement unit is used for performing information enhancement processing on the first service log to be processed to obtain a first enhanced service log to be processed;

the second obtaining unit is specifically configured to:

and acquiring first processed log data according to the first enhanced service log to be processed and the log data.

10. The apparatus of claim 9, further comprising:

and the preprocessing unit is used for preprocessing the first service log to be processed, and the preprocessing comprises log information format normalization, null value and abnormal value detection.

Images