CN108989471A - The management method and device of log in network system - Google Patents

Abstract

The invention discloses a log management method and device in a network system. The method includes: after receiving the log download request sent by the client, obtaining the log data corresponding to the download request; determining the size of the data volume of the log data; comparing the data volume of the log data with the preset The data volume threshold is compared to obtain a comparison result; if the comparison result is greater than the data volume threshold, the log data is sent in batches.

Description

Log management method and device in network system

technical field

The invention relates to the field of information processing, in particular to a log management method and device in a network system.

Background technique

Network devices, systems, and service programs, etc., will generate an event record called log during operation; each line of log records the description of related operations such as date, time, user, and action.

Windows network operating systems are designed with various log files, such as application logs, security logs, system logs, Scheduler service logs, FTP logs, WWW logs, DNS server logs, etc., which are based on the services opened by your system vary with each other. When we perform some operations on the system, these log files usually record some relevant content of the operation, which is quite useful for system security staff. For example, if someone detects the system by IPC, the system will quickly record the IP, time, user name, etc. used by the detector in the security log. After using FTP to detect, it will record the IP, time, username used for the probe, etc.

Of course, how valuable log data is to achieving network security depends on two factors: First, your systems and devices must be properly set up to log the data you need. Second, you must have the right tools, training, and resources available to analyze the data collected.

At present, many websites output the log to an excel at one time when downloading the log. When the log content is not very large, there will be no problem. However, with the use of the website, the logs are getting more and more, and the amount of data is getting bigger and bigger. At this time, if a large amount of data is output to an excel at one time, problems such as stuck, request timeout, and memory overflow will easily occur. .

Contents of the invention

In order to solve the above technical problems, the present invention provides a log management method and device in a network system.

In order to achieve the purpose of the present invention, the present invention provides a kind of log management method in the network system, comprising:

After receiving the log download request sent by the client, obtaining the log data corresponding to the download request;

Determine the size of the data volume of the log data;

Comparing the data volume of the log data with a preset data volume threshold to obtain a comparison result;

If the comparison result is greater than the data volume threshold, the log data is sent in batches.

Wherein, the method also has the following characteristics: the determining the size of the data volume of the log data includes:

Obtain the storage format requested in the download request;

According to the requested storage format, generate a temporary file for storing the log data;

Determine the data size of the temporary file.

Wherein, the method also has the following characteristics: before sending the log data in batches, the method also includes:

sending prompt information for downloading the data in batches;

receiving a confirmation operation on the prompt message;

Wherein, the operation of sending the log data in batches is performed after the confirmation operation is received.

Wherein, the method also has the following characteristics: before sending the log data in batches, the method also includes:

Obtaining network bandwidth information used by the client to download the log data;

According to the size of the network bandwidth, determine the number of batches of the log data and the amount of data sent each time;

Wherein, the sending of the log data in batches includes:

The log data is sent out according to the determined number of batches and the amount of data sent in each batch.

Wherein, the method also has the following characteristics: before sending the log data in batches, the method also includes:

Acquiring the occupancy state of the network bandwidth used by the client to download the log data;

After the last batch of log data is sent, according to the state of the network bandwidth, control the sending time of each batch of log data;

Wherein, the sending of the log data in batches includes:

At the determined sending time, the corresponding batch of log data is sent out.

In order to achieve the purpose of the present invention, the present invention also provides a log management device in a network system, including:

The first obtaining module is used to obtain the log data corresponding to the download request after receiving the log download request sent by the client;

A first determining module, configured to determine the size of the data volume of the log data;

A comparison module, configured to compare the data volume of the log data with a preset data volume threshold to obtain a comparison result;

The first sending module is configured to send the log data in batches if the comparison result is greater than the data volume threshold.

Wherein, the device also has the following characteristics: the determination module includes:

an acquisition unit, configured to acquire the storage format requested in the download request;

A generating unit, configured to generate a temporary file for storing the log data according to the requested storage format;

A determining unit, configured to determine the data size of the temporary file.

Wherein, the device also has the following characteristics: the device also includes:

The second sending module is used to send prompt information for downloading the data in batches before sending the log data in batches;

A receiving module, configured to receive a confirmation operation on the prompt information;

Wherein, the first sending module executes the operation of sending the log data in batches after receiving the confirmation operation.

Wherein, the device also has the following characteristics: the device also includes:

The second obtaining module is used to obtain the network bandwidth information used by the client to download the log data before sending the log data in batches;

The second determination module is used to determine the number of batches of the log data and the amount of data sent each time according to the size of the network bandwidth;

Wherein, the first sending module sends the log data according to the determined number of batches and the amount of data sent in each batch.

Wherein, the device also has the following characteristics: the device also includes:

A third obtaining module, configured to obtain the occupancy state of the network bandwidth used by the client to download the log data before sending the log data in batches;

A control module, configured to control the sending time of each batch of log data according to the state of the network bandwidth after the last batch of log data is sent;

Wherein, the first sending module sends the corresponding batch of log data at the determined sending time.

In the embodiment provided by the present invention, after receiving the log download request sent by the client, the log data corresponding to the download request is obtained. Unlike the prior art that directly sends the log data to the client, the present invention first determines The size of the data volume of the log data, and compare the data volume of the log data with the preset data volume threshold, and determine whether to directly send the data to the client according to the comparison result, if the comparison result is greater than the specified If the above data volume threshold is exceeded, the log data will be sent in batches to reduce the probability of problems such as stuck, request timeout, and memory overflow, and improve the stability of data transmission.

Additional features and advantages of the invention will be set forth in the description which follows, and in part will be apparent from the description, or may be learned by practice of the invention. The objectives and other advantages of the invention may be realized and attained by the structure particularly pointed out in the written description and claims hereof as well as the appended drawings.

Description of drawings

The accompanying drawings are used to provide a further understanding of the technical solution of the present invention, and constitute a part of the description, and are used together with the embodiments of the application to explain the technical solution of the present invention, and do not constitute a limitation to the technical solution of the present invention.

Fig. 1 is the flowchart of the management method of log in the network system provided by the present invention;

FIG. 2 is a structural diagram of a log management device in a network system provided by the present invention.

Detailed ways

In order to make the purpose, technical solution and advantages of the present invention more clear, the embodiments of the present invention will be described in detail below in conjunction with the accompanying drawings. It should be noted that, in the case of no conflict, the embodiments in the present application and the features in the embodiments can be combined arbitrarily with each other.

The steps shown in the flowcharts of the figures may be performed in a computer system, such as a set of computer-executable instructions. Also, although a logical order is shown in the flowcharts, in some cases the steps shown or described may be performed in an order different from that shown or described herein.

FIG. 1 is a flowchart of a log management method in a network system provided by the present invention. The methods shown in Figure 1 include:

Step 101. After receiving the log download request sent by the client, obtain the log data corresponding to the download request;

Step 102, determining the size of the data volume of the log data;

Step 103, comparing the data volume of the log data with a preset data volume threshold to obtain a comparison result;

Step 104, if the comparison result is greater than the data volume threshold, send the log data in batches.

In the embodiment of the method provided by the present invention, after receiving the log download request sent by the client, the log data corresponding to the download request is obtained. Unlike the prior art that directly sends the log data to the client, the present invention first Determine the size of the data volume of the log data, and compare the data volume of the log data with a preset data volume threshold, determine whether to directly send the data to the client according to the comparison result, and if the comparison result is greater than If the data volume threshold is set, the log data will be sent in batches to reduce the probability of problems such as stuck, request timeout, and memory overflow, and improve the stability of data transmission.

The method embodiment provided by the present invention is described further below:

The present invention is mainly to solve problems such as request timeout, stuck, memory overflow, etc., which are prone to occur when downloading logs in large quantities.

The basic idea of the solution of the present invention is specifically as follows:

1) When downloading large batches of logs, download them in batches;

2) The server returns data in batches according to the batch download request.

In a method embodiment provided by the present invention, the determining the size of the data volume of the log data includes:

Obtain the storage format requested in the download request;

According to the requested storage format, generate a temporary file for storing the log data;

Determine the data size of the temporary file.

Specifically, when the user configures the log data to be downloaded, the user can be prompted to set the save format. After the user completes the setting, the download request operation is triggered, and the save format of the log data configured by the user in the download request can be read; Before sending the log data, first calculate the data volume when the log data is sent in the corresponding storage format according to the requested download format, such as excel, txt, etc., and finally determine the data volume to be sent to the user instead of the The amount of data in the log itself, in order to more accurately obtain the amount of data that the user needs to receive.

Through the above method, the determined data volume will more accurately determine the data volume of the log data, and can more accurately determine the size between the data volume and the data volume threshold, thereby ensuring the accuracy of the operation.

Wherein, before sending the log data in batches, the method further includes:

sending prompt information for downloading the data in batches;

receiving a confirmation operation on the prompt message;

Wherein, the operation of sending the log data in batches is performed after the confirmation operation is received.

Specifically, after determining the need for batch operations on the server side, it is necessary to consult the processing results on the client side to ensure that network services can be better completed, and at the same time to remind users that they need to obtain log data in batches. , to ensure that users can wait to receive all log data.

Of course, if the user does not perform the confirmation operation, the data will be transmitted according to the default transmission method, such as sending the log data directly.

Wherein, when sending the log data in batches, in order to fully match the network resources of the server side and the client side, before sending the log data in batches, the method further includes:

Obtaining network bandwidth information used by the client to download the log data;

According to the size of the network bandwidth, determine the number of batches of the log data and the amount of data sent each time;

Wherein, the sending of the log data in batches includes:

The log data is sent out according to the determined number of batches and the amount of data sent in each batch.

Specifically, obtain the bandwidth size of the client, determine the amount of data that can be transmitted through the bandwidth resources of the network bandwidth each time according to the bandwidth size, and then determine the total amount of log data that needs to be divided into according to the amount of data transmitted each time The total number of batches.

Preferably, since the network bandwidth of the server side and the client side often needs to transmit other data, the current network bandwidth occupancy should also be considered when performing batch transmission of log data. If the network bandwidth is idle, it can be timely Send the data volume of this batch, and if the network bandwidth is in a busy state, then control the time to postpone sending the data volume of this batch, and wait for the network bandwidth to be in an idle state before sending. The specific implementation is as follows:

Wherein, before sending the log data in batches, the method further includes:

Acquiring the occupancy state of the network bandwidth used by the client to download the log data;

After the last batch of log data is sent, according to the state of the network bandwidth, control the sending time of each batch of log data;

Wherein, the sending of the log data in batches includes:

At the determined sending time, the corresponding batch of log data is sent out.

It can be seen from the above that the operation of batch transmission combined with the state of the network bandwidth can reduce the probability of problems such as request timeout, stuck, memory overflow, etc. while taking into account the normal transmission of the network, and effectively improve the transmission stability.

The method mentioned in the present invention will be further described below with the application examples provided by the present invention:

A way to download logs in batches when downloading large batches of logs. The specific implementation process is as follows:

1) Before downloading the log, obtain the data volume of the log;

2) According to the amount of log data obtained, it is judged whether it is necessary to request downloading in batches;

3) If the amount of data is large, multiple requests are made based on the amount of data;

Specifically, for example, the log data has a total of 1 million pieces of data, and the determined batch download strategy is to download in 10 requests, each with 100,000 pieces of data; each request has the parameters required for batching: batch Number page and batch number pageSize;

For example, the parameter pageSize of the first request: 100,000, page: 1; the parameter of the second request pageSize: 100,000, page: 2; and so on;

4) The server obtains the request and parameters, and returns the log.

The method provided by the application example of the invention solves the problems of request overtime, stuck, memory overflow and the like when downloading a large number of logs, enhances the stability of the product and improves the competitiveness of the product.

FIG. 2 is a structural diagram of a log management device in a network system provided by the present invention. The device shown in Figure 2 includes:

The first acquiring module 201 is configured to acquire log data corresponding to the download request after receiving the log download request sent by the client;

The first determination module 202 is configured to determine the size of the data volume of the log data;

A comparison module 203, configured to compare the data volume of the log data with a preset data volume threshold to obtain a comparison result;

The first sending module 204 is configured to send the log data in batches if the comparison result is greater than the data volume threshold.

In an apparatus embodiment provided by the present invention, the determination module 202 includes:

an acquisition unit, configured to acquire the storage format requested in the download request;

A generating unit, configured to generate a temporary file for storing the log data according to the requested storage format;

A determining unit, configured to determine the data size of the temporary file.

In an embodiment of the device provided by the present invention, the device further includes:

The second sending module is used to send prompt information for downloading the data in batches before sending the log data in batches;

A receiving module, configured to receive a confirmation operation on the prompt information;

Wherein, the first sending module 204 executes the operation of sending the log data in batches after receiving the confirmation operation.

In an embodiment of the device provided by the present invention, the device further includes:

The second obtaining module is used to obtain the network bandwidth information used by the client to download the log data before sending the log data in batches;

The second determination module is used to determine the number of batches of the log data and the amount of data sent each time according to the size of the network bandwidth;

Wherein, the first sending module 204 sends the log data according to the determined number of batches and the amount of data sent in each batch.

In an embodiment of the device provided by the present invention, the device further includes:

A third obtaining module, configured to obtain the occupancy state of the network bandwidth used by the client to download the log data before sending the log data in batches;

A control module, configured to control the sending time of each batch of log data according to the state of the network bandwidth after the last batch of log data is sent;

Wherein, the first sending module 204 sends out the corresponding batch of log data at the determined sending time.

In the device embodiment provided by the present invention, after receiving the log download request sent by the client, the log data corresponding to the download request is obtained. Unlike the prior art that directly sends the log data to the client, the present invention first Determine the size of the data volume of the log data, and compare the data volume of the log data with a preset data volume threshold, determine whether to directly send the data to the client according to the comparison result, and if the comparison result is greater than If the data volume threshold is set, the log data will be sent in batches to reduce the probability of problems such as stuck, request timeout, and memory overflow, and improve the stability of data transmission.

Those of ordinary skill in the art can understand that all or part of the steps of the above-mentioned embodiments can be implemented using a computer program flow, the computer program can be stored in a computer-readable storage medium, and the computer program can be run on a corresponding hardware platform (such as system, device, device, device, etc.), and when executed, includes one or a combination of the steps of the method embodiment.

Optionally, all or part of the steps in the above embodiments can also be implemented using integrated circuits, and these steps can be fabricated into individual integrated circuit modules, or multiple modules or steps among them can be fabricated into a single integrated circuit module accomplish. As such, the present invention is not limited to any specific combination of hardware and software.

The devices/functional modules/functional units in the above embodiments can be realized by general-purpose computing devices, and they can be concentrated on a single computing device, or distributed on a network composed of multiple computing devices.

When each device/functional module/functional unit in the above-mentioned embodiments is realized in the form of a software function module and sold or used as an independent product, it can be stored in a computer-readable storage medium. The computer-readable storage medium mentioned above may be a read-only memory, a magnetic disk or an optical disk, and the like.

The above is only a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Anyone skilled in the art can easily think of changes or substitutions within the technical scope disclosed in the present invention. Should be covered within the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope described in the claims.

Claims (10)

1. A method for managing logs in a network system, comprising: After receiving the log download request sent by the client, obtaining the log data corresponding to the download request; Determine the size of the data volume of the log data; Comparing the data volume of the log data with a preset data volume threshold to obtain a comparison result; If the comparison result is greater than the data volume threshold, the log data is sent in batches. 2. The method according to claim 1, wherein the determining the size of the data volume of the log data comprises: Obtain the storage format requested in the download request; According to the requested storage format, generate a temporary file for storing the log data; Determine the data size of the temporary file. 3. The method according to claim 1, wherein, before sending the log data in batches, the method further comprises: sending prompt information for downloading the data in batches; receiving a confirmation operation on the prompt message; Wherein, the operation of sending the log data in batches is performed after the confirmation operation is received. 4. The method according to claim 1 or 3, wherein, before sending the log data in batches, the method further comprises: Obtaining network bandwidth information used by the client to download the log data; According to the size of the network bandwidth, determine the number of batches of the log data and the amount of data sent each time; Wherein, the sending of the log data in batches includes: The log data is sent out according to the determined number of batches and the amount of data sent in each batch. 5. The method according to claim 4, wherein, before sending the log data in batches, the method further comprises: Acquiring the occupancy state of the network bandwidth used by the client to download the log data; After the last batch of log data is sent, according to the state of the network bandwidth, control the sending time of each batch of log data; Wherein, the sending of the log data in batches includes: At the determined sending time, the corresponding batch of log data is sent out. 6. A log management device in a network system, characterized in that it comprises: The first obtaining module is used to obtain the log data corresponding to the download request after receiving the log download request sent by the client; A first determining module, configured to determine the size of the data volume of the log data; A comparison module, configured to compare the data volume of the log data with a preset data volume threshold to obtain a comparison result; The first sending module is configured to send the log data in batches if the comparison result is greater than the data volume threshold. 7. The device according to claim 6, wherein the determining module comprises: an acquisition unit, configured to acquire the storage format requested in the download request; A generating unit, configured to generate a temporary file for storing the log data according to the requested storage format; A determining unit, configured to determine the data size of the temporary file. 8. The device according to claim 6, further comprising: The second sending module is used to send prompt information for downloading the data in batches before sending the log data in batches; A receiving module, configured to receive a confirmation operation on the prompt information; Wherein, the first sending module executes the operation of sending the log data in batches after receiving the confirmation operation. 9. The device according to claim 6 or 8, characterized in that the device further comprises: The second obtaining module is used to obtain the network bandwidth information used by the client to download the log data before sending the log data in batches; The second determination module is used to determine the number of batches of the log data and the amount of data sent each time according to the size of the network bandwidth; Wherein, the first sending module sends the log data according to the determined number of batches and the amount of data sent in each batch. 10. The device according to claim 9, further comprising: A third obtaining module, configured to obtain the occupancy state of the network bandwidth used by the client to download the log data before sending the log data in batches; A control module, configured to control the sending time of each batch of log data according to the state of the network bandwidth after the last batch of log data is sent; Wherein, the first sending module sends the corresponding batch of log data at the determined sending time.