CN102346710B - Dynamic stubbing technology based time-delay analysis method for data packet processing - Google Patents

Abstract

The invention discloses a dynamic stubbing technology based time-delay analysis method for data packet processing. The method comprises the following five main steps of: 1, collecting a kernel symbol table and acquiring a function list capable of being stubbed; 2, determining a function list to be stubbed; 3, configuring a stubbing module, inserting a section of codes into each function to be stubbed, and adding function addresses and time stamps for function calling to a current data packet under processing; 4, starting a packet sending and receiving module; and 5, counting and analyzing the data packets stored in the step 4 to obtain the time consumed by each function of a kernel. The method disclosed by the invention has the advantages of less expense and very strong instantaneity and adaptability, can be used for elaborately analyzing the expense brought by each function and time delay for the data packets to wait in a queue in a process of treating the data packets by the Linux kernel, can provide a powerful basis for network analysis and optimization, and has a better practical value and a wide application prospect in the technical field of computers.

Description

A packet processing delay analysis method based on dynamic instrumentation technology

(1) Technical field

The present invention relates to a data packet processing delay analysis method based on dynamic stub technology, in particular to a method for analyzing the processing delay of each function of the operating system kernel in the data packet processing process based on dynamic stub technology , belonging to the field of computer technology.

(2) Background technology

In recent years, with the continuous enrichment of network applications, the distribution of networks has become wider and wider, and the network traffic has also increased. New requirements and challenges have been put forward for the network performance of data centers. In order to cope with this challenge, it is necessary for the data center to adjust the network structure and improve the processing capacity of the server. However, in order to increase server utilization, simplify server management and maintenance, improve application availability, and reduce costs, many data centers have begun to adopt virtualization technology. Virtualization technology has many advantages, but it also introduces additional overhead, especially in I/O virtualization. In a virtualized environment, the processing of data packets is more complicated and the transmission path is longer, making the overall network performance poor.

In order to optimize network performance, it is necessary to know where the bottlenecks of the system are and which aspects of the system cost the most, and then make targeted improvements. At present, many people are studying how to improve network performance in a virtualized environment. Most of these studies start from the principle of virtualization, trying to reduce the overhead of virtualization, increase network bandwidth, and reduce network delay. However, these studies have not quantitatively analyzed the processing delay of the data packet, and currently there is no tool that can automatically analyze the delay of each function in the entire processing process from receiving to sending the data packet. Based on the dynamic stub technology, the present invention designs a method capable of automatically analyzing the time delay of each function of the operating system kernel in the process of sending and receiving data packets.

Instrumentation technology is widely used in software testing. The GCC compiler also provides corresponding functions for testing code coverage. Some operating systems also support instrumentation technology, such as DTrace in Solaris and Kprobe in Linux. The dynamic instrumentation technology is one step further. It can dynamically insert a piece of code into a certain position of the program when the program is running to collect information or change the behavior of the program without any modification to the original program. The present invention analyzes the operating system data packet processing time delay based on the dynamic stub insertion technology.

(3) Contents of the invention

1. Purpose

The purpose of the present invention is to provide a data packet processing delay analysis method based on dynamic stub technology, which can count the time consumed by each function of the operating system kernel in the data packet processing process and the time spent waiting in the queue , which provides a strong basis for further network optimization.

2. Technical solution

The technical scheme of the present invention is as follows: a data packet processing delay analysis method based on dynamic stub technology, its overall framework is shown in Figure 1, the machine in Figure 1 can be a real computer, or can be through virtualization The virtual computer created by software virtualization, machine 1 and machine 2 in Figure 1 are mainly divided from the perspective of function, and they can be deployed on the same physical machine at the same time. The main function of machine 1 is to provide a control interface, analyze and output the results. The contract sending module and statistics module are deployed on machine 1. The main function of machine 2 is to collect information and feed back the collected information to machine 1. Deploy on machine 2 Added the instrumentation module and packet receiving module. The specific process is as shown in Figure 2, a method for analyzing the time delay of data packet processing based on dynamic pile insertion technology of the present invention, the method comprises the following steps:

Step 201. Collect the kernel symbol table on machine 2, obtain the corresponding relationship between kernel functions and addresses, output a list of functions that can be instrumented, and pass it to machine 1.

Step 202 . Determine the functions that need to be analyzed on machine 1 according to the optimization goal, output a list of functions that need to be stubbed, and pass it to machine 2 .

Step 203. Transfer the function list determined in step 202 to machine 2. Machine 2 uses dynamic instrumentation technology to insert a piece of code at the head of the function to be monitored. The code will write the address of the function and the current timestamp into data pack.

Step 204. Start the packet receiving module and the packet sending module. Machine 1 starts to send data packets, and after the data packets are delivered to the kernel of Machine 2, if it is detected that the destination of the data packets is the receiving module of Machine 2, the address and timestamp of the function currently being executed are written into the data packets . When the data packet arrives at the packet receiving module of machine 2, the packet receiving module returns the data packet to machine 1, and the packet sending module of machine 1 stores the data packet from machine 2 in the form of a file.

Step 205. Machine 1 counts all data packets from the receiving module of Machine 2, calculates and outputs the statistical results.

Wherein, step 201 collects the kernel symbol table on the machine 2 is to collect the symbol table of the running kernel, the kernel function and the address of the variable in the memory are recorded in the kernel symbol table, under the Linux system, by reading the /boot directory The kernel symbol table can be obtained by downloading the System.map corresponding to the kernel version or directly reading /proc/ksyms. In addition, the kernel symbol table can also be obtained by analyzing the kernel file or kernel debugging information. Since the kernel symbol table not only records the address of the kernel function in memory, but also records the address of the kernel variable, it is necessary to analyze the read kernel symbol table, filter out those non-function kernel symbols, and output all kernel symbols. The symbol table of functions, these functions are the list of functions that can be inserted, and after they are determined, they are passed to machine 1.

Wherein, step 202 determines the functions to be analyzed on the machine 1 according to the optimization target, and outputs a list of functions to be stubbed. After machine 1 receives the list of functions that can be instrumented from machine 2, it can manually or automatically determine the list of functions that need to be instrumented. Manual determination requires familiarity with the kernel processing process, knowing which functions the kernel will call to process data packets, and then manually writing the functions that need to be inserted into a file to form a list of functions that need to be inserted. In addition to manual determination, you can also write some scripts to automatically determine through name matching. Of course, manual determination can reduce the number of stubs and reduce the impact of stubs on network performance.

Among them, step 203 first transfers the function list determined in step 202 to machine 2, and then uses dynamic instrumentation technology on machine 2 to insert a small piece of code at the head of all functions that need to be monitored. After the insertion is completed, the kernel executes This function will first execute the inserted code, and then execute the original function. The function of the inserted code is to detect the data packet currently being processed, and if certain conditions are met, the address of the currently executing function and the current timestamp are added to the back of the data packet.

Wherein, step 204 activates the packet sending module of machine 1 and the packet receiving module of machine 2 after the posting module is deployed. Both the packet sending module of machine 1 and the packet receiving module of machine 2 are applications running in user mode. After the packet sending module of machine 1 is activated, it will send a data packet to the packet receiving module of machine 2 at a certain rate according to the user's configuration. The structure of the data packet is shown in Figure 3. The data packet contains two parts of information, and the meta information part records The total size of the data packet, the offset of the data information, and the offset of the information that needs to be inserted at present; the data information is relatively simple, which is a table of function addresses and timestamps. The timestamp is written at the end of the data message. The data information part of the data packet sent by the packet sending module of machine 1 is filled with zeros. After the data packet reaches machine 2, the stub module located in the kernel will write the address of the function to be monitored and the timestamp when the kernel calls the function into The data information part of the data packet, and modify the meta information of the data packet. After the data packet is delivered to the packet receiving module, the packet receiving module will not modify the data packet, but directly returns the data packet to the packet sending module of machine 1.

Among them, after the packet sending module of machine 1 receives the data packet returned by machine 2 in step 205, it first stores the data packet in the form of a file, and then combines the kernel symbol table collected in step 201 to analyze the data information of the data packet , you can calculate which functions are called by the kernel during packet processing, and the time consumed by each function. If you are familiar with the packet processing process of the kernel, you can also analyze the queuing delay of the packet during processing by viewing the time stamps of entering and exiting the queue.

3. Advantages and effects

The present invention is based on a data packet processing delay analysis method based on dynamic stub technology. Compared with the prior art, its main advantages are: (1) the present invention can track the processing process of the data packet in the whole process, and analyze it in detail. The time spent by each function during packet reception and transmission, as well as the time spent queuing in the queue. (2) Since the sending rate of the data packet is controllable by the user, the overhead brought by the instrumentation module can be effectively controlled by adjusting the sending rate of the data packet, so the total system overhead is very small. (3) The present invention has good real-time performance and adaptability. The real-time performance means that the result of data packet time delay analysis reflects the situation of data packet processing time delay under the current environment. However, as the network load changes, the delay of the data packet will also change greatly. Because the system overhead brought by the present invention is small, it can be used even under heavy network load, so it has a strong adaptability.

(4) Description of drawings

Fig. 1 is a schematic diagram of the overall framework of the packet processing delay analysis method of the present invention

Fig. 2 is a schematic flow chart of the packet processing delay analysis method of the present invention

Fig. 3 is a schematic structural diagram of a data packet used in the data packet processing delay analysis method of the present invention

The symbols in the figure are explained as follows:

201-205 Step number.

(5) Specific implementation methods

In order to make the object, technical solution and advantages of the present invention more clearly, the present invention will be further described in detail below in conjunction with the accompanying drawings and specific embodiments.

The main idea of the present invention is to use dynamic stub technology to collect and record the address of the function related to data packet processing in the kernel and the timestamp information of being called, and then obtain the information of each function in the kernel through the statistics and analysis of these information. The time spent in the process of sending and receiving and the time spent queuing in the queue provide detailed analysis data for network optimization.

The present invention is provided with two machines, namely machine 1 and machine 2, both machines are real computers, the specific model is Lenovo Wanquan R510 server, as shown in Figure 1, machine 1 and machine 2 are mainly divided from the perspective of function Yes, they can be deployed on the same physical machine at the same time. The main function of machine 1 is to provide a control interface, analyze and output the results. The contract sending module and statistics module are deployed on machine 1. The main function of machine 2 is to collect information and feed back the collected information to machine 1. Deploy on machine 2 Added the instrumentation module and packet receiving module.

The present invention requires each machine to adopt the Linux operating system in terms of software, and the kernel of the Linux operating system on the machine 2 should be version 2.6.18 or later, so as to support the kprobe technology. The specific implementation is roughly divided into three stages. The first stage is initialization. The main work is to obtain the kernel symbol table and determine the functions that need to be inserted; the second stage is to deploy the instrumentation module on machine 2. Due to the collection And record the function address and time stamp; the third stage is to start sending and receiving data packets, and count and analyze the results to get the results.

An example is used to illustrate below, as shown in Figure 2, a kind of data packet processing delay analysis method based on dynamic stub technology of the present invention, the concrete steps of this method are as follows:

Step 201: Collect the kernel symbol table of machine 2. The collection method is to first check whether /proc/ksyms exists, and if it exists, copy its content to a file, and then pass it to machine 1. If it does not exist, check whether there is a System.map file matching the running kernel in the /boot directory, and if so, pass it to machine 1. If not, you need to analyze the kernel file or kernel debugging information to obtain it. If it is still If there is no way to obtain it, the collection of kernel symbol information fails. After obtaining the kernel symbol table, it is necessary to filter out those non-function kernel symbol information, so as to obtain a list of functions that can be inserted.

Step 202: If the kernel symbol information of the machine 2 is successfully collected in step 201, a list of functions that can be instrumented can be obtained, and then the list of functions that need to be instrumented can be determined manually or automatically.

Step 203: After determining the functions that need to be inserted, pass those functions that need to be inserted to machine 2. After machine 2 obtains the list of functions that need to be inserted, the next job is to add those functions that need to be inserted. Add a piece of code to record the address of the function and the timestamp when the function was called. This requires the help of the kprobe mechanism provided by Linux. The kprobe mechanism can insert a piece of code anywhere in the kernel to collect kernel-related information. For the convenience of deployment, SystemTap can be used. SystemTap is a scripting language based on kprobe. It can translate the script into a C file, compile it into a kernel module, and dynamically add it to the kernel. When using it, you only need to provide it with the function name and the code segment that needs to be inserted. Therefore, after obtaining the list of functions that need to be inserted, the work of instrumentation can be easily completed through SystemTap.

Step 204: Start the packet receiving module and the packet sending module. After completing the posting work, you can start the packet receiving module of machine 2, which will wait for the data packet from machine 1 on a certain port. After starting the packet receiving module, start the packet sending module on machine 1, machine 1 will send data packets to machine 2 at a certain rate, and at the same time receive the data packets returned by machine 2, and store the data packets in the form of files for later Statistics and analysis can continue.

Step 205: In step 204, the data packet received from the machine 2 is stored in a file. This step is mainly to read the file, count and analyze the time consumed by each function, and output the result.

Fig. 3 is a schematic structural diagram of a data packet used in the data packet processing delay analysis method of the present invention

Finally, it should be noted that the above embodiments are only used to illustrate and not limit the technical solutions of the present invention, although the present invention has been described in detail with reference to the above embodiments, those of ordinary skill in the art should understand that: the present invention can still be modified Or an equivalent replacement, any modification or partial replacement without departing from the spirit and scope of the present invention shall fall within the scope of the claims of the present invention.

Claims (1)

1. based on a processing data packets Delay Analysis for dynamic pitching pile technology, it is characterized in that: the method concrete steps are as follows:

Step one. on machine 2, collect Kernel Symbol Table, obtain the corresponding relation of kernel function and address, export the function list that can carry out pitching pile, and pass to machine 1;

Step 2. according to the target optimized on machine 1, determine the function of Water demand, export the function list needing pitching pile, and pass to machine 2;

Step 3. be delivered on machine 2 by determined for step 2 function list, machine 2 uses dynamic pitching pile technology to insert one section of code at the head of the function needing monitoring, and this code can by the address of function and current timestamp write packet;

Step 4. start packet receiving module and module of giving out a contract for a project; Machine 1 starts to send packet, data packet delivery to machine 2 kernel after, if detect, the destination of packet is the packet receiving module of machine 2, just by the address of the current function performed and timestamp write packet; When packet arrives the packet receiving module of machine 2, packet is returned to machine 1 by packet receiving module, and the packet from machine 2 stores by the module of giving out a contract for a project of machine 1 in the form of a file;

Step 5. all packets from machine 2 receiver module added up by machine 1, calculate and export statistics;

Wherein, described machine 1 and machine 2 are real computing machines, or by the virtual virtual machine out of virtualization software, they are deployed in same physical machine simultaneously; The function of machine 1 is to provide control inerface, analyzes and Output rusults, machine 1 deploy give out a contract for a project module and statistical module, and the function of machine 2 is collection information, and by the information feed back collected to machine 1, machine 2 deploy pitching pile module and packet receiving module;

Wherein, " on machine 2, collecting Kernel Symbol Table " described in step one refers to: the symbol table collecting the kernel run, kernel function and the address of variable in internal memory is have recorded in Kernel Symbol Table, under linux system, by System.map corresponding with kernel version under reading/boot catalogue or directly read/proc/ksyms just gets Kernel Symbol Table, in addition, also by Kernel Symbol Table that analysis kernel file or kernel tailoring information obtain; Owing to not only have recorded the address in internal memory of kernel function in Kernel Symbol Table, also have recorded the address of kernel variable, therefore the Kernel Symbol Table to reading is needed to analyze, filter out the interior nuclear symbol of those non-functional, export the symbol table of all kernel function, these functions are exactly the function list that can carry out pitching pile, after determining, are passed to machine 1;

Wherein, described in step 2 " according to the target optimized on machine 1; determine the function of Water demand; export the function list needing pitching pile; and pass to machine 2 " refer to: after machine 1 receives the function list of energy pitching pile of machine 2, determine the function list needing pitching pile manually or automatically; Manually determine to need to be familiar with for kernel processes process, know which function kernel can call and carry out handle packet, the then manual function writing in files of the pitching pile formation that will need needs the function list of pitching pile; Except manually determining, also can write a little step and automatically be determined by the method for name matching;

Wherein, " being delivered on machine 2 by determined for step 2 function list; machine 2 uses dynamic pitching pile technology to insert one section of code needing the head of function of monitoring; this code can by the address of function and current timestamp write packet " described in step 3 refers to: need the head of the function of monitoring to insert after one section of code completes, first kernel can perform the code of insertion when performing this function, then perform original function; The function of the code inserted detects the current packet processed, if satisfy condition, adds after packet by the address of the current function performed and current timestamp;

Wherein, " start packet receiving module and give out a contract for a project module " described in step 4 refers to: the packet receiving module of give out a contract for a project module and the machine 2 of machine 1 is all operate in the application program under User space; Packet can be sent according to certain speed that is configured to of user to the packet receiving module of machine 2 after the module of giving out a contract for a project of machine 1 is activated, packet comprises two parts information, metamessage part have recorded packet size altogether, data message side-play amount and need the side-play amount of insertion information at present; Data message is exactly the form of a function address and timestamp, and function address and timestamp are all written to the end of data message by pitching pile module at every turn; The data information portion zero padding of the packet of the module transmission of giving out a contract for a project of machine 1, after packet arrives machine 2, timestamp when needing address and this function of kernel calls of the function of monitoring can be write the data information portion of packet by the pitching pile module being arranged in kernel, and the metamessage of Update Table bag; After data packet delivery to packet receiving module, packet receiving module can not Update Table bag, but directly packet is returned to the module of giving out a contract for a project of machine 1;

Wherein, described in step 5 " all packets from machine 2 receiver module added up by machine 1, calculate and export statistics " refer to: after the module of giving out a contract for a project of machine 1 receives the packet that machine 2 returns, first packet is stored in the form of a file, the Kernel Symbol Table collected of integrating step two again after end, by analyzing the data message of packet, calculate kernel and have invoked which function in processing data packets process, and the time that each function consumes, if to the words that the processing data packets process of kernel is familiar with, by checking the timestamp into queue and dequeue, the queuing delay of packet in processing procedure can also be analyzed.