CN114679298A - A data screening method and device for application identification intelligence database - Google Patents

Abstract

The invention discloses a data screening method and a data screening device for an application identification information library, relates to the technical field of network security, and mainly aims to realize refining and compression of network flow data under the condition of completely retaining field information of effective identifiable applications, so that invalid content redundancy in the information library is reduced, and the identification efficiency of the information library is improved. The main technical scheme of the invention is as follows: analyzing the collected target flow into message data in a preset format; processing data which cannot identify a flow source in the message data according to a preset rule to obtain identifiable data; classifying the identifiable data by using a preset label, and performing clustering calculation on the identifiable data of the same classification to obtain at least one data set; and respectively extracting at least one piece of identifiable data from the data sets corresponding to all the classifications, and adding the data into an application identification intelligence library. The invention is used for screening data of the application identification information base.

Description

A data screening method and device for application identification intelligence database

technical field

The invention relates to the technical field of network security, in particular to a data screening method and device for an application identification intelligence base.

Background technique

The identification of applications at the entrance of the network is very important. Whether it is a network security product or a professional traffic analysis engine, the accurate identification of application traffic can not only gain insight into the operation of the entire network, but also make user behavior based on specific needs. To a certain extent, it can not only ensure the efficient operation of business flows, but also prevent network disconnection events caused by intranet poisoning. Therefore, it is necessary to establish an application identification intelligence database to realize the rapid identification of network traffic related to the application. When identifying application-related network traffic, it affects the efficiency of identification.

At present, when the application identification intelligence database is established, the processing method for network traffic data is generally to reduce the redundancy of field information content by extracting part of the field information and deduplication after parsing the network traffic. The location and name of field information in network traffic can be described as various, and only extracting some fields will lose other valid application-identifiable field information, which may easily lead to missed identification. Therefore, how to realize the refinement and compression of network traffic data while ensuring that the field information that can effectively identify the application is completely preserved when the intelligence database is established has become a problem to be solved at present.

SUMMARY OF THE INVENTION

In view of the above-mentioned problems, the present invention provides a data screening method, system and electronic device for an application identification information base, the main purpose of which is to realize the refining and compression of network traffic data under the condition of completely retaining the field information that can effectively identify the application, Thereby reducing the redundancy of invalid content in the intelligence base to improve the identification efficiency of the intelligence base.

In order to solve the above-mentioned technical problems, the present invention proposes the following scheme:

In a first aspect, the present invention provides a data screening method for an application identification intelligence base, the method comprising:

Parse the collected target traffic into packet data in a preset format;

Process the data in the packet data that cannot identify the source of the traffic according to the preset rules to obtain identifiable data;

Classify the identifiable data by using preset labels, and perform cluster calculation on the identifiable data of the same classification to obtain at least one data set;

At least one piece of the identifiable data is respectively extracted from the data sets corresponding to all the categories, and added to the application identification intelligence database.

In a second aspect, the present invention provides a data screening device for an application identification intelligence base, the device comprising:

A parsing unit, used to parse the collected target traffic into message data in a preset format;

a processing unit, configured to process, according to preset rules, the data of the unidentifiable traffic source in the packet data obtained by the parsing unit, and obtain identifiable data;

a computing unit, configured to use a preset label to classify the identifiable data obtained in the processing unit, and perform cluster calculation on the identifiable data of the same classification to obtain at least one data set;

an extraction unit, configured to extract at least one piece of the identifiable data from the data sets corresponding to all categories in the computing unit;

The adding unit is configured to add at least one piece of the identifiable data obtained in the extracting unit to the application identification intelligence database.

In order to achieve the above object, according to a third aspect of the present invention, a storage medium is provided, the storage medium includes a stored program, wherein when the program is run, the device where the storage medium is located is controlled to execute the above-mentioned first aspect. Apply data screening methods that identify intelligence bases.

In order to achieve the above object, according to a fourth aspect of the present invention, a processor is provided for running a program, wherein when the program runs, the data screening method for an application identification intelligence database of the first aspect is executed.

With the above technical solution, the present invention provides a data screening method and device for an application identification information base. Through the data screening solution provided by the present invention, the screening requirements for traffic data can be realized when the application identification database is established. After the data is parsed into packet data, the data that cannot identify the traffic source in the packet data can be processed according to the preset rules, so as to remove some unidentifiable traffic source data in the packet data, so as to realize the traffic data analysis. The content of redundant fields is deleted, and then the identifiable data is classified by the preset label, so that the identifiable data of the same preset label can be divided into one category, and then the identifiable data of the same category is clustered to calculate Reduce the computational burden of clustering calculation and reduce the computational burden of clustering calculation, and obtain different data sets based on clustering calculation, which can be understood as identifying the characteristic data of the application from different dimensions, in the corresponding data set When the dimensions are comprehensive enough, it can be considered that the valid and complete field information required to identify the application is retained, and finally at least one piece of identifiable data is extracted from the data sets corresponding to all classifications and added to the application identification intelligence database, so as to ensure When the intelligence database is established, the field information that can effectively identify the application is completely retained, and the network traffic data can be refined and compressed, thereby reducing the redundancy of invalid content in the intelligence database, and improving the identification efficiency of the intelligence database.

The above description is only an overview of the technical solutions of the present invention, in order to be able to understand the technical means of the present invention more clearly, it can be implemented according to the content of the description, and in order to make the above and other purposes, features and advantages of the present invention more obvious and easy to understand , the following specific embodiments of the present invention are given.

Description of drawings

Various other advantages and benefits will become apparent to those of ordinary skill in the art upon reading the following detailed description of the preferred embodiments. The drawings are for the purpose of illustrating preferred embodiments only and are not to be considered limiting of the invention. Also, the same components are denoted by the same reference numerals throughout the drawings. In the attached image:

1 shows a flowchart of a data screening method for an application identification intelligence base provided by an embodiment of the present invention;

FIG. 2 shows a flowchart of another data screening method using an identification intelligence base provided by an embodiment of the present invention;

3 shows a block diagram of a data screening apparatus for an application identification intelligence base provided by an embodiment of the present invention;

FIG. 4 shows a block diagram of the composition of another data screening apparatus for applying an identification information base provided by an embodiment of the present invention.

Detailed ways

Exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. While exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be embodied in various forms and should not be limited by the embodiments set forth herein. Rather, these embodiments are provided so that the present disclosure will be more thoroughly understood, and will fully convey the scope of the present disclosure to those skilled in the art.

In the field of network security technology, it is very important to identify applications at the entrance of the network. For the management of network traffic, applications are often identified by establishing an application identification intelligence database. It is composed of traffic data samples with different characteristics, and if there are more traffic data unrelated to application identification, it will affect the efficiency of the intelligence database to identify application traffic, and the existing processing methods for network traffic data are generally based on analysis After network traffic, the redundancy of field information content is reduced by extracting some field information and deduplication. However, the location and name of the field information that can identify the application in the network traffic can be described as various, and only extracting some fields will lose other fields. The field information that can effectively identify the application will easily lead to the situation of missed identification, and if the missed identification rate of the application is too high, it will lead to inaccurate identification, let go of the traffic that should not be let go, and cause harm to network security. Threats, such as some applications carry security loopholes, causing security risks, and some applications will upload personal privacy information, causing personal information leakage, etc., posing more severe challenges to application identification. Therefore, how to realize the refinement and compression of network traffic data while ensuring that the field information that can effectively identify the application is completely preserved when the intelligence database is established has become a problem to be solved at present. To this end, the embodiments of the present invention provide a data screening method for an application identification intelligence database, through which the refinement and compression of network traffic data can be achieved while completely retaining field information that can effectively identify applications, thereby reducing The invalid content in the intelligence base is redundant to improve the identification efficiency of the intelligence base. The specific implementation steps are shown in Figure 1, including:

101. Parse the collected target traffic into packet data in a preset format.

In this step, the target traffic is the collected network traffic of the application, and the network traffic of the application includes known application traffic samples and unknown application traffic samples. The traffic is collected, and the collection method for the target traffic may be offline collection, real-time collection, crawler crawling, or specific system interface and other related methods to collect data, etc., which is not specifically limited in this embodiment. The message is the data unit exchanged and transmitted in the network, which contains the complete data information to be sent. The length of the message is very inconsistent, and the length is unlimited and variable. For different types of traffic, there are different analysis methods. For example, for the traffic whose application layer is http protocol, the characteristics of the corresponding packets of this type of traffic are fixed, that is, http packets, which are the information used for http protocol interaction. , The http message of the requesting end is called a request message, and the one of the responding end is called a response message. The http message is divided into a message header and a message body, which are generally separated by blank lines. Therefore, fields such as host, Location, and User-Agent in the message header of the http message data can be directly extracted and parsed as characteristic information representing the traffic of the http protocol, so as to identify the source of the traffic subsequently.

102. Process the data in the packet data that cannot identify the traffic source according to a preset rule, and obtain identifiable data.

It should be noted that, in this step, the preset rule is a processing rule set based on the characteristics of the data whose traffic source cannot be identified, wherein, the data whose traffic source cannot be identified is the data that does not belong to or is not created by the application. Through this preset rule, the data in the packet data that cannot identify the source of the traffic can be processed to obtain identifiable data, and the identifiable data is used to represent the data that can identify the source of the traffic, that is, the data belonging to or created by the application, including However, it is not limited to application content data, application cache data, application configuration data, etc., which are not specifically limited in this embodiment.

103. Classify the identifiable data by using a preset label, and perform a clustering calculation on the identifiable data of the same classification to obtain at least one data set.

In this step, the preset label may be a basis for judging whether the identifiable data originates from the same application, and specifically may be domain name information in the field information corresponding to the identifiable data. When there is a preset label in the identifiable data When the data is the same, it can be considered that the data with the same preset label belongs to or is created by the same application, and then the data belonging to or created by the same application can be classified into the same category, and the data set is the identifiable data of the same category. The result generated by the clustering calculation is to perform similarity calculation on the identifiable data of the same classification, and divide the similar identifiable data into the same data set, so that step 104 can be executed subsequently.

104. Extract at least one piece of identifiable data from the data sets corresponding to all the classifications, and add it to the application identification intelligence database.

It should be noted that all classified datasets refer to multiple datasets belonging to different applications obtained by clustering of data that belong to or created by different applications. Due to clustering, the features in the same category can be similar. The identifiable data is divided into sets. Therefore, in order to avoid the situation that the content in the established application identification intelligence database is too redundant, it is only necessary to extract at least one piece of identifiable data from one data set as feature data and add it to the application identification intelligence database. , that is to say, the identifiable data in the same category can be understood as traffic data originating from the same application, and these identifiable data can be divided into multiple data sets through clustering, and different data sets can be understood as different data sets from different Dimension to identify the feature data of the application, therefore, when the dimension corresponding to the dataset is comprehensive enough, it can be considered that the valid and complete field information required to identify the application is retained, and since the feature data represented in the same dataset is Similarly, the amount of identifiable data selected from a data set can characterize the degree of redundancy in the data in the intelligence base. As for the extraction method of identifiable data, it may be randomly selected based on the identifiable data in the data set, or may be selected independently after manual analysis, which is not specifically limited in this implementation.

It can be seen based on the implementation of the above-mentioned FIG. 1 that the present invention provides a data screening method for an application identification information database. Through the data screening scheme provided by the present invention, the screening requirements for traffic data can be realized when the application identification database is established, After the traffic data is parsed into packet data, the data in the packet data that cannot identify the source of the traffic can be removed to reduce the content of redundant fields in the traffic data, and then the identifiable data can be classified and processed. Clustering calculation, to reduce the calculation amount of clustering calculation, reduce the computational burden of clustering calculation, and obtain different data sets based on clustering calculation, which can be understood as identifying the characteristic data of the application from different dimensions, in the data When the dimensions corresponding to the set are comprehensive enough, it can be considered that the valid and complete field information required to identify the application is retained, and finally at least one piece of identifiable data is extracted from the data sets corresponding to all categories and added to the application identification intelligence database. Therefore, while ensuring that the field information of valid and identifiable applications is completely retained when the intelligence base is established, the network traffic data can be refined and compressed, thereby reducing the redundancy of invalid content in the intelligence base, and improving the identification efficiency of the intelligence base.

Further, as a refinement and expansion of the embodiment shown in FIG. 1 , the embodiment of the present invention also provides another method for detecting suspicious traffic, as shown in FIG. 2 , and the specific steps are as follows:

201. Acquire corresponding packet data according to the request object in the target traffic.

It should be noted that a target traffic may contain multiple packet data, and one packet data generally corresponds to one request object. In the specific implementation process, there are cases where different packet data corresponds to the same request object. Therefore, in order to filter the packet data in the target traffic so as to exclude the packet data irrelevant to the request object, the corresponding packet data can be obtained based on the request object. The specific process can be: according to the request object to be processed, the The packet data in the target traffic is parsed to obtain the request object parsed from the packet data, and the parsed request objects are compared with the request objects to be processed one by one. If they are consistent, the parsed request objects are extracted. The corresponding message data is regarded as the message data corresponding to the request object to be processed, so as to realize the filtering of the message data, so as to perform the subsequent step 202 .

202. Parse the content of the request header of the packet data in a key-value pair format to obtain parsing data.

Among them, in the key-value pair format, the key is the field name in the request header content, and the value is the field information corresponding to the field name. In this step, since the target traffic corresponds to the traffic whose application layer is http protocol, the characteristics of the corresponding packets of this type of traffic are fixed, that is, http packets, so the request header part of the packet data can be directly extracted and parsed , and the request header part of the parsed message data is a plurality of fields generated in the format of key-value pairs, that is, the fields in the key-value format, where the key is the field name, and for the key, it will have a corresponding unique value, value is the field information corresponding to the key.

According to the method of steps 201-202, the corresponding packet data is obtained according to the request object in the target traffic, so as to realize the filtering of the packet data based on the request object, and request the filtered packet data in the format of key-value pairs Parsing the header content ensures that the parsed data only contains the content that is parsed for the content of the request header in the message data, and does not need to parse other parts of the message data again, further reducing the parsing data for the message data. amount to reduce memory usage.

203. Acquire field information corresponding to the specified field in the parsing data.

In this step, the specified fields are multiple feature fields preset by the user and need to be processed. Specifically, the field names corresponding to the specified fields can be matched with the field names in the parsed data. If the matching is successful, and When the field name has unique corresponding field information, the field information corresponding to the matched field name in the parsed data may be directly obtained, so as to perform the subsequent step 204 .

204. Process the field information based on a preset rule for processing a specified field to obtain identifiable data.

The preset rule is used to clear the field information of the specified field that does not have data information identifying the traffic source. In this step, the preset rule is a processing rule set based on the characteristics of the data information that cannot identify the source of the traffic. After acquiring the field information corresponding to the specified field in the parsed data, the corresponding preset rule can be found according to the specified field, Then, the field information corresponding to the specified field is processed based on the preset rules, so that the field information of the specified field does not have the data information that identifies the traffic source, so as to obtain the data information that identifies the traffic source, that is, the data can be identified.

It should be noted that, after the field information is processed based on the preset rules for processing specified fields, it may happen that the field information is null, and in this case, in order to avoid affecting the specified fields whose field information is null, the storage space is occupied. , correspondingly, it can detect whether the field information of the specified field after processing is empty, if it is empty, the specified field will be deleted from the parsed data.

Exemplarily, for different specified fields, the corresponding preset rules are as follows:

(1) For fields in the form of url_path, split by / and process them separately, including:

For pure numbers, replace with a space;

16 or 32 bit md5, replaced by a space;

If it contains md5 data in the format of xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx, replace it with a space;

For 40 or more numbers or letter combinations, replace with a space;

For the mac address type, replace it with a space;

Suffix with audio (mp3, wav, flc, etc.), video (mp4, avi, rm, rmvb, ts, etc.), picture (jpg, jpeg, png, gif, bmp, etc.) or network resource class (js, css, html) is ending, the part is replaced with a space;

Except for php, asp, aspx, jsp, jspx, ashx, action and no suffix, they are all replaced with a space;

After all processing is completed, remove leading and trailing spaces and /, if the result is empty, delete url_path.

(2) For fields in the form of url_query, separate them with & and process them separately, including:

If the value is a number or character, delete the value;

If the value is 16 or 32-bit md5, delete the value;

If the value contains md5 data in the format of xxxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx, delete the value;

If the value is a combination of more than 40 numbers or letters, delete the value;

If value is a mac address type, delete value;

If the value length is less than 4, delete the value;

After the above steps are processed, if there is no value and the key is a number or character, delete the key;

After the above steps are processed, if there is no value and the key length is less than 4 or greater than 39, delete the key;

Among them, the source data has no value, when splicing, do not add =;

(3) For User-Agent (User-Agent and UA fields), the string that does not represent the application in the regular match replacement field information is a space, including providing the browser type, operating system and version you are using to the visiting website, Identification of information such as CPU type, browser rendering engine, browser language, browser plug-ins, etc.;

After the replacement is complete, remove the leading and trailing spaces, if the last is empty, delete it.

(4) For other headers, delete the unidentifiable information fields, including:

accept, cache-control, connection, content-length, content-type, accept-encoding, cookie, content-encoding, transfer-encoding, accept-charset, charset, accept-language, date, applanguage, language, contenttype, qimei36, rakey, q-qimei, empty, value contains time format (^.*[0-9]{4}[0-9]{2}:[0-9]{2}:[0-9]{2} .*$), delete the field;

For example, Sun, 25Jul 2021 13:17:55GMT, where,

If the key contains token, date, sign, cookie or md5, delete the field;

If value is a combination of numbers or symbols, delete the field;

Among them, the value format is key1=value1&key2=value2 The formdata format is processed in the same way as the query;

If it is in formdata format;

If the value is a number or character, delete the value;

If the value is 16 or 32-bit md5, delete the value;

If the value contains md5 data in the format of xxxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx, delete the value;

If the value is a combination of more than 40 numbers or letters, delete the value;

If value is a mac address type, delete value;

If the value length is less than 4, delete the value;

If the value is empty after processing, delete the key.

According to the method in steps 203-204, by acquiring the field information corresponding to the specified field in the parsed data, and processing the field information based on the preset rules for processing the specified field, the parsed data in the packet data that cannot identify the traffic source can be directly processed. Remove, to achieve effective screening of identifiable data, so as to realize the deletion of redundant field content in traffic data, so as to reduce the amount of subsequent calculations for identifiable data.

Further, after parsing the data for processing and obtaining identifiable data, there may be duplicate identifiable data in the identifiable data. Therefore, in order to further reduce the redundancy caused by the content corresponding to the duplicate data information in the identifiable data, avoid Repeated calculation occurs in subsequent clustering of the same classification, which reduces the computational burden of clustering. It is also possible to obtain data information corresponding to each group of key-value pairs in the identifiable data before classifying the identifiable data using preset labels, and then detect Whether there is duplicate data information with the same key and value in the data information, if so, select a group of data information in the duplicate data information to keep in the identifiable data.

205. Acquire a preset label of identifiable data.

The preset label is used to represent the domain name information in the field information corresponding to the identifiable data. In this step, the domain name information is the corresponding host field in the field information corresponding to the identifiable data, the host field represents the domain name of the server and the port number monitored by the server, and for the traffic of the application layer http protocol, the host field is used to search the domain name, The application directly related to the domain name can be queried. If the query cannot be found, the main domain name in the host field is extracted, and the application related to the main domain name is queried, so as to preliminarily determine the application name corresponding to the identifiable data.

206. Divide the identifiable data corresponding to the same preset label into the same category.

In this step, after the application name is preliminarily determined in step 205, the identifiable data can be classified into the same category based on the application name, that is, the identifiable data belonging to the same application name are classified into the same classification, and then the identifiable data belonging to the same application name are classified into the same classification. Subsequent step 207 is performed on the identifiable data it contains on a class-by-class basis.

According to the method of steps 205-206, by obtaining preset labels of identifiable data and classifying the identifiable data of the same preset label into one category, the identifiable data belonging to the same application name can be classified, so that the same application The identifiable data can be subsequently calculated in units of classes (applications), which effectively avoids the interference of identifiable data in different classifications on subsequent clustering calculations, ensures the accuracy of subsequent calculations, and reduces subsequent clustering. It provides a basis for improving the processing efficiency of identifiable data.

207. Use a hierarchical clustering algorithm to calculate the similarity distance between any two identifiable data in the same category.

Among them, the similarity distance is calculated based on the key-value pair information of the identifiable data. It should be noted that the similarity distance is used to represent the similarity of any two data in the identifiable data. The hierarchical clustering algorithm in this step adopts the top-down splitting clustering algorithm. The principle is to look at all the data first. Then, the division is calculated until all identifiable data with the same similar distance are divided into multiple data sets, that is, similar identifiable data are divided together to represent that they have the same data characteristics. The algorithm is as follows:

①Assume there are N pieces of data, namely N1, N2, N3...Nn;

②Select a data arbitrarily, and divide the data into two sets {N1} and {N2, N3, N4...Nn};

③ Calculate the distance between N1 and the rest of the data separately, that is, take N1 as the reference point, calculate the distance between the rest of the data and N1, and divide the data into {N1}{N2,N5,N7}{N3 according to the principle of the same distance as a data set }{N4,N6,N8...Nn};

④Select a data Ni whose distance has not been calculated from the divided data set, that is, the data Ni as the reference point. The distance calculation is performed based on the data Ni and the data in other data sets. According to the same set and the same distance, it is a data set. The principle of dividing the data again, the same data set is retained, and the data is divided into {N1}{N2,N5,N7}{N3}{N4,N6,N8}{N9,N10...Nn}.

Repeat steps ③ and ④ until all data have been calculated by distance, and the distance function of the clustering algorithm is generally Euclidean distance, Manhattan distance, cosine distance, etc. In this implementation, a self-defined distance function is used to calculate the similarity distance .

Based on the refinement and expansion of the similarity distance in step 207, it may specifically include:

Obtain the key-value pair information corresponding to the identifiable data, and calculate the similarity distance between the two according to the key-value pair information contained in any two identifiable data. Among them, if the key in the key-value pair information exists in both identifiable data and the value is the same, the first calculation result is determined, and if the key in the key-value pair information exists in the two identifiable data and the value is different, Then determine the second calculation result, if the key in the key-value pair information only exists in one of the identifiable data, then obtain the third calculation result, the sum of the first calculation result, the second calculation result and the third calculation result is The similarity distance corresponding to the key-value pair information, the similarity distance of the identifiable data is the sum of the similarity distances of all the key-value pair information.

Exemplarily, by comparing the values of the keys of data A and data B, the similarity distance is calculated as follows:

The key is in A, not in B, and the set value of the distance is recorded as 100;

The key is in B, not in A, and the set value of the distance is recorded as 100;

The key is in both A and B, but the value is not equal, and the set value of the distance is recorded as 1;

The key is in both A and B, and the value is equal, and the set value of the distance is recorded as 0;

The distance between data A and data B is the sum of the above distances. The specific value of the set value of the distance can be set by self-definition, which is not specifically limited in this embodiment.

Corresponding to the above calculation method, data A and data B can be selected from the identifiable data respectively. It is assumed that data A includes three key-value pairs of x=1, y=2 and t=5, and data B includes x=1, y=2 and t=5. There are three key-value pairs of z=3 and t=4, where x/y/z/t is the field name, that is, the key, and the corresponding 1/2/3/4 is the field information, that is, the value. It can be seen by comparison:

The key-value pair x=1 exists in both data A and data B, and the key and value are the same, which can be recorded as 0;

The key-value pairs y=2 and z=3 exist only in data A or only in data B, respectively, and can be recorded as 100*2=200;

The key-value pairs t=5 and t=4 only exist in data A or only in data B, respectively, but the keys are the same and the values are different, which can be recorded as 1;

Therefore, the similarity distance between data A and data B is 0+200+1=201.

208. Divide identifiable data with the same similarity distance into the same data set according to the similarity distance.

It should be noted that since the similarity distance is used to represent the degree of similarity between any two data in the identifiable data, and the identifiable data with the same similarity distance has the same characteristics, therefore, for the same characteristics identifiable data that can be grouped into the same dataset.

According to the method of steps 207-208, by obtaining the key-value pair information corresponding to the identifiable data, the similarity distance between the two is calculated according to the key-value pair information contained in any two identifiable data, and then the identifiable data with the same similarity distance is calculated. The identification data is divided into the same data set, that is, the distance calculation is performed based on the comparison of the key-value pair information, and the comparison of the keys and values in the key-value pair information and the custom assignment for the comparison result can eliminate the difference of the same key. The impact of value on the results, thereby optimizing the similarity division between data and reducing the amount of identifiable data retained by the subsequent intelligence database.

209. Extract at least one piece of identifiable data from the data sets corresponding to all the classifications, and add it to the application identification intelligence database.

This step is combined with the description of step 104 in the above method, and the same content is not repeated here.

It can be seen from the above-mentioned implementation manner of FIG. 2 that the present invention provides a data screening method for an application identification intelligence database. First, the corresponding message data is obtained according to the request object in the target traffic, so as to realize the analysis of the message data based on the request object. and parses the content of the request header of the filtered packet data in the format of key-value pairs, which can ensure that the parsed data only contains the content parsed for the content of the request header in the packet data, without the need to parse the content of the request header in the packet data. Other parts of the text data are parsed again to further reduce the amount of parsed data for the message data, thereby reducing the memory occupation. Then, by obtaining the field information corresponding to the specified fields in the parsed data, and based on the preset rules for processing the specified fields The field information processing can directly remove the parsed data that cannot identify the traffic source in the packet data, realize the effective screening of the identifiable data, and delete the redundant field content in the traffic data, so as to reduce the subsequent targets for the identifiable data. The amount of data calculation, by obtaining the preset labels of identifiable data, and classifying the identifiable data of the same preset label into one category, the identifiable data belonging to the same application name can be classified, so that the identifiable data of the same application can be identified. Data can be used for subsequent calculations in units of classes (applications), which effectively avoids the interference between identifiable data in different categories on subsequent clustering calculations, ensures the accuracy of subsequent calculations, and reduces the amount of subsequent clustering calculations. This provides a basis for improving the processing efficiency of identifiable data. Then, by obtaining the key-value pair information corresponding to the identifiable data, the similarity distance between the two is calculated according to the key-value pair information contained in any two identifiable data, and then Divide identifiable data with the same similarity distance into the same data set, that is, the distance calculation is performed based on the comparison of key-value pair information, and the comparison of keys and values in key-value pair information is used to customize the comparison results. Assignment can eliminate the influence of different values of the same key on the results, optimize the similarity division between data, and reduce the retention of identifiable data in the follow-up intelligence database. To the application identification intelligence database, to realize the data screening requirements when the application identification intelligence database is established.

Further, as an implementation of the method embodiments shown in FIGS. 1-2 above, an embodiment of the present invention provides a data screening device for an application identification information base, which is used to completely retain field information that can effectively identify applications. Under the circumstance, the network traffic data can be refined and compressed, so as to reduce the redundancy of invalid content in the intelligence base, and improve the identification efficiency of the intelligence base. The embodiments of the apparatus correspond to the foregoing method embodiments. For ease of reading, this embodiment will not repeat the details of the foregoing method embodiments one by one, but it should be clear that the apparatus in this embodiment can correspondingly implement the foregoing method embodiments. the entire contents of the example. Specifically as shown in Figure 3, the device includes:

a parsing unit 31, configured to parse the collected target traffic into message data in a preset format;

The processing unit 32 is configured to process the data of the unidentifiable traffic source in the packet data obtained in the parsing unit 31 according to the preset rule, and obtain identifiable data;

The calculation unit 33 is used to classify the identifiable data obtained in the processing unit 32 by using a preset label, and perform cluster calculation on the identifiable data of the same classification to obtain at least one data set;

The extraction unit 34 is used to extract at least one piece of identifiable data from the data sets corresponding to all classifications in the computing unit 33;

The adding unit 35 is configured to add at least one piece of identifiable data acquired in the extracting unit 34 to the application identification intelligence database.

Further, as shown in Figure 4, the parsing unit 31 includes:

The first obtaining module 311 is configured to obtain corresponding message data according to the request object in the target traffic;

The parsing module 312 is configured to parse the request header content of the message data obtained in the first obtaining module 311 in a key-value pair format to obtain parsed data. In the key-value pair format, the key is the field name in the request header content, The value is the field information corresponding to the field name.

Further, as shown in FIG. 4 , the processing unit 32 includes:

The second obtaining module 321 is used to obtain the field information corresponding to the specified field in the parsed data;

The processing module 322 is used to process the field information obtained in the second acquisition module 321 based on the preset rules for processing the specified fields to obtain identifiable data, and the preset rules are used to clear the field information of the specified fields without identifying the traffic source. data information.

Further, as shown in FIG. 4 , the processing unit 32 further includes:

The detection module 323 is used to detect whether the field information of the specified field processed by the processing module 322 is empty;

The deletion module 324 is configured to delete the specified field from the parsed data if it is detected in the detection module 323 that the processed field information of the specified field is empty.

Further, as shown in Figure 4, the calculation unit 33 includes:

The third obtaining module 331 is configured to obtain a preset label of the identifiable data, and the preset label is used to represent the domain name information in the field information corresponding to the identifiable data;

The classification module 332 divides the identifiable data corresponding to the preset labels obtained in the same third obtaining module 331 into the same classification;

The calculation module 333 is used to calculate the similarity distance between any two identifiable data in the same classification obtained in the classification module 332 by using the hierarchical clustering algorithm, and the similarity distance is calculated based on the key-value pair information of the identifiable data;

The dividing module 334 is configured to divide the identifiable data with the same similarity distance into the same data set according to the similarity distance obtained in the calculation module 333 .

Further, as shown in Figure 4, the calculation module 333 includes:

The acquisition submodule 3331 is used to acquire the key-value pair information corresponding to the identifiable data;

The calculation submodule 3332 is used to calculate the similarity distance between the two according to the key-value pair information in any two identifiable data obtained in the acquisition submodule 3331;

Among them, if the key in the key-value pair information exists in both identifiable data and the values are the same, the first calculation result is determined; if the key in the key-value pair information exists in both identifiable data and the values are different, Then determine the second calculation result; if the key in the key-value pair information only exists in one of the identifiable data, then obtain the third calculation result; the sum of the first calculation result, the second calculation result and the third calculation result is the key value For the similarity distance corresponding to the information, the similarity distance of the identifiable data is the sum of the similarity distances of all key-value pairs of information.

Further, as shown in Figure 4, the device also includes:

The obtaining unit 36 obtains the data information corresponding to each group of key-value pairs in the identifiable data;

The detection unit 37 is used to detect whether there is duplicate data information with the same key and value in the data information obtained in the acquisition unit 36;

The retention unit 38 is configured to select a group of data information in the repeated data information to be retained in the identifiable data if the detection unit 37 detects that duplicate data information with the same key and value exists in the data information.

Further, an embodiment of the present invention further provides a storage medium, where the storage medium is used to store a computer program, wherein when the computer program runs, the device where the storage medium is located is controlled to execute the application described in the above-mentioned FIG. 1-2 Identify data screening methods for intelligence bases.

Further, an embodiment of the present invention further provides a processor for running a program, wherein when the program is running, the data screening method of the application identification intelligence database described in FIG. 1-2 is executed.

In the above-mentioned embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference may be made to the relevant descriptions of other embodiments.

It can be understood that the relevant features in the above-mentioned methods, systems and electronic devices may refer to each other. In addition, "first", "second", etc. in the above-mentioned embodiments are used to distinguish each embodiment, and do not represent the advantages and disadvantages of each embodiment.

Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the system, device and unit described above may refer to the corresponding process in the foregoing method embodiments, which will not be repeated here.

The algorithms and displays provided herein are not inherently related to any particular computer, virtual system, or other device. Various general-purpose systems can also be used with teaching based on this. The structure required to construct such a system is apparent from the above description. Furthermore, the present invention is not directed to any particular programming language. It should be understood that various programming languages may be used to implement the inventions described herein, and that the descriptions of specific languages above are intended to disclose the best mode for carrying out the invention.

In addition, memory may include non-persistent memory in computer readable media in the form of random access memory (RAM) and/or non-volatile memory, such as read only memory (ROM) or flash memory (flash RAM), including at least one memory chip.

As will be appreciated by those skilled in the art, the embodiments of the present application may be provided as a method, a system, or a computer program product. Accordingly, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present application may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) having computer-usable program code embodied therein.

The present application is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the present application. It will be understood that each flow and/or block in the flowchart illustrations and/or block diagrams, and combinations of flows and/or blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to the processor of a general purpose computer, special purpose computer, embedded processor or other programmable data processing device to produce a machine such that the instructions executed by the processor of the computer or other programmable data processing device produce Means for implementing the functions specified in a flow or flow of a flowchart and/or a block or blocks of a block diagram.

These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory result in an article of manufacture comprising instruction means, the instructions The apparatus implements the functions specified in the flow or flow of the flowcharts and/or the block or blocks of the block diagrams.

These computer program instructions can also be loaded on a computer or other programmable data processing device to cause a series of operational steps to be performed on the computer or other programmable device to produce a computer-implemented process such that The instructions provide steps for implementing the functions specified in the flow or blocks of the flowcharts and/or the block or blocks of the block diagrams.

In a typical configuration, a computing device includes one or more processors (CPUs), input/output interfaces, network interfaces, and memory.

Memory may include non-persistent memory in computer readable media, random access memory (RAM) and/or non-volatile memory in the form of, for example, read only memory (ROM) or flash memory (flash RAM). Memory is an example of a computer-readable medium.

Computer-readable media includes both persistent and non-permanent, removable and non-removable media, and storage of information may be implemented by any method or technology. Information may be computer readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read only memory (ROM), Electrically Erasable Programmable Read Only Memory (EEPROM), Flash Memory or other memory technology, Compact Disc Read Only Memory (CD-ROM), Digital Versatile Disc (DVD) or other optical storage, Magnetic tape cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission medium that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include transitory computer-readable media, such as modulated data signals and carrier waves.

It should also be noted that the terms "comprising", "comprising" or any other variation thereof are intended to encompass a non-exclusive inclusion such that a process, method, article or device comprising a series of elements includes not only those elements, but also Other elements not expressly listed, or which are inherent to such a process, method, article of manufacture, or apparatus are also included. Without further limitation, an element qualified by the phrase "comprising a..." does not preclude the presence of additional identical elements in the process, method, article of manufacture or apparatus that includes the element.

It will be appreciated by those skilled in the art that the embodiments of the present application may be provided as a method, a system or a computer program product. Accordingly, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present application may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) having computer-usable program code embodied therein.

The above are merely examples of the present application, and are not intended to limit the present application. Various modifications and variations of this application are possible for those skilled in the art. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of this application shall be included within the scope of the claims of this application.

Claims (10)

1. A data screening method for an application identification information base is characterized by comprising the following steps:

analyzing the collected target flow into message data in a preset format;

processing data which cannot identify a flow source in the message data according to a preset rule to obtain identifiable data;

classifying the identifiable data by using a preset label, and performing clustering calculation on the identifiable data of the same classification to obtain at least one data set;

and respectively extracting at least one piece of identifiable data from the data sets corresponding to all the classifications, and adding the data into an application identification intelligence library.

2. The method according to claim 1, wherein parsing the collected target traffic into message data in a preset format comprises:

acquiring corresponding message data according to a request object in the target flow;

analyzing the request header content of the message data in a key-value pair format to obtain analyzed data, wherein in the key-value pair format, a key is a field name in the request header content, and a value is field information corresponding to the field name.

3. The method according to claim 2, wherein processing data that cannot identify a traffic source in the packet data according to a predetermined rule to obtain identifiable data comprises:

Acquiring the field information corresponding to a specified field in the analysis data;

and processing the field information based on the preset rule for processing the specified field to obtain the identifiable data, wherein the preset rule is used for clearing data information which does not have an identified flow source in the field information of the specified field.

4. The method of claim 3, wherein processing the field information based on the preset rule for processing the specified field comprises:

detecting whether the field information of the specified field after being processed is empty;

and if the data is null, deleting the specified field from the analysis data.

5. The method of any of claims 1-4, wherein classifying the identifiable data using a predetermined label and performing a clustering computation on the identifiable data of the same classification to obtain at least one data set comprises:

acquiring a preset label of the identifiable data, wherein the preset label is used for representing domain name information in the field information corresponding to the identifiable data;

dividing the identifiable data corresponding to the same preset label into the same category;

Calculating a similar distance between any two identifiable data in the same classification by using a hierarchical clustering algorithm, wherein the similar distance is calculated based on key-value pair information of the identifiable data;

and dividing the identifiable data with the same similar distance into the same data set according to the similar distance.

6. The method of claim 4, wherein calculating the similarity distance between any two of the identifiable data in the same category using a hierarchical clustering algorithm comprises:

acquiring the key-value pair information corresponding to the identifiable data;

calculating the similar distance between any two identifiable data according to the key-value pair information in the two identifiable data;

if the key exists in the two identifiable data in the key-value pair information and the values are the same, determining a first calculation result; if the key exists in the two identifiable data in the key value pair information and the values of the key are different, determining a second calculation result; if the key in the key-value pair information only exists in one identifiable data, acquiring a third calculation result; the sum of the first calculation result, the second calculation result and the third calculation result is the similar distance corresponding to the key-value pair information, and the similar distance of the identifiable data is the sum of the similar distances of all the key-value pair information.

7. The method of claim 2, wherein prior to classifying the identifiable data using a predetermined label and performing a cluster computation on the identifiable data of the same classification to obtain at least one data set, the method further comprises:

acquiring data information corresponding to each group of key value pairs in the identifiable data;

detecting whether repeated data information with the same key and value exists in the data information;

if so, selecting one set of the data information in the repeated data information to be retained in the identifiable data.

8. A data screening apparatus for application identification intelligence libraries, the apparatus comprising:

the analysis unit is used for analyzing the acquired target flow into message data in a preset format;

the processing unit is used for processing data which cannot identify a flow source in the message data obtained in the analysis unit according to a preset rule to obtain identifiable data;

the calculation unit is used for classifying the identifiable data obtained in the processing unit by using a preset label and performing cluster calculation on the identifiable data of the same classification to obtain at least one data set;

An extracting unit, configured to extract at least one piece of the identifiable data from the data sets corresponding to all the classifications in the computing unit, respectively;

and the adding unit is used for adding at least one piece of identifiable data acquired in the extracting unit into an application identification intelligence library.

9. A storage medium, characterized in that the storage medium comprises a stored program, wherein the apparatus on which the storage medium is located is controlled to perform the method according to any one of claims 1 to 7 when the program is run.

10. A processor, characterized in that the processor is configured to run a program, wherein the program when running performs the method of any one of claims 1 to 7.

Images