CN107222472A - A kind of user behavior method for detecting abnormality under Hadoop clusters - Google Patents

Abstract

The invention discloses the user behavior method for detecting abnormality under a kind of Hadoop clusters, this method is by gathering and analyzing user behavior data, got off with logged, characteristic vector is formed according to the behavioural characteristic of user, utilize parallel Principal Component Analysis Algorithm processing feature vector set, the efficient behavior pattern for obtaining user, by contrasting the abnormal behaviour problem for finding that user produces when accessing HDFS with historical pattern, the security threat hidden under Hadoop clusters is found simultaneously, reaches the effect for ensureing HDFS safety.Effective monitoring is set up in data access behavior of the present invention not only to user, timely note abnormalities behavior, the data safety of Hadoop clusters is ensured, model training efficiency is also improved by parallelization Principal Component Analysis Algorithm, the problem of conventional model training effectiveness is low is solved.

Description

A user behavior anomaly detection method under Hadoop cluster

technical field

The invention relates to a method for detecting abnormal user behavior, in particular to a method for detecting abnormal user behavior based on a Hadoop cluster.

Background technique

In recent years, the Hadoop platform, as an excellent distributed computing system, has played an increasingly important role in large-scale data processing in enterprises. However, because Hadoop did not consider security factors at the beginning of its development, although some security mechanisms were added later, Hadoop's security audit mechanism, access control mechanism, and identity authentication mechanism are all passive static security technologies that cannot monitor user behavior. monitoring, which leads to vulnerability to hidden security attacks. For example, illegal users steal the account numbers and passwords of legitimate users, and obtain relevant permissions to illegally access data; data leakage problems are likely to occur during malicious intrusion, maintenance, and media loss, and it is difficult to guarantee the data security of the cluster. Data is the carrier of information. Once a data disaster occurs, it may cause immeasurable losses to users. Therefore, it is necessary to establish effective monitoring of user data access behaviors, detect abnormal behaviors in a timely manner, and ensure the data security of Hadoop clusters.

In terms of monitoring based on user behavior activities, there are relatively few domestic studies. Ashish Kamra and others proposed an anomaly detection method for relational database access patterns, which is based on user SQL query logs, but this method is only for relational databases, not suitable for user behavior monitoring on big data platforms; Mohiuddin Solaimani et al proposed a Spark-based virtual machine performance anomaly detection framework, the purpose of which is to find out which users occupy a large amount of resources by detecting virtual machine performance anomalies, resulting in unbalanced resource sharing and affecting cluster operation efficiency, but Spark is a memory-based computing , when the data scale is large or the intermediate results exceed the memory size, it cannot be processed; Liu Peng proposed an abstract architecture for databases and a general solution to abnormal behavior detection, but did not give a specific algorithm; Fredrik Valeur et al. proposed A machine learning-based SQL attack behavior detection method is proposed, but only for network-based back-end databases.

Traditional user behavior anomaly detection methods are mainly in database and cluster performance anomalies. Databases are generally for relational databases, etc., which cannot be applied in the distributed environment of Hadoop clusters, and cluster performance anomalies are not outstanding under the load balancing mechanism of the Hadoop platform itself, and the accuracy of anomaly detection results is not high. In addition, the data scale under the Hadoop cluster is usually very large, and the model training algorithm based on traditional principal component analysis is relatively inefficient.

Therefore, the optimization method of the present invention will establish effective monitoring of the user's data access behavior, find abnormal behavior in time, ensure the data security of the Hadoop cluster, and also improve the model training efficiency through the parallel principal component analysis algorithm, and solve the problem of traditional model training efficiency. low problem.

Contents of the invention

The purpose of the present invention is to overcome existing technical deficiencies, provide a kind of user behavior anomaly detection method under Hadoop cluster, not only can solve the abnormal behavior monitoring problem for user accessing HDFS data under Hadoop cluster, but also to traditional principal component The analysis algorithm is processed in parallel to solve the problem of low model training efficiency.

The purpose of the present invention is achieved by the following technical solutions: a user behavior abnormal detection method under a Hadoop cluster, comprising the following steps:

S1: User behavior data collection: Obtain HDFS audit logs from cluster NameNode nodes through Hadoop log management service (Log4j) and store them in the database;

S2: data preprocessing;

S3: model training: extract one of the user's partial eigenvector sets as training data and construct it as a sample data matrix, carry out dimensionality reduction processing on the sample data based on the parallel principal component analysis algorithm proposed by the present invention, obtain the sample mean value and transformation matrix, store into the user model library. The model training method for other users is the same. Among them, the transformation matrix mainly completes the function of mapping the sample from the original space to the principal component subspace;

S4: Abnormal user behavior detection: For a certain user, match the user's current behavior pattern (feature vector) with the historical behavior pattern trained by the user model. If they do not match, it is an abnormal behavior.

The user behavior data collection is to use Hadoop log management service, and default Hadoop has integrated Apache's open source project Log4j, through the Log4j log management service, the audit log of HDFS is obtained from the cluster NameNode node and stored in the database;

Described user behavior data is the audit record when the user visits the HDFS behavior, and the record includes access date and time, user identification, file operation command, client IP address;

The data preprocessing includes the following steps:

S21: Extract and count data, read audit records from the database, and count the number of occurrences of each file operation command within a time window for each user's audit records;

S22: Construct feature vectors.

The feature vector is constructed based on the frequency domain attribute, and the feature vector is represented by x=(x ₁ , x ₂ ,..., x ₁₃ ). There are 13 kinds of file operation commands in the feature vector, and each dimension The value represents the number of occurrences of a file operation command within the time window, and a set of feature vectors can be obtained by sequentially proceeding. Among them, the 13th dimension corresponds to the number of types of HDFS file operation commands. The feature vector set can be used as both model training data and test data;

Described model training comprises the following substeps:

S31: Construct a sample data matrix according to the extracted model training data;

S32: Calculate the variance matrix and sample mean value based on parallelized principal component analysis, divide the sample matrix horizontally into N blocks, and obtain the sample mean value and covariance matrix based on the MapReduce calculation model;

S33: Calculate the eigenvalues and corresponding eigenvectors of the covariance matrix, and determine the number of principal components k according to the variance contribution rate;

S34: Determine the principal components according to the variance contribution rate and construct a transformation matrix, construct a transformation matrix according to the eigenvectors corresponding to the top k eigenvalues, and the product of the sample matrix and the transformation matrix is the principal component matrix;

S35: Store the obtained sample mean value and transformation matrix into the model database for use in anomaly detection.

The described abnormal user behavior detection includes the following sub-steps:

S41: For a certain user, extract a feature vector from the test data, and perform mean value adjustment processing;

S42: Calculate the Euclidean distance between the mean-adjusted vector and the principal component reconstruction of the vector, if the distance is greater than a preset threshold, it is an abnormal behavior; otherwise, it is a normal behavior;

The principal component reconstruction of the mean-adjusted vector is to train the mean-adjusted vector to obtain a transformation matrix, and then map it to the principal component subspace, and then use the transposition of the transformation matrix to convert the mapped new vector The vector obtained by reconstructing back to the original space;

The described abnormal user behavior detection method divides the abnormal detection of user behavior into two situations for testing:

(1) If the false detection rate of the detection method is to be tested, a part of the data extracted from a user's feature vector data is used as training data, and the remaining part is used as test data;

(2) If the detection rate of the detection method is to be tested, part of the data extracted from one user's feature vector data is used as training data, and parts of other users are extracted as test data.

The beneficial effect of the present invention is: provide a kind of effective, correct abnormal behavior detection method for HDFS file data access behavior under Hadoop cluster, this method has overcome the problem that traditional abnormal detection method is not applicable under Hadoop cluster environment, and The principal component analysis algorithm used in this method is improved in parallel, which improves the efficiency of model training.

Description of drawings

Fig. 1 is a flowchart of the present invention;

Fig. 2 is the model training flowchart of the present invention;

Fig. 3 is a flow chart of abnormal user behavior detection of the present invention;

Fig. 4 is a process diagram of parallel processing of principal component analysis in the present invention.

detailed description

The technical solution of the present invention will be further described in detail below in conjunction with the accompanying drawings, but the protection scope of the present invention is not limited to the following description.

As shown in Figure 1, a user behavior anomaly detection method under a Hadoop cluster comprises the following steps:

S1: User behavior data collection. Hadoop integrates Apache's open source project Log4j by default, and obtains HDFS audit logs from the cluster NameNode through the Log4j log management service and stores them in the database;

S2: Preprocessing of data. Read the audit records from the database, for each user's audit records, based on a time window, count the number of occurrences of each file operation command within the time period, and combine them to form a feature vector, the feature vector is expressed by x=(x ₁ ,x ₂ ,…,x ₁₃ ), there are 13 kinds of file operation commands in total, and the value of each dimension represents the number of occurrences of a file operation command in the time window, and a set of feature vectors can be obtained by performing sequentially, that is, Pattern to be detected. The feature vector set can be used as model training data and test data;

S3: Model training: extract one of the user's partial feature vector sets as training data and construct it as a sample data matrix, perform dimensionality reduction processing on the sample data based on the parallel principal component analysis algorithm proposed by the present invention, obtain the sample mean value and transformation matrix, store into the user model library. The model training method for other users is the same. Among them, the transformation matrix mainly completes the function of mapping the sample from the original space to the principal component subspace;

S4: Abnormal user behavior detection: For a certain user, match the current behavior pattern (feature vector) of the user with the historical behavior pattern trained by the user model. If they do not match, it is an abnormal behavior.

As shown in Figure 2, the steps of model training are:

S31: Construct a sample data matrix according to the extracted model training data (feature vector set);

S32: As shown in Figure 4, based on parallel principal component analysis, find the variance matrix and sample mean value, divide the sample matrix horizontally into N blocks, and obtain the sample mean value and covariance matrix based on the MapReduce computing model; extract one of the users Part of the eigenvector set is used as training data and constructed as a sample data matrix. Based on the parallel principal component analysis algorithm proposed by the present invention, the dimensionality reduction process is performed on the sample data to obtain the sample mean value and transformation matrix, which are stored in the user model library. The model training method for other users is the same. Among them, the transformation matrix mainly completes the function of mapping the sample from the original space to the principal component subspace;

The specific parallel principal component analysis formula is: get the eigenvector matrix X _i , X _i =[X1,X2,...,X13] ^T , and the mean matrix and covariance matrix of X are denoted as μ=E(X) and Σ=D(X).

S33: Calculate the eigenvalues and corresponding eigenvectors of the covariance matrix, and determine the number of principal components k according to the variance contribution rate;

S34: Determine the principal components according to the variance contribution rate and construct a transformation matrix, construct a transformation matrix according to the eigenvectors corresponding to the top k eigenvalues, and the product of the sample matrix and the transformation matrix is the principal component matrix;

S35: Obtain the principal component matrix according to the transformation matrix, and store the obtained sample mean value and transformation matrix into the model database for use in anomaly detection.

As shown in Figure 3, user behavior anomaly detection, for a certain user, matches the user's current behavior pattern (feature vector) with the historical behavior pattern trained by the user model, if it does not match, it is an abnormal behavior, specifically Proceed as follows:

S41: using the behavior feature vector of the current user as test data;

S42: Under the MapReduce framework, the behavior feature vector and mean value of the current user are adjusted to the data to be detected;

S43: Calculate the distance between the feature vector to be detected and the reconstructed feature vector of the principal component;

S44: judgment threshold: if the distance is greater than the threshold, the current user behavior is classified as an abnormal behavior record, and if the threshold is not exceeded, the current user behavior is classified as a normal behavior;

S45: Judging whether there is still test data: if there is still test data, re-adjust the mean value, and end the test when no test data exists.

The above descriptions are only preferred embodiments of the present invention. It should be understood that the present invention is not limited to the form disclosed herein, and should not be regarded as excluding other embodiments, but can be used in various other combinations, modifications and environments, and Modifications can be made within the scope of the ideas described herein, by virtue of the above teachings or skill or knowledge in the relevant art. However, changes and changes made by those skilled in the art do not depart from the spirit and scope of the present invention, and should all be within the protection scope of the appended claims of the present invention.

Claims (10)

1. a user behavior anomaly detection method under a Hadoop cluster, is characterized in that, it comprises the following steps: S1: user behavior data collection, described user behavior data comprises the audit record that user visits Hadoop cluster HDFS; S2: Data preprocessing, for each user's audit record, based on a time window, count the user behavior characteristics in the time window to form a feature vector, and then apply it to different users and different time windows in turn, and then get the inclusion A feature vector set of multiple users and their behavior characteristics in different time periods; S3: model training, extracting part of the feature vector set of each user as training data and constructing it as a sample data matrix, carrying out dimensionality reduction processing on the sample data, obtaining the sample mean value and transformation matrix, and the transformation matrix transforms the sample from the original space Mapped to the principal component subspace; S4: Abnormal user behavior detection. 2. the abnormal detection method of user behavior under a kind of Hadoop cluster according to claim 1, is characterized in that, described audit record comprises access date and time, user identification, file operation order, client IP address; The audit records are obtained from the cluster NameNode node through the Hadoop log management service. 3. the user behavior anomaly detection method under a kind of Hadoop cluster according to claim 1, is characterized in that, described user behavior characteristic, comprises this user in several log records in described time window, each kind The number of occurrences of the file operation command. 4. the abnormal user behavior detection method under a kind of Hadoop cluster according to claim 4, is characterized in that, the feature vector that described user behavior feature constitutes is expressed as x=(x ₁ , x ₂ ,..., x _n ), where n is the total number of file operation commands, and the value of each dimension of the feature vector represents the number of occurrences of a file operation command within the time window. 5. the abnormal user behavior detection method under a kind of Hadoop cluster according to claim 1, is characterized in that, described sample data dimensionality reduction process, comprises the following substeps: S21: Extract and count data, read audit records from the database, and count the number of occurrences of each file operation command within a time window for each user's audit records; S22: Construct feature vectors. 6. the abnormal user behavior detection method under a kind of Hadoop cluster according to claim 5, it is characterized in that, this feature vector is to construct feature vector based on frequency domain attribute, and this feature vector has 13 kinds of file operation commands altogether, each The value of the dimension represents the number of occurrences of a file operation command within the time window. Among them, the 13th dimension corresponds to the number of types of HDFS file operation commands. The feature vector set can be used as both model training data and test data. 7. the abnormal user behavior detection method under a kind of Hadoop cluster according to claim 1, is characterized in that, described model training comprises the following substeps: S31: Construct a sample data matrix according to the extracted model training data; S32: Find variance matrix and sample mean value based on parallel principal component analysis, divide sample matrix horizontally into N blocks, obtain sample mean value and covariance matrix based on MapReduce computing model; S33: Calculate the eigenvalues and corresponding eigenvectors of the covariance matrix, and determine the principal component quantity k according to the variance contribution rate; S34: construct transformation matrix according to the eigenvectors corresponding to the top k eigenvalues, the product of sample matrix and transformation matrix is the principal component matrix; S35: Store the obtained sample mean value and transformation matrix into the model database for use in anomaly detection. 8. the abnormal user behavior detection method under a kind of Hadoop cluster according to claim 1, is characterized in that, described user behavior abnormal detection comprises the following substeps: S41: For a certain user, extract a feature vector from the test data, and perform mean value adjustment processing; S42: Calculate the Euclidean distance between the vector after the mean adjustment process and the principal component reconstruction of the vector, if the distance is greater than a preset threshold, it is an abnormal behavior; otherwise, it is a normal behavior. 9. the user behavior anomaly detection method under a kind of Hadoop cluster according to claim 6, it is characterized in that, the described principal component reconstruction of the vector adjusted by the mean is obtained through training the vector adjusted by the mean The transformation matrix is then mapped to the principal component subspace, and then the transformed vector is reconstructed back to the original space by using the transposition of the transformation matrix. 10. the abnormal detection method of user behavior under a kind of Hadoop cluster according to claim 1, is characterized in that, the abnormal detection of user behavior is divided into two kinds of situations and tests: (1) If the false detection rate of the detection method is to be tested, a part of the data extracted from a user's feature vector data is used as training data, and the remaining part is used as test data; (2) If the detection rate of the detection method is to be tested, part of the data extracted from one user's feature vector data is used as training data, and parts of other users are extracted as test data.