CN115567572B

CN115567572B - Method, device, equipment and storage medium for determining abnormality of an object

Info

Publication number: CN115567572B
Application number: CN202211125374.7A
Authority: CN
Inventors: 张丽; 杜悦艺; 孙亚生; 朱欤
Original assignee: Beijing Baidu Netcom Science and Technology Co Ltd
Current assignee: Beijing Baidu Netcom Science and Technology Co Ltd
Priority date: 2022-09-15
Filing date: 2022-09-15
Publication date: 2025-05-27
Anticipated expiration: 2042-09-15
Also published as: CN115567572A

Abstract

The present disclosure provides a method, device, equipment and storage medium for determining the abnormality of an object, which relates to the field of computer technology, especially to the technical fields of artificial intelligence, big data, machine learning, etc. The specific implementation scheme is: determining detailed features according to the web page access data of the target object; determining vertical comparison features according to the web page access data of the target object and the web page access data of other objects; determining horizontal comparison features according to the web page access data of the target object in the current time period and the web page access data of the target object in the previous time period; and performing abnormality detection on the vertical comparison features, horizontal comparison features and detailed features to obtain the target abnormality of the target object.

Description

Method, device, equipment and storage medium for determining object anomaly degree

Technical Field

The present disclosure relates to the field of computer technology, and in particular, to the technical fields of artificial intelligence, big data, machine learning, and the like.

Background

As the network scale is increasingly enlarged and the network structure is complicated, the information security problem is more and more serious. How to secure sensitive information has become an important issue. In recent years, network attacks have gradually moved from external attacks to internal attacks. The internal attack has the characteristics of generality, weak specificity, strong concealment and the like, can easily bypass the firewall and the monitoring of an intrusion detection system, is more difficult to prevent than external attacks such as external network viruses, hacking attacks and the like, and causes larger loss. Firewalls, IDSs (intrusion detection systems), have failed to meet these security requirements. Internal attacks are typically caused by internal employees, such as business espionage, transfer portability, off-duty returns, and the like. How to mine out abnormal risk users in a large number of user access behaviors is a urgent problem to be solved.

Disclosure of Invention

The present disclosure provides a method, apparatus, device, storage medium, and program product for determining object anomaly.

According to one aspect of the disclosure, a method for determining object anomaly is provided, which comprises determining detail features according to webpage access data of a target object, determining longitudinal contrast features according to the webpage access data of the target object and webpage access data of other objects, determining transverse contrast features according to the webpage access data of the target object in a current time period and the webpage access data of the target object in a last time period, and performing anomaly detection on the longitudinal contrast features, the transverse contrast features and the detail features to obtain target anomaly of the target object.

According to another aspect of the disclosure, an apparatus for determining an object anomaly degree is provided, which includes a first determining module configured to determine a detail feature according to web page access data of a target object, a second determining module configured to determine a longitudinal contrast feature according to web page access data of the target object and web page access data of other objects, a third determining module configured to determine a transverse contrast feature according to web page access data of the target object in a current time period and web page access data of the target object in a previous time period, and a detecting module configured to perform anomaly detection on the longitudinal contrast feature, the transverse contrast feature, and the detail feature to obtain a target anomaly degree of the target object.

Another aspect of the present disclosure provides an electronic device comprising at least one processor and a memory communicatively coupled to the at least one processor, wherein the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the methods shown in the embodiments of the present disclosure.

According to another aspect of the disclosed embodiments, there is provided a non-transitory computer-readable storage medium storing computer instructions for causing a computer to perform the method shown in the disclosed embodiments.

According to another aspect of the disclosed embodiments, there is provided a computer program product comprising a computer program/instruction, characterized in that the computer program/instruction, when executed by a processor, implements the steps of the method shown in the disclosed embodiments.

It should be understood that the description in this section is not intended to identify key or critical features of the embodiments of the disclosure, nor is it intended to be used to limit the scope of the disclosure. Other features of the present disclosure will become apparent from the following specification.

Drawings

The drawings are for a better understanding of the present solution and are not to be construed as limiting the present disclosure. Wherein:

FIG. 1 is a schematic view of an application scenario of a method, apparatus, electronic device and storage medium for determining object anomaly according to an embodiment of the present disclosure;

FIG. 2 schematically illustrates a flow chart of a method of determining object anomaly in accordance with an embodiment of the present disclosure;

FIG. 3 schematically illustrates a flow chart of a method of anomaly detection of longitudinal contrast features, lateral contrast features, and detail features, in accordance with an embodiment of the present disclosure;

FIG. 4A schematically illustrates a method of determining a first test result according to an embodiment of the disclosure;

FIG. 4B schematically illustrates a method of determining a second detection result according to an embodiment of the disclosure;

FIG. 4C schematically illustrates a method of determining a third test result according to an embodiment of the disclosure;

FIG. 4D schematically illustrates a method of determining target anomalies according to an embodiment of the present disclosure;

FIG. 5 schematically illustrates a method of determining object anomaly in accordance with an embodiment of the present disclosure;

FIG. 6 schematically illustrates a block diagram of an apparatus for determining object anomaly in accordance with an embodiment of the present disclosure;

FIG. 7 schematically illustrates a block diagram of an example electronic device that may be used to implement embodiments of the present disclosure.

Detailed Description

Exemplary embodiments of the present disclosure are described below in conjunction with the accompanying drawings, which include various details of the embodiments of the present disclosure to facilitate understanding, and should be considered as merely exemplary. Accordingly, one of ordinary skill in the art will recognize that various changes and modifications of the embodiments described herein can be made without departing from the scope and spirit of the present disclosure. Also, descriptions of well-known functions and constructions are omitted in the following description for clarity and conciseness.

An application scenario of the method, the apparatus, the electronic device, and the storage medium for determining the object anomaly provided in the present disclosure will be described below with reference to fig. 1.

Fig. 1 is an application scenario schematic diagram of a method, an apparatus, an electronic device, and a storage medium for determining object anomaly according to an embodiment of the present disclosure. It should be noted that fig. 1 illustrates only an example of an application scenario in which the embodiments of the present disclosure may be applied to help those skilled in the art understand the technical content of the present disclosure, but it does not mean that the embodiments of the present disclosure may not be applied to other devices, systems, environments, or scenarios.

As shown in fig. 1, the application scenario 100 may include terminal devices 101, 102, 103, a network 104, and a server 105. The network 104 is used as a medium to provide communication links between the terminal devices 101, 102, 103 and the server 105. The network 104 may include various connection types, such as wired, wireless communication links, or fiber optic cables, among others.

The user may interact with the server 105 via the network 104 using the terminal devices 101, 102, 103 to receive or send messages or the like. Various communication client applications, such as shopping class applications, web browser applications, search class applications, instant messaging tools, mailbox clients, social platform software, etc. (by way of example only) may be installed on the terminal devices 101, 102, 103.

The terminal devices 101, 102, 103 may be a variety of electronic devices having a display screen and supporting web browsing, including but not limited to smartphones, tablets, laptop and desktop computers, and the like.

The server 105 may be a server providing various services, such as a background management server providing support for websites browsed by users using the terminal devices 101, 102, 103. The background management server may analyze and process the received data such as the user request, and feed back the processing result (e.g., the web page, information, or data obtained or generated according to the user request) to the terminal device.

Illustratively, in this embodiment, the terminal devices 101, 102, 103 may be in an intranet environment. The user can browse the intranet sites by using the terminal devices 101, 102, 103. The server 105 may be, for example, a background management server that provides support for intranet sites browsed by users using the terminal devices 101, 102, 103.

The terminal devices 101, 102, 103 may generate a log of surfing behavior when a user accesses a website using the terminal devices 101, 102, 103. The log of online behavior records which web pages the user accessed specifically at the descriptive time, and whether the access was successful, and may also record cookie information, uesr-agent information, etc. when the web pages were accessed. Wherein uesr-agent may enable the server to identify the operating system and version used by the user, the CPU (central processing unit) type, browser and version, browser rendering engine, browser language, browser plug-in, etc.

It should be noted that, the method for determining the anomaly degree of the object provided by the embodiment of the disclosure may be generally performed by the server 105. Accordingly, the apparatus for determining object anomaly provided by the embodiments of the present disclosure may be generally provided in the server 105. The method of determining object anomaly provided by the embodiments of the present disclosure may also be performed by a server or a server cluster that is different from the server 105 and is capable of communicating with the terminal devices 101, 102, 103 and/or the server 105. Accordingly, the apparatus for determining object anomaly provided by the embodiments of the present disclosure may also be provided in a server or a server cluster that is different from the server 105 and is capable of communicating with the terminal devices 101, 102, 103 and/or the server 105.

It should be understood that the number of terminal devices, networks and servers in fig. 1 is merely illustrative. There may be any number of terminal devices, networks, and servers, as desired for implementation.

The server can be a cloud server, also called a cloud computing server or a cloud host, and is a host product in a cloud computing service system, so that the defects of high management difficulty and weak service expansibility in the traditional physical hosts and VPS service (' ' Virtual PRIVATE SERVER ' ' or ' VPS ' ' for short) are overcome. The server may also be a server of a distributed system or a server that incorporates a blockchain.

In the technical scheme of the disclosure, the related processes of collecting, storing, using, processing, transmitting, providing, disclosing, applying and the like of the personal information of the user all conform to the regulations of related laws and regulations, necessary security measures are adopted, and the public order harmony is not violated.

In the technical scheme of the disclosure, the authorization or consent of the user is obtained before the personal information of the user is obtained or acquired.

The method of determining object anomaly provided by the present disclosure will be described below in connection with fig. 2.

Fig. 2 schematically illustrates a flowchart of a method of determining object anomaly in accordance with an embodiment of the present disclosure.

As shown in fig. 2, the method 200 of determining object anomaly degree includes determining detail features according to web page access data of a target object in operation S210.

According to an embodiment of the present disclosure, the target object may comprise, for example, a user to be detected, which may access the web page through the terminal device.

According to embodiments of the present disclosure, the web page access data may include, for example, data recorded by the terminal device and/or the server when the user accessed the web page, and may include, for example, at least one of the number of user agent information used per unit time of the target object, the web page accessed per unit time of the target object, the number of accesses per web page, the total number of failures to access per web page per unit time of the target object, and the difference between the number of user agent information of the non-working time and the number of user agent information of the working time per unit time of the target object. The user agent information may include, for example, a user-agent. The unit time can be set according to actual needs, for example, a day.

According to embodiments of the present disclosure, the detail features may be used to represent the fine-grained internet behavior features of the user to be detected.

Then, in operation S220, the vertical contrast feature is determined according to the web page access data of the target object and the web page access data of the other objects.

According to embodiments of the present disclosure, other objects may include, for example, other users in addition to the user to be detected.

According to embodiments of the present disclosure, the vertical contrast feature may be used to represent the internet surfing behavior feature of the user to be detected and other users.

In operation S230, the lateral contrast characteristic is determined according to the web page access data in the current time period of the target object and the web page access data in the previous time period of the target object.

According to embodiments of the present disclosure, the time period may be set according to actual needs, for example. For example, the inter-period may be set to 1 week.

According to embodiments of the present disclosure, the lateral contrast feature may be used to represent the internet surfing behavior feature during the current time period and during the previous time period of the user to be detected.

In operation S240, anomaly detection is performed on the longitudinal contrast feature, the transverse contrast feature, and the detail feature to obtain a target anomaly of the target object.

According to embodiments of the present disclosure, the target anomaly may be used to represent the anomaly level of the target object. Whether the target object is an abnormal user may be determined according to the target abnormality. For example, it may be determined whether the target abnormality of the target user is greater than an abnormality threshold, and if so, the target user is determined to be an abnormal user. The abnormality threshold may be set according to actual needs.

According to the embodiment of the disclosure, whether a large deviation exists between the internet surfing behavior characteristics between the user to be detected and other users can be detected according to the longitudinal comparison characteristics. And detecting whether a larger deviation exists between the surfing behavior characteristic in the current time period of the user to be detected and the surfing behavior characteristic in the previous time period according to the longitudinal comparison characteristic. And detecting whether the internet surfing behavior characteristic of the user to be detected is abnormal or not according to the detail characteristic. And then determining the target anomaly degree of the user to be detected according to the three detection results.

According to the method for determining the anomaly degree of the object, the anomaly detection is performed by adopting multiple dimensions. Anomaly detection is not only by comparison of the user himself at different times, but also by comparison between the user and other users, i.e. by combining a plurality of angles, transverse, longitudinal etc. Lateral contrast is a change in relatively long-term behavioral habits used to capture user acquaintance. Vertical contrast is to capture the difference in behavior of a user compared to other users per unit of time. Meanwhile, the user anomaly degree is depicted by longitudinal comparison and transverse comparison, so that the abnormal user can be better and comprehensively detected.

According to another embodiment of the present disclosure, the vertical contrast feature may include, for example, the number of user agent information used per unit time of the intranet user, the number of times each web page is accessed per unit time, the number of failures to access each web page per unit time, and the difference between the number of user agent information for non-working time and the number of user agent information for working time per unit time of the user, etc. The lateral comparison feature may include, for example, a difference between an average of the number of user agent information used by the intranet user during the current time period and an average of the number of user agent information used during the last time period, a difference between an average number of accesses of each web page by the user during the current time period and an average number of accesses of corresponding web pages by the last time period, a difference between an average number of failed accesses of each web page by the user during the current time period and an average number of failed accesses of corresponding web pages by the last time period, a change in a difference between the number of user agent information of the user during the non-working time and the working time of the user agent information of the user during the current time period over the last time period, and the like. The detail features may include, for example, the number of times each web page is accessed per unit time during the user's current time period.

By using various types of features such as user agent information, access failure times, access times in non-working hours and the like, the dimension of the features is more comprehensive. In addition, the longitudinal comparison feature, the transverse comparison feature and the detail feature are used simultaneously, wherein the longitudinal comparison feature and the transverse comparison feature comprise the features of a summary class, a polymerization class and an average class, and the detail feature comprises a time sequence feature, so that the angle of the features is more comprehensive. Therefore, the user anomaly degree can be better represented, and the anomaly detection accuracy is improved.

According to the embodiment of the disclosure, the log of the surfing behavior of the intranet user can be collected. The internet surfing behavior log records what intranet web page is accessed by all intranet users at what time, whether the access is successful or not, cookie information, user-agent information and the like when the intranet web page is accessed. The web page access data may then be extracted from the log of online behavior. The web page access data may include a web page accessed by a user, whether each access is successful, user-agent information accessed each time, and the like.

According to the embodiment of the disclosure, the user-agent can be directly used as user agent information. For example, the user-agent may be encoded according to the page browsing amount corresponding to each user-agent, so as to obtain the encoded information of the user-agent, where the larger the page browsing amount corresponding to the user agent field is, the smaller the corresponding user agent code is. In this embodiment, the code may be stored in the form of a character string, and the smaller the code, the smaller the number of bits, and the smaller the occupied storage space. By encoding the user-agent, the occupation of the storage space can be reduced.

According to the disclosed embodiments, behavioral characteristics of the user's inactivity time are taken into account when detecting user anomalies. Based on this, for example, if the user amount within a certain reference unit time is smaller than the predetermined user amount, it may be determined that the reference unit time belongs to the non-operation period. The reference unit time may be set according to actual needs, for example, may be 1 hour. The predetermined user amount may be set according to actual needs, for example, an average internet surfing user amount per unit time in a predetermined time may be counted, and then the predetermined user amount is determined according to the average internet surfing user amount. Wherein the predetermined time may be set according to actual needs, for example, it may be the last 14 days.

According to the disclosed embodiments, for example, web pages with a user quantity less than a user quantity threshold or a number of accesses less than a number of accesses threshold may be combined and counted as one web page. Therefore, the dimension of the features can be reduced, and the feature sparseness is avoided. In addition, the detection of the abnormality can be more comprehensive.

FIG. 3 schematically illustrates a flow chart of a method of anomaly detection of longitudinal contrast features, lateral contrast features, and detail features, in accordance with an embodiment of the present disclosure.

As shown in fig. 3, the method 340 for detecting anomalies of the longitudinal contrast feature, the transverse contrast feature, and the detail feature includes performing anomalies of the longitudinal contrast feature to obtain a first detection result in operation S341.

According to the embodiment of the disclosure, the first machine learning model may be used for detecting the abnormality of the transverse comparison feature, so as to obtain a first detection result. The first machine learning model may include at least one of an isolated forest model, hbos (Histogram-based Outlier Score, histogram-based anomaly detection) model, and copod (Copula-based Outlier Detection, copula function-based anomaly detection), for example.

In operation S342, the anomaly detection is performed on the transverse contrast feature, so as to obtain a second detection result.

According to the embodiment of the disclosure, the anomaly detection can be performed on the transverse contrast feature by using a second machine learning model, so as to obtain a second detection result. Wherein the second machine learning model may include at least one of an isolated forest model, hbos model, and copod model, for example. The first machine learning model and the second machine learning model may be the same or different.

In operation S343, abnormality detection is performed on the detail features, and a third detection result is obtained.

According to the embodiment of the disclosure, for example, the abnormality detection can be performed on the detail features by using a deep learning model, so as to obtain a third detection result. The deep learning model may include at least one of a self-coding model and a clustering model, for example. The self-encoding model may include, for example, an LSTM (long short term memory artificial intelligence network) model. The cluster model may include, for example, a kmeans (k-means cluster) model.

In operation S344, the target abnormality of the target object is determined according to the first, second, and third detection results.

According to the embodiments of the present disclosure, for example, the first detection result, the second detection result, and the third detection result may be comprehensively considered to determine the target abnormality of the target object.

It should be noted that any execution order may be adopted between operations S341 to S343, which is not specifically limited in the present disclosure.

According to the embodiment of the disclosure, for example, internet surfing behavior features of a plurality of users can be collected in advance as training data, the training data is input into a network model, training is performed by using a gradient descent method, and parameters of an optimization model are updated through multiple rounds of iteration until convergence.

The method for determining the target anomaly degree of the target object described above is further described with reference to fig. 4A to 4D in conjunction with the specific embodiment. Those skilled in the art will appreciate that the following example embodiments are merely for the understanding of the present disclosure, and the present disclosure is not limited thereto.

Illustratively, in this embodiment, the anomaly detection may be performed on the longitudinal contrast feature and the transverse contrast feature by using an isolated forest model, hbos model, and copod model, respectively, and on the detail feature by using a self-coding model and a clustering model.

Wherein the isolated forest model applies an isolated forest algorithm. The isolated forest algorithm is an anomaly detection algorithm capable of processing large-scale multidimensional data. The outlier forest algorithm considers few and very different outlier samples, so that during the construction of the binary tree, the outlier samples are very easily isolated and thus closest to the root node. However, the isolated forest algorithm has some drawbacks, such as that some users only perform very abnormally on individual indexes, but other indexes are normal. Then according to the logic of the isolated forest algorithm, if many indexes are compared with abnormal users, the abnormal users on the individual indexes are easily judged as normal users. And the isolated forest algorithm has certain randomness.

Thus, for more comprehensive anomaly assessment, hbos and copod models are also introduced in this embodiment for anomaly detection. hbos is a combination of univariate methods, which cannot model the dependency relationship between features, but has a high calculation speed and is friendly to a large dataset. hbos is that each dimension of the dataset is independent of the other, then each dimension is divided into bins (bins), the higher the density of the bins, the lower the corresponding degree of anomaly. hbos has the advantages of simplicity, low cost, parallel calculation, suitability for a large amount of data and good actual effect in actual use. However hbos has the disadvantage that the relationship between the different features cannot be taken into account.

Therefore, a copod model is also used in the embodiment, and the model jointly models all the features through a copula probability function, so that the relations among different features are fully considered. The model has better detection effect when carrying out abnormal detection on more data sets.

In addition, in the present embodiment, anomaly detection can be performed on detail features using an LSTM-based self-encoding model. On the other hand, intermediate features from the coding model, i.e. feature representations after dimension reduction, may be determined. Based on this intermediate feature, a kmeans cluster model can be constructed. Inputting the intermediate features into a kmeans clustering model to obtain the distance from each user to a clustering center as the basis of the user anomaly ranking. The method has the advantages that the effect of the kmeans clustering model in the multidimensional characteristic scene is poor, the dimension of the input of the kmeans clustering model can be reduced through the intermediate characteristic of the self-coding model, and the detection effect is improved. In addition, the powerful timing feature learning capability of the LSTM-based self-encoding model can be fully utilized.

Based on this, fig. 4A schematically illustrates a method schematic diagram of determining a first detection result according to an embodiment of the present disclosure.

As shown in fig. 4A, longitudinal contrast feature 410 may be input into isolated forest model 421, for example, to determine a first anomaly 431 of the target object, in accordance with an embodiment of the present disclosure. The longitudinal contrast feature 410 is input hbos to the model 422 to determine a second anomaly 432 of the target object. The longitudinal contrast feature 410 is input copod into the model 423 to determine a third anomaly 433 of the target object. Then, the first degree of abnormality 431, the second degree of abnormality 432, and the third degree of abnormality 433 may be determined as the first detection result.

Fig. 4B schematically illustrates a method of determining a second detection result according to an embodiment of the disclosure.

As shown in fig. 4B, according to an embodiment of the present disclosure, the lateral contrast feature 410' may be input into an isolated forest model 421', for example, to determine a fourth anomaly 431' of the target object. The lateral contrast feature 410 'is input hbos' to the model 422 'to determine a fifth anomaly 432' of the target object. The lateral contrast feature 410 'is input copod' to the model 423 'to determine a sixth anomaly 433' of the target object. Then, the fourth degree of abnormality 431', the fifth degree of abnormality 432', and the sixth degree of abnormality 433' are determined as the second detection result.

Fig. 4C schematically illustrates a method of determining a third detection result according to an embodiment of the disclosure.

As shown in fig. 4C, in accordance with an embodiment of the present disclosure, the detail features 440 may be input from the encoding model 451, for example, to determine a seventh anomaly 461 of the intermediate features and the target object. The intermediate features are input into the cluster model 452 to determine an eighth anomaly 462 of the target object. Then, the seventh abnormality degree 461 and the eighth abnormality degree 462 are determined as the third detection result.

Fig. 4D schematically illustrates a method of determining target anomaly according to an embodiment of the present disclosure.

As shown in fig. 4D, according to an embodiment of the present disclosure, for example, the first degree of abnormality 431, the second degree of abnormality 432, the third degree of abnormality 433, the fourth degree of abnormality 431', the fifth degree of abnormality 432', the sixth degree of abnormality 433', the seventh degree of abnormality 461, and the eighth degree of abnormality 462 may be normalized and normalized, respectively. Then, the normalized and normalized first degree of abnormality 431, second degree of abnormality 432, third degree of abnormality 433, fourth degree of abnormality 431', fifth degree of abnormality 432', sixth degree of abnormality 433', seventh degree of abnormality 461, and eighth degree of abnormality 462 are added to obtain target degree of abnormality 470.

Normalization may include, for example, Z-score processing, according to embodiments of the present disclosure. Normalization may be used, for example, to translate the degree of anomaly of each model output to between 0 and 1.

According to embodiments of the present disclosure, on the one hand, each different type of feature needs to be matched to a particular type of model to achieve the best results. For example, high-dimensional features are less effective if implemented using a clustering model. For example, the timing characteristics of the mindset are matched based on a self-coding model to obtain better effect, because the timing information can be better learned. On the other hand, each model has the advantages, and the detection results of a plurality of models are integrated, namely the advantages of a plurality of models are integrated, so that comprehensive detection can be realized. Finally, the detection results of the models can be mutually verified, so that the reliability of user abnormality is enhanced. For example, if multiple models consider the user as an abnormal user, the probability that the user is considered abnormal increases greatly.

The method of determining object anomaly shown above is further described with reference to FIG. 5 in connection with an exemplary embodiment. Those skilled in the art will appreciate that the following example embodiments are merely for the understanding of the present disclosure, and the present disclosure is not limited thereto.

In this embodiment, the object is an intranet user, which is hereinafter referred to as a user.

Fig. 5 schematically illustrates a method of determining object anomaly in accordance with an embodiment of the present disclosure.

In fig. 5, it is shown that data preparation 510 may be performed. According to the embodiment of the disclosure, the log of the surfing behavior of the intranet user can be collected. The internet surfing behavior log records what intranet web page is accessed by all intranet users at what time, whether the access is successful or not, cookie information, user-agent information and the like when the intranet web page is accessed. The web page access data may then be extracted from the log of online behavior. The web page access data may include a web page accessed by a user, whether each access is successful, user-agent information accessed each time, and the like. Illustratively, in this embodiment, the webpage access data of the intranet user of the last 14 days may be extracted from the internet surfing behavior log.

The web page access data may then be preprocessed 520. Some large companies may have large data volumes, and in order to achieve feature extraction faster and better, the original data may be simply and efficiently encoded in this embodiment. For example, the length of a string of a specific user-agent is 500, which occupies very much database space and also seriously affects feature extraction efficiency. Thus, the user quantity and pv (page view) quantity corresponding to all user-agents can be determined. Each user-agent is encoded in descending order of the pv amount, and the larger the pv amount is, the smaller the encoding is. The code can be stored in the form of a character string, and the smaller the code is, the smaller the bit number is, and the smaller the occupied storage space is. For example, the codes may be sequentially incremented from 1. Based on this, the character string of the user-agent with the largest number of occurrences has a length of 500, and can be encoded as 1, with a length of 1. It can be seen that the size of the space occupied after encoding is changed from the original 500 characters to 1 character. In an actual application scene, the number of user-agents in a company is at most tens of thousands, so that the occupied space after all user-agents are encoded is at most 5 characters.

In addition, in order to better evaluate the user anomaly, in this embodiment, various behavior features of the user in a non-working hour may be extracted. In an actual application scene, the working time of each company is not completely the same, and in order to better adapt to different situations of each company, a unique non-working hour of each company can be prepared according to the actual data situation of each company. Illustratively, in this embodiment, an inactive hour is considered if the user volume for an hour is less than 30% of the average user volume for each hour for the last 14 days.

The number of intranet pages of middle and small companies is generally about 100 or less, but the number of intranet pages of some large companies may exceed 100. In this embodiment, in order to handle the situation that the number of intranet web pages is large, it may be determined that the number of users is smaller than the threshold of user number or the number of accesses is smaller than the threshold of number of accesses, and these long-tail web pages are combined. The user quantity threshold and the access frequency threshold can be set according to actual needs respectively. For example, it may be determined that a web page with a user number of less than 5 or a number of accesses of less than 200 is a long-tail web page, and these long-tail web pages are combined into a class of web pages. The long-tail web page may be a web page which is abandoned by a company, but may also be some abnormal web pages, which can be used as a breakthrough point for detecting abnormal behaviors. The long-tail web pages are combined, not deleted directly, so that not only is the poor model effect caused by ultrahigh-dimension characteristic explosion and characteristic sparseness prevented, but also the abnormality degree can be evaluated more comprehensively.

Next, feature construction 530 may be performed. According to embodiments of the present disclosure, for example, multi-dimensional features may be used, including various types of features, including not only explicit, sequential, such as detail features 541, but also aggregate summary, such as lateral contrast features 542 and longitudinal contrast features 543. In addition, from a comparison point of view, in the present embodiment, the user anomaly degree can be evaluated using both the lateral, longitudinal, and other dimensions. Specifically, the vertical comparison may include, for example, whether there is a large deviation in the behavior characteristics between the user and other users each day, and if so, determining that the user is an abnormal user. The lateral comparison may include, for example, whether there is a large deviation between the behavior feature of the last week of the user and the behavior feature of the last week, and if so, determining that the user is an abnormal user. The specific judgment logic can be learned by the model according to the data distribution, and the degree of abnormality of the user is detected based on the model trained by learning so as to judge whether the user is an abnormal user or not.

According to embodiments of the present disclosure, the vertical contrast feature 543 may include, for example, the number of user-agents used by the intranet user per day, the number of accesses to each web page per day, the number of failures to access each web page per day, and the difference between the number of user-agents for the non-operating hours and the number of user-agents for the operating hours per day, etc. The lateral comparison feature 542 may include, for example, a difference between an average of the number of user-agents used by the intranet user in the last week and an average of the number of user-agents used in the previous week, a difference between an average number of accesses of each web page of the last week and an average number of accesses of corresponding web pages of the previous week, a difference between an average number of failed accesses of each web page of the last week and an average number of failed accesses of corresponding web pages of the previous week, a variation of a difference between a number of user-agents of the last week for non-operational hours and a number of user-agents of operational hours compared to a difference between a number of user-agents of the previous week for non-operational hours and a number of user-agents of operational hours, and the like. The details feature 541 may include, for example, the number of times the user accessed each web page every day for the last 7 days.

Then, the features obtained after data preprocessing and feature construction can be input into a network model. Illustratively, the network models in this embodiment may include an isolated forest model 551, hbos model 552, copod model 553, a self-encoding model 554, and a kmeans cluster model 555. Specifically, the trained isolated forest models 551, hbos, 552, and copod 553 may be used to detect anomalies with respect to the longitudinal contrast features 543 and the lateral contrast features 542, respectively, and output anomalies. Simultaneously, anomaly detection is performed on detail features 541 using a trained self-coding model 554 and kmeans cluster model 555, and anomalies are output.

Then, the degree of abnormality of the output of each model is normalized first, and then normalized. The normalization process may include, for example, a Z-score process. After normalization processing, the anomaly degree of anomaly detection output of all models for all users is between 0 and 1. Specifically, each user has 8 anomalies between 0 and 1, which are the anomalies output by three models, i.e., an isolated forest, hbos, copod, etc., modeled based on the transverse contrast feature 542, the anomalies of three models, i.e., an isolated forest, hbos, copod, etc., modeled based on the longitudinal contrast feature 543, and the anomalies of two models, i.e., self-encoding, kmeans clustering, etc., modeled based on the detail time sequence class feature 541.

Then, the abnormality degrees after 8 model outputs and normalization can be added to obtain the final abnormality degree of the user. Next, the users may be ranked in order of the degree of abnormality from large to small, and the first n users may be determined as abnormal users. Wherein n can be set according to actual requirements. For example, the first 5% of users may be produced as outliers.

The apparatus for determining object abnormality provided in the present disclosure will be described below with reference to fig. 6.

Fig. 6 schematically illustrates a block diagram of an apparatus for determining object anomaly in accordance with an embodiment of the present disclosure.

As shown in fig. 6, the apparatus 600 for determining the object anomaly degree includes a first determining module 610, a second determining module 620, a third determining module 630, and a detecting module 640.

A first determining module 610 is configured to determine detail features according to web page access data of the target object.

The second determining module 620 is configured to determine the longitudinal contrast feature according to the web page access data of the target object and the web page access data of the other objects.

The third determining module 630 is configured to determine the transverse contrast feature according to the web page access data in the current time period of the target object and the web page access data in the previous time period of the target object.

The detection module 640 is configured to perform anomaly detection on the longitudinal contrast feature, the transverse contrast feature, and the detail feature, so as to obtain a target anomaly degree of the target object.

According to embodiments of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.

Fig. 7 schematically illustrates a block diagram of an example electronic device 700 that may be used to implement embodiments of the present disclosure. Electronic devices are intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The electronic device may also represent various forms of mobile devices, such as personal digital processing, cellular telephones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions, are meant to be exemplary only, and are not meant to limit implementations of the disclosure described and/or claimed herein.

As shown in fig. 7, the apparatus 700 includes a computing unit 701 that can perform various appropriate actions and processes according to a computer program stored in a Read Only Memory (ROM) 702 or a computer program loaded from a storage unit 708 into a Random Access Memory (RAM) 703. In the RAM 703, various programs and data required for the operation of the device 700 may also be stored. The computing unit 701, the ROM 702, and the RAM 703 are connected to each other through a bus 704. An input/output (I/O) interface 705 is also connected to bus 704.

Various components in the device 700 are connected to the I/O interface 705, including an input unit 706, e.g., keyboard, mouse, etc., an output unit 707, e.g., various types of displays, speakers, etc., a storage unit 708, e.g., magnetic disk, optical disk, etc., and a communication unit 709, e.g., network card, modem, wireless communication transceiver, etc. The communication unit 709 allows the device 700 to exchange information/data with other devices via a computer network, such as the internet, and/or various telecommunication networks.

The computing unit 701 may be a variety of general and/or special purpose processing components having processing and computing capabilities. Some examples of computing unit 701 include, but are not limited to, a Central Processing Unit (CPU), a Graphics Processing Unit (GPU), various specialized Artificial Intelligence (AI) computing chips, various computing units running machine learning model algorithms, a Digital Signal Processor (DSP), and any suitable processor, controller, microcontroller, etc. The calculation unit 701 performs the respective methods and processes described above, for example, a method of determining the degree of abnormality of an object. For example, in some embodiments, the method of determining object anomaly may be implemented as a computer software program tangibly embodied on a machine-readable medium, such as storage unit 708. In some embodiments, part or all of the computer program may be loaded and/or installed onto device 700 via ROM 702 and/or communication unit 709. When the computer program is loaded into the RAM 703 and executed by the computing unit 701, one or more steps of the above-described method of determining object anomaly may be performed. Alternatively, in other embodiments, the computing unit 701 may be configured to perform the method of determining object anomaly by any other suitable means (e.g., by means of firmware).

Various implementations of the systems and techniques described here above can be implemented in digital electronic circuitry, integrated circuit systems, field Programmable Gate Arrays (FPGAs), application Specific Integrated Circuits (ASICs), application Specific Standard Products (ASSPs), systems On Chip (SOCs), complex Programmable Logic Devices (CPLDs), computer hardware, firmware, software, and/or combinations thereof. These various embodiments may include being implemented in one or more computer programs that are executable and/or interpretable on a programmable system including at least one programmable processor, which may be a special or general purpose programmable processor, operable to receive data and instructions from, and to transmit data and instructions to, a storage system, at least one input device, and at least one output device.

Program code for carrying out methods of the present disclosure may be written in any combination of one or more programming languages. These program code may be provided to a processor or controller of a general purpose computer, special purpose computer, or other programmable data processing apparatus such that the program code, when executed by the processor or controller, causes the functions/operations specified in the flowchart and/or block diagram to be implemented. The program code may execute entirely on the machine, partly on the machine, as a stand-alone software package, partly on the machine and partly on a remote machine or entirely on the remote machine or server.

In the context of this disclosure, a machine-readable medium may be a tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. The machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of a machine-readable storage medium would include an electrical connection based on one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

To provide for interaction with a user, the systems and techniques described here can be implemented on a computer having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user and a keyboard and a pointing device (e.g., a mouse or a trackball) by which the user can provide input to the computer. Other kinds of devices may also be used to provide for interaction with a user, for example, feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback), and input from the user may be received in any form, including acoustic input, speech input, or tactile input.

The systems and techniques described here can be implemented in a computing system that includes a background component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front-end component (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such background, middleware, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a Local Area Network (LAN), a Wide Area Network (WAN), and the Internet.

The computer system may include a client and a server. The client and server are typically remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other.

It should be appreciated that various forms of the flows shown above may be used to reorder, add, or delete steps. For example, the steps recited in the present disclosure may be performed in parallel or sequentially or in a different order, provided that the desired results of the technical solutions of the present disclosure are achieved, and are not limited herein.

The above detailed description should not be taken as limiting the scope of the present disclosure. It will be apparent to those skilled in the art that various modifications, combinations, sub-combinations and alternatives are possible, depending on design requirements and other factors. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present disclosure are intended to be included within the scope of the present disclosure.

Claims

1. A method of determining object anomaly, comprising:

determining detail characteristics according to webpage access data of the target object;

Determining longitudinal contrast characteristics according to the webpage access data of the target object and the webpage access data of other objects;

Determining transverse contrast characteristics according to the webpage access data in the current time period of the target object and the webpage access data in the last time period of the target object, and

Performing anomaly detection on the longitudinal comparison feature, the transverse comparison feature and the detail feature to obtain target anomaly degree of the target object;

The web page access data comprises user agent information, long tail web pages, and differences between the number of user agent information of non-working time and the number of user agent information of working time in unit time, wherein the method further comprises the following steps:

Determining page browsing amounts corresponding to the user agent amounts, and encoding the user agent information according to descending order of the page browsing amounts to obtain encoded user agent information;

Determining the reference unit time as a non-working period in response to the user quantity within the reference unit time being less than a predetermined user quantity;

And determining at least one webpage with the user quantity smaller than a user quantity threshold or the access frequency smaller than an access frequency threshold as a long-tail webpage, and combining at least one long-tail webpage to obtain the combined long-tail webpage.

2. The method according to claim 1, wherein the anomaly detection of the longitudinal contrast feature, the transverse contrast feature and the detail feature to obtain the anomaly degree of the target object comprises:

performing anomaly detection on the longitudinal comparison characteristic to obtain a first detection result;

performing anomaly detection on the transverse comparison characteristic to obtain a second detection result;

Performing abnormality detection on the detail characteristics to obtain a third detection result, and

And determining the target anomaly degree of the target object according to the first detection result, the second detection result and the third detection result.

3. The method according to claim 2, wherein the performing anomaly detection on the longitudinal alignment feature to obtain a first detection result includes:

inputting the longitudinal contrast features into an isolated forest model to determine a first anomaly of the target object;

inputting hbos the longitudinal contrast features into a model to determine a second anomaly of the target object;

Inputting copod the longitudinal contrast feature into a model to determine a third anomaly of the target object, and

Determining the first abnormality degree, the second abnormality degree, and the third abnormality degree as the first detection result.

4. A method according to claim 3, wherein said performing anomaly detection on said lateral contrast feature to obtain a second detection result comprises:

inputting the transverse contrast features into an isolated forest model to determine a fourth anomaly of the target object;

inputting hbos the lateral contrast features into a model to determine a fifth anomaly of the target object;

inputting copod the lateral contrast feature into a model to determine a sixth anomaly of the target object, and

Determining the fourth abnormality degree, the fifth abnormality degree, and the sixth abnormality degree as the second detection result.

5. The method according to claim 4, wherein the performing anomaly detection on the detail features to obtain a third detection result includes:

Inputting the detail features from the coding model to determine a seventh anomaly of the intermediate features and the target object;

Inputting the intermediate features into a clustering model to determine an eighth anomaly of the target object, and

And determining the seventh abnormality degree and the eighth abnormality degree as the third detection result.

6. The method of claim 5, wherein the determining the target anomaly of the target object based on the first detection result, the second detection result, and the third detection result comprises:

The first degree of abnormality, the second degree of abnormality the third degree of abnormality, the fourth degree of abnormality normalizing and normalizing the fifth anomaly degree, the sixth anomaly degree, the seventh anomaly degree, and the eighth anomaly degree; and

And adding the normalized first degree of abnormality, the normalized second degree of abnormality, the normalized third degree of abnormality, the normalized fourth degree of abnormality, the normalized fifth degree of abnormality, the normalized sixth degree of abnormality, the normalized seventh degree of abnormality and the normalized eighth degree of abnormality to obtain a target degree of abnormality.

7. The method of claim 1, wherein the web page access data further comprises at least one of an amount of user agent information used per unit time of the target object, web pages accessed per unit time of the target object, a number of accesses per web page, and a total number of failures to access per web page per unit time of the target object.

8. An apparatus for determining object anomaly, comprising:

the first determining module is used for determining detail characteristics according to webpage access data of the target object;

the second determining module is used for determining longitudinal contrast characteristics according to the webpage access data of the target object and the webpage access data of other objects;

A third determining module for determining a transverse contrast characteristic according to the web page access data in the current time period of the target object and the web page access data in the last time period of the target object, and

The detection module is used for carrying out anomaly detection on the longitudinal comparison feature, the transverse comparison feature and the detail feature to obtain the target anomaly degree of the target object;

wherein the web page access data comprises user agent information, a long tail web page, and a difference value between the number of user agent information of non-working time and the number of user agent information of working time in unit time, and the device further comprises:

The user agent information processing module is used for determining page browsing amounts corresponding to the user agent amounts respectively, and encoding the user agent information according to descending order of the page browsing amounts to obtain encoded user agent information;

A non-working time period determining module, configured to determine that a reference unit time is a non-working time period in response to a user quantity within the reference unit time being less than a predetermined user quantity;

and the long-tail webpage processing module is used for determining at least one webpage with the user quantity smaller than the user quantity threshold or the access frequency smaller than the access frequency threshold as a long-tail webpage, and carrying out merging processing on at least one long-tail webpage to obtain the long-tail webpage after merging processing.

9. An electronic device, comprising:

At least one processor, and

A memory communicatively coupled to the at least one processor, wherein,

The memory stores instructions executable by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-7.

10. A non-transitory computer readable storage medium storing computer instructions for causing a computer to perform the method of any one of claims 1-7.

11. A computer program product comprising computer programs/instructions which, when executed by a processor, implement the steps of the method of any of claims 1-7.