CN117708079A

CN117708079A - Asset data processing method and device, electronic equipment and storage medium

Info

Publication number: CN117708079A
Application number: CN202311825857.2A
Authority: CN
Inventors: 李绣雯; 郭超; 肖超雄; 蔡玉光; 王雷; 祁万红; 何瑞; 曹博新; 崔强; 覃征平; 田子玉; 孙军亮
Original assignee: China Electronics Industry Engineering Co ltd; Secworld Information Technology Beijing Co Ltd
Current assignee: China Electronics Industry Engineering Co ltd; Secworld Information Technology Beijing Co Ltd
Priority date: 2023-12-27
Filing date: 2023-12-27
Publication date: 2024-03-15

Abstract

The embodiment of the application relates to the field of network security, and provides an asset data processing method, device, electronic equipment and storage medium.

Description

Asset data processing method and device, electronic equipment and storage medium

Technical Field

The embodiment of the application relates to the field of network security, in particular to an asset data processing method, an asset data processing device, electronic equipment and a storage medium.

Background

With the development and popularization of the internet, network security is an important issue of great concern, and network security events also seriously affect information security and privacy protection of people, so that accurate identification of assets from network traffic is particularly important, and the network security event management system can help organizations to better manage assets, identify security risks, improve security protection capability, timely cope with security events and guarantee safe and stable operation of the network.

Meanwhile, in practice, besides accurately identifying the resources from the network traffic, the security analyst wants to know the information of the real responsible person generating the network traffic, so that the operations such as further research, judgment, analysis and tracing are convenient to be performed on the data, for example, the specific responsible person can be tracked timely and accurately once the network security problems such as network attack occur.

Disclosure of Invention

The embodiment of the application aims to provide an asset data processing method, an asset data processing device, electronic equipment and a storage medium, which can accurately identify an asset from network traffic and locate a responsible person through the asset.

In order to achieve the above purpose, the technical solution adopted in the embodiment of the present application is as follows:

in a first aspect, an embodiment of the present application provides an asset data processing method, applied to an electronic device, where the electronic device communicates with an ES cluster, the method includes:

acquiring a Radius protocol log in real time;

extracting AAA assets from the Radius protocol log based on a pre-configured target configuration file for asset identification;

and acquiring target CRM data matched with the AAA asset from all Customer Relationship Management (CRM) data which are imported into the ES cluster in advance, and merging the target CRM data into the AAA asset.

Optionally, the target configuration file includes a plurality of target fields, and the Radius protocol log includes a field value corresponding to each of the target fields;

the step of extracting AAA assets from the Radius protocol log based on the preconfigured target profile for asset identification, comprises:

extracting a field value corresponding to each target field from the Radius protocol log based on the target fields;

and formatting the field value corresponding to each target field into a dictionary type to obtain the AAA asset, wherein the AAA asset comprises the target fields and the formatted field value corresponding to each target field.

Optionally, the AAA asset includes an encapsulation protocol field, a username field, a call site ID field, and respective corresponding field values, and the CRM data includes username information and call site ID information;

the step of obtaining target CRM data matched with the AAA asset from all customer relationship management CRM data pre-imported into the ES cluster comprises the following steps:

determining the encapsulation type corresponding to the Radius protocol log according to the field value corresponding to the encapsulation protocol field;

If the encapsulation type is a fixed network type, acquiring target CRM data with the same field value of the user name information and the user name field from all CRM data;

and if the encapsulation type is a mobile network type, acquiring target CRM data with the same field value of the calling site ID information and the calling site ID field from all the CRM data.

Optionally, the electronic device is also in communication with all of the S3 cluster, the kafka cluster, and the nokia platform;

before the step of acquiring the Radius protocol log in real time, the method further comprises the step of importing CRM data into the ES cluster, which comprises the following steps:

acquiring a file to be imported and file name and file attribute information of the file to be imported, wherein the file to be imported comprises a plurality of pieces of CRM data to be imported;

judging whether the file to be imported has repeated import or exists in the S3 cluster according to the file name;

when the fact that the file to be imported does not exist in repeated import and does not exist in the S3 cluster is determined, carrying out file qualification checking on the file to be imported based on the file attribute information;

after confirming that the file qualification verification is passed, calling an uploading file interface of the S3 cluster, and storing the file to be imported into the S3 cluster;

Invoking a download file interface of the S3 cluster, and downloading the CRM data to be imported from the S3 cluster one by one and performing field qualification check to obtain each piece of CRM data passing the field qualification check;

writing all the CRM data passing the field qualification tests into the kafka cluster, and calling the Noah platform to transfer all the CRM data passing the field qualification tests from the kafka cluster to the ES cluster.

Optionally, the electronic device is also in communication with a Redis cluster;

the step of importing CRM data into the ES cluster further includes:

extracting field values corresponding to each key field from the CRM data according to a plurality of predefined key fields for each piece of CRM data, obtaining a matching file corresponding to the CRM data and storing the matching file into the Redis cluster, wherein the matching file comprises the plurality of key fields and the field values corresponding to each key field;

the step of obtaining target CRM data matched with the AAA asset from all customer relationship management CRM data pre-imported into the ES cluster and merging the target CRM data into the AAA asset comprises the following steps:

Determining a target matching file matched with the AAA asset from all matching files stored in the Redis cluster;

and acquiring target CRM data corresponding to the target matching file from all CRM data, and merging the target CRM data to the AAA asset.

Optionally, the AAA asset includes an encapsulation protocol field, a username field, a call site ID field, and respective corresponding field values, and the matching file includes a username information field, a call site ID information field, and respective corresponding field values;

the step of determining a target matching file matched with the AAA asset from all matching files stored in the dis cluster includes:

if the encapsulation type is a fixed network type, acquiring target matching files with the field values corresponding to the user name information fields being the same as the field values corresponding to the user name fields from all the matching files;

and if the encapsulation type is a mobile network type, acquiring target matching files, of which the field values corresponding to the calling site ID information fields are the same as the field values corresponding to the calling site ID fields, from all the matching files.

Optionally, the step of determining the encapsulation type corresponding to the Radius protocol log according to the field value corresponding to the encapsulation protocol field includes:

if the field value corresponding to the encapsulation protocol field is a first set value, determining that the encapsulation type is a fixed network type;

and if the field value corresponding to the encapsulation protocol field is a second set value, determining that the encapsulation type is a mobile network type.

Optionally, the electronic device is further in communication with an ES cluster, a Redis cluster, and a nokia platform, where the ES cluster stores all Radius protocol logs, and the Radius protocol logs include IP addresses; the Redis cluster stores all AAA assets within a set time period, wherein the AAA assets comprise asset IP addresses and the target CRM data;

the method further comprises the steps of:

acquiring all AAA assets stored in the Redis cluster according to a preset time interval;

for each AAA asset, determining each target Radius protocol log with the same IP address as the asset IP address of the AAA asset from all Radius protocol logs stored in the ES cluster;

invoking the Noah platform to enrich the target CRM data in the AAA assets into each of the target Radius protocol logs stored by the ES cluster.

In a second aspect, an embodiment of the present application further provides an asset data processing apparatus, which is applied to an electronic device, where the electronic device communicates with an ES cluster, and the apparatus includes:

the journal acquisition module is used for acquiring the Radius protocol journal in real time;

the asset extraction module is used for extracting AAA assets from the Radius protocol log based on a pre-configured target configuration file for asset identification;

and the merging module is used for acquiring target CRM data matched with the AAA asset from all customer relationship management CRM data which are imported into the ES cluster in advance and merging the target CRM data into the AAA asset.

In a third aspect, an embodiment of the present application further provides an electronic device, including a processor and a memory, where the memory is configured to store a program, and the processor is configured to implement the asset data processing method in the first aspect when executing the program.

In a fourth aspect, embodiments of the present application also provide a computer readable storage medium having stored thereon a computer program which, when executed by a processor, implements the asset data processing method of the first aspect described above.

Compared with the prior art, the asset data processing method, device, electronic equipment and storage medium provided by the embodiment of the application are characterized in that the Radius protocol log is obtained in real time, and the AAA asset is extracted from the Radius protocol log based on the pre-configured target configuration file for asset identification, so that the AAA asset of network traffic can be accurately identified through the Radius protocol log because the Radius protocol log usually records user authentication, authorization and accounting information, meanwhile, target CRM data matched with the AAA asset is obtained from all CRM data pre-imported into an ES cluster, and the target CRM data is merged into the AAA asset, and because the CRM data usually comprises various user information, the target CRM data matched with the AAA asset is merged into the AAA asset, so that a specific responsible person can be positioned through the AAA asset.

Drawings

Fig. 1 shows a schematic diagram of an application architecture according to an embodiment of the present application.

Fig. 2 shows a schematic diagram two of an application architecture according to an embodiment of the present application.

Fig. 3 shows a schematic flow chart of an asset data processing method according to an embodiment of the present application.

Fig. 4 shows a schematic diagram of a target field provided in an embodiment of the present application.

Fig. 5 shows a second flowchart of an asset data processing method according to an embodiment of the present application.

Fig. 6 shows a schematic diagram of a Radius protocol log according to an embodiment of the present application.

Fig. 7 shows a schematic diagram ii of a Radius protocol log according to an embodiment of the present application.

Fig. 8 shows an interface schematic diagram after AAA asset warehousing according to an embodiment of the present application.

Fig. 9 shows a third flowchart of an asset data processing method according to an embodiment of the present application.

Fig. 10 shows a flowchart of a method for processing asset data according to an embodiment of the present application.

Fig. 11 is a schematic diagram illustrating a CRM data import flow provided in an embodiment of the present application.

Fig. 12 shows an interface schematic after CRM data importing is completed according to an embodiment of the present application.

Fig. 13 shows a fifth flowchart of an asset data processing method according to an embodiment of the present application.

Fig. 14 is a schematic diagram of an interface after enrichment of a Radius protocol log according to an embodiment of the present application.

Fig. 15 shows a block schematic diagram of an asset data processing device according to an embodiment of the present application.

Fig. 16 shows a block schematic diagram of an electronic device according to an embodiment of the present application.

Icon: 100-asset data processing means; 101-configuring a module; 102-a CRM data importing module; 103-a log acquisition module; 104-an asset extraction module; 105-a data merging module; 106, a data enrichment module; 10-an electronic device; 11-a processor; 12-memory; 13-bus.

Detailed Description

The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application.

With the rapid development and popularization of the internet, network security has been an important issue of great concern, and network security events have also seriously affected people's information security and privacy protection.

Asset identification is an important link in network security, and mainly provides support for subsequent security management and attack detection by identifying assets such as devices, application programs, operating systems and the like in the network. Current methods for asset identification mainly include Active Scanning (Active Scanning), passive listening (Passive Listening), and network traffic analysis (Network Traffic Analysis).

The active scanning is to scan the network by using an automation tool, discover and identify devices and hosts connected with the network, and can be realized by sending network requests, detecting open ports, analyzing protocols and the like, and the active scanning can acquire more comprehensive asset information, but certain flow and load are required to be introduced into the network.

Passive interception is the observation and analysis of device and host information in communication by sniffing network traffic, which can detect packets transmitted through the network and extract key information therefrom, such as IP addresses, MAC addresses, operating system fingerprints, etc., which do not actively interfere with the network, but may not be able to obtain all asset information.

Network traffic analysis is the discovery and identification of devices and hosts connected to a network by monitoring and analyzing network traffic. Network traffic analysis may obtain information about network topology and assets by analyzing traffic patterns, sources and destinations, etc., and may also be used to detect abnormal activity and potential threats.

Therefore, it is important to identify accurate assets from network traffic, which can help organizations to better manage assets, identify security risks, improve security protection capability, and timely cope with security events, so as to ensure safe and stable operation of the network.

Based on this, the embodiment of the application provides an asset data processing method, on one hand, by acquiring a Radius protocol log in real time and extracting an AAA asset from the Radius protocol log based on a pre-configured target configuration file for asset identification, because the Radius protocol log usually records user authentication, authorization and accounting information, the AAA asset of network traffic can be accurately identified through the Radius protocol log, on the other hand, target CRM data matched with the AAA asset is acquired from all CRM data pre-imported into an ES cluster, and the target CRM data is merged into the AAA asset, and because the CRM data usually comprises various user information, the target CRM data matched with the AAA asset is merged into the AAA asset, so that a specific responsible person can be located through the AAA asset.

The following detailed description refers to the accompanying drawings.

The asset data processing method provided by the embodiment of the application is applied to electronic equipment. The electronic device may be a server, e.g., a single server, a cluster of servers, etc., or a terminal, e.g., a desktop computer, a notebook computer, etc. The embodiments of the present application do not impose any limitation on this.

Referring to fig. 1, fig. 1 shows a schematic diagram of an application architecture of an asset data processing method according to an embodiment of the present application, where an electronic device communicates with a sensor, a GP (General Purpose), a dis cluster, an ES cluster, an S3 cluster, a kafka cluster, and a nokia platform.

The sensor is used for collecting a Radius protocol log from the Radius server in real time and sending the Radius protocol log to the electronic equipment, so that the electronic equipment extracts the AAA asset from the Radius protocol log based on a pre-configured target configuration file for asset identification.

The sensor may be a hardware device for monitoring network traffic, such as a network traffic analyzer, intrusion detection system, etc., which may capture network traffic by way of a mirrored port, port listening, etc., and provide the functionality of monitoring and analyzing logs generated by the Radius server. The sensor can also be software installed on the Radius server, the software has the functions of network traffic monitoring and log capturing, and the log generated by the Radius server can be monitored in real time.

The GP is used to store data that needs to be shown through a web page, for example, file import status of a file to be imported, page data after the CRM (Customer Relationship Management ) import is completed, interface data after AAA asset warehousing, interface data after Radius protocol log enrichment, and the like.

Redis clusters are used to store data that needs to be accessed and processed quickly, e.g., matching files for CRM data that matches the AAA assets quickly, AAA assets for enrichment into Radius protocol logs, etc.

The ES cluster is used to store and index large-scale text data, such as imported CRM data, AAA assets incorporating target CRM data, sensor-collected Radius protocol logs, logs other than Radius protocol logs, and the like.

The S3 cluster is used to store large-scale data and static files, for example, files to be imported, etc.

The kafka cluster acts as a middleware to provide data transfer and message queues, allowing asynchronous communication between different application systems, from which a producer can publish data to the kafka's message queues (called topics) from which consumers can subscribe to data and process.

For example, the sensor sends a Radius protocol log to one topic of the kafka cluster, the electronic device consumes the Radius protocol log from the topic, extracts the AAA asset from the Radius protocol log and then places the AAA asset into another topic of the kafka cluster, then the electronic device again consumes the AAA asset from the topic to match with the CRM data, and after matching to the target CRM data, the target CRM data is merged to the AAA asset and then stored in the ES cluster. In another example, in the CRM data import process, the electronic device stores CRM data downloaded from the S3 cluster and passing the field eligibility check in topic of the kafka cluster, and then uses the nokia platform to consume and write the CRM data from the topic into the ES cluster.

The Norian platform is used to store various text data to the ES cluster and enrich specific data to the ES cluster, e.g., store imported completed CRM data to the ES cluster, store AAA assets incorporating target CRM data to the ES cluster, enrich AAA assets to the Radius protocol log stored by the ES cluster, etc.

In other words, the embodiment of the application builds an analysis platform to realize the asset data processing method provided by the embodiment of the application. Referring to fig. 2, the analysis platform includes 5 levels of data access, middleware, storage layer, service layer and application layer.

The data access comprises a Noah platform and a sensor, wherein the sensor is used for collecting a Radius protocol log from a Radius server in real time, and the Noah platform is used for realizing data storage and data enrichment of the ES cluster.

The middleware includes a kafka cluster for implementing asynchronous communications between different application systems.

The storage layer comprises GP, redis cluster, ES cluster and S3 cluster, and is used for storing different types of data.

The service layer comprises web services and AAA asset services, wherein the web services are used for realizing web page display of different data, such as page display after CRM data is imported, interface display after AAA asset storage, interface display after Radius protocol log enrichment and the like.

The AAA asset services include asset enrichment services, asset matching CRM information, CRM import services, and Radius log extraction asset services, which are core content of the asset data processing method provided by the embodiments of the present application. Wherein the asset enrichment service refers to enriching AAA assets into a Radius protocol log. Asset matching CRM information refers to matching AAA assets with CRM data and, after matching to target CRM data, merging the target CRM data to the AAA assets. The CRM import service refers to importing CRM data into an ES cluster. Radius log extract asset service refers to extracting AAA assets from a Radius protocol log.

The application layer includes AAA assets where the AAA assets incorporate target CRM data. The user can retrieve the AAA asset of any one network traffic through the application layer and locate the real responsible person who generated that network traffic through the AAA asset.

Referring to fig. 3 on the basis of the application architecture shown in fig. 1 and fig. 2, fig. 3 shows a flow chart of an asset data processing method according to an embodiment of the present application, where the asset data processing method is applied to the electronic device in fig. 1, and may include the following steps:

s103, acquiring a Radius protocol log in real time.

In this embodiment, the Radius protocol log is collected by the sensor from the Radius server in real time and sent to the analysis platform. The Radius server is generally used for authenticating a user and controlling the access of the user to the network resource according to the identity and authority of the user, and when the identity of the user and the access authority of the authorized user are verified, the Radius server generates corresponding logs, wherein the logs comprise authentication information, access request, authorization result and the like of the user, namely a Radius protocol log.

In this embodiment, the sensor collects the Radius protocol log from the Radius server in real time and sends the Radius protocol log to the analysis platform, and the analysis platform stores the Radius protocol log in the kafka cluster after receiving the Radius protocol log, for example, the Radius is the ty_radius, the group is the radius_asset, and the data in the Radius is the data source for extracting the AAA asset later.

S104, extracting AAA assets from the Radius protocol log based on the pre-configured target configuration file for asset identification.

In this embodiment, AAA assets are commonly referred to as Authentication, authorization, and Accounting (Accounting) assets, which are key information used to manage user Authentication, authorize access, and record user activity.

Wherein the authentication asset includes user authentication information, such as a user name, password, digital certificate, biometric feature, etc., for confirming the identity of the user, ensuring that only authorized users can access the system or network resource. The authorized assets include information of the rights and resources to which the user is granted access, rules and restrictions to be followed in determining which resources the user can access and which operations to perform, and in accessing those resources. Accounting assets include information that records user activity and access resources, such as login time, accessed resources, operational records, etc., for monitoring user activity, auditing, and security event responses.

Since the Radius protocol log typically records user authentication, authorization, and accounting information in a particular format, AAA assets can be extracted from the Radius protocol log by parsing the Radius protocol log and based on a pre-configured target profile for asset identification.

The process of configuring the target configuration file on the analysis platform is described first, and includes steps S1 to S2.

S1, configuring a target field required to be used as an asset identification in a Radius protocol log into an initial configuration file (for example, a sensor_log.proto file) of an analysis platform. For example, referring to fig. 4, the target field may include:

1. serial_num: a serial number identifying a unique serial number of the Radius protocol log;

2. access_time: access time, which refers to the timestamp of the Radius protocol log record;

3. sip: a source IP address;

4. sipv6: a source IPv6 address;

5. sport: a source port number;

6. dip: a destination IP address;

7. dipv6: a destination IPv6 address;

8. dport: a destination port number;

9. src_mac: a source MAC address;

10. dst_mac: a destination MAC address;

11. set_key: a session key for identifying a unique key for a network session;

12. vendor_id: vendor ID, which refers to the vendor ID of a device or application;

13. device_ip: the IP address of the equipment refers to the IP address of the equipment;

14. code: code, refers to a particular code or state;

15. user_name: a user name;

16. framed_protocol: encapsulation protocol, which refers to the type of encapsulation protocol;

17. frame_ip_address: encapsulating the IP address;

18. rolling_station_id: a call site ID;

19. acid_session_id: a charging session ID identifying a unique ID of the charging session;

20. acct_status_type: charging status type.

It should be noted that the target field shown in fig. 1 is only an example, and in practice, the target field may be flexibly configured according to service requirements, so long as the name of the target field is ensured to be consistent with the name of the related field in the Radius protocol log.

S2, instantiating the initial configuration file to generate a target configuration file.

For example, protoc-version is performed; the protocol_out=. Sensor_log.pro command, i.e. the version number of the Protocol buffer compiler is queried and the compiler is used to compile the sensor_log.pro file into Python code, the content of the sensor_log.pro file is updated into the sensor_log_pb2.py file, which is a data parsing file, i.e. the target configuration file, which is used for asset identification.

Thus, based on the target configuration files configured in steps S1 to S2, referring to fig. 5, the process of extracting AAA assets from the Radius protocol log may include sub-steps S1041 to S1042.

S1041, extracting a field value corresponding to each target field from the Radius protocol log based on the target fields.

S1042, format the field value corresponding to each target field into dictionary type to obtain AAA asset, which includes multiple target fields and the formatted field value corresponding to each target field.

In this embodiment, as known from step S193, the Radius protocol log is stored in the kafka cluster, for example, the topic is ty_radius, and the group is radius_asset, so the Radius protocol log is consumed from the topic.

Meanwhile, since the Radius protocol log records user authentication, authorization and accounting information in a specific format, the Radius protocol log needs to be parsed first, and then a field value corresponding to each target field is extracted from the parsed Radius protocol log to be formatted into a dictionary type, so as to obtain an AAA asset comprising a plurality of target fields and formatted field values corresponding to each target field.

In the process of consuming the Radius protocol LOG, calling the generated SENSOR_LOG instantiation object in the sensor_log_pb2.Py, taking the read Radius protocol LOG as a parameter to be input, and then calling the ParseFromString () method in the class to map to a field value corresponding to the target field in the Radius protocol LOG according to the configured target field, namely extracting the field value corresponding to each target field from the Radius protocol LOG; the mapped data is then formatted into a dictionary type by the pbjson.pb2json () method, i.e., the field value corresponding to each target field is formatted into a dictionary type, the formatted field value corresponding to the target field is stored in the dictionary type data, and the formatted field value is written into the kafka cluster, e.g., topic is ty_radius_asset.

Optionally, the field value corresponding to a portion of the target fields in the Radius protocol log is described as shown in table 1 below:

table 1 description of field values corresponding to partial target fields in Radius protocol log

Meanwhile, for the field values corresponding to the target fields in the Radius protocol log, the field values are different, and represent different meanings, for example, a frame_protocol field, when the field value of the field is 1, the Radius protocol log is generated by a fixed network; when the field value of the field is 7, it indicates that the Radius protocol log is mobile network generated.

For example, referring to fig. 6, fig. 6 is a schematic diagram of a Radius protocol log generated by a mobile network. Meanwhile, in the process of collecting the Radius protocol log, the sensor indicates the target field by adopting specific numbers, for example, 4 numbers of AVP Type 1, AVP Type 7, AVP Type 8 and AVP Type 31 are respectively 4 fields of user_name, frame_ protocol, framed _ip_address and grouping_station_id, and the value modes of the field values corresponding to different target fields are different, for example, the value of AVP Type 1 is a character string, the value of AVP Type 7 is an integer enumeration value, the value of AVP Type 8 is a character string ip address, and the value of AVP Type 31 is a character string.

As can be seen from FIG. 6, there is no AVP Type 1, and the field value corresponding to the user_name is obtained as a null value; the AVP Type 7, the AVP Type 8 and the AVP Type 31 are respectively obtained, the field value corresponding to the frame_protocol is 7, the field value corresponding to the frame_ip_address is 10.223.202.59, and the field value corresponding to the rolling_station_id is 96894691591.

For another example, referring to fig. 7, fig. 7 is a schematic diagram of a Radius protocol log generated by a mobile network. As can be seen from fig. 7, there are AVP Type 1, AVP Type 7, AVP Type 8 and AVP Type 31, and the corresponding obtained field value is GGFGG@bait.ooredoo.om for the user_name, 1 for the frame_protocol, 145.255.91.229 for the frame_ip_address, and e0:00:84:28:80:ce for the frame_station_id.

S105, obtaining target CRM data matched with the AAA asset from all customer relationship management CRM data of the pre-imported ES cluster, and merging the target CRM data into the AAA asset.

In this embodiment, the CRM data is relevant user information stored in the CRM system, generally including, but not limited to: basic information such as name, gender, age, identification number, contact (phone number, mailbox, etc.), address, etc.; account information, such as account number, account type, account status, account balance, etc.; activity records, such as transaction records, purchase history, service requests, complaint records, communication history, and the like; social media information, e.g., activity, comments, feedback, etc., on the social media platform.

Obviously, since CRM data generally includes various user information, acquiring target CRM data matching AAA assets from all CRM data pre-imported into the ES cluster and incorporating the target CRM data into the AAA assets enables locating specific responsible persons through the AAA assets, i.e., locating the actual responsible persons who generated the Radius protocol log corresponding to the AAA assets.

As can be seen from the foregoing step S104, the AAA asset including the plurality of target fields and the formatted field value corresponding to each target field is stored in the kafka cluster, for example, the topic is the ty_radius_asset. Therefore, the AAA asset in the topic can be read and matched with all CRM data of the pre-imported ES cluster, the target CRM data matched with the AAA asset is found and combined with the AAA asset, and then the AAA asset combined with the target CRM data is stored in the ES cluster, namely, the warehousing of the AAA asset is completed.

In one possible implementation, the AAA asset includes an encapsulation protocol field, a user name field, a call site ID field, and respective corresponding field values, i.e., a frame_ protocol, user _name, a rolling_station_id field, and respective corresponding field values. The CRM data includes user name information and call site ID information.

Thus, referring again to FIG. 5, the process of obtaining target CRM data matching AAA assets from all customer relationship management CRM data pre-imported into the ES cluster may include sub-steps S1051-S1053.

S1051, determining the encapsulation type corresponding to the Radius protocol log according to the field value corresponding to the encapsulation protocol field.

S1052, if the package type is the fixed network type, acquiring target CRM data with the same field value corresponding to the user name field and the user name information from all the CRM data.

S1053, if the encapsulation type is the mobile network type, acquiring the target CRM data with the same field value of the calling site ID information and the calling site ID field from all the CRM data.

In this embodiment, in the process of matching the AAA asset with the CRM data, the encapsulation type corresponding to the Radius protocol log may be determined according to the field value corresponding to the encapsulation protocol field in the AAA asset, that is, the encapsulation type is truly a fixed network or a mobile network through the value of the frame_protocol, and different methods are used to find the target CRM data matched with the AAA asset according to the encapsulation type.

If the value of the field corresponding to the encapsulation protocol field is a first set value, for example, the value of the frame_protocol is 1, it is determined that the encapsulation type is a fixed network type, that is, fiber access, and at this time, the target CRM data matched with the AAA asset is found from all CRM data through the user name in the AAA asset.

If the field value corresponding to the encapsulation protocol field is a second set value, for example, the value of frame_protocol is 7, it is determined that the encapsulation type is a mobile network type, that is, the mobile phone accesses the internet, and at this time, the target CRM data matched with the AAA asset is found from all CRM data through the call site ID in the AAA asset.

In this embodiment, after the AAA asset combined with the target CRM data is stored in the ES cluster, that is, after the AAA asset is put in storage, the AAA asset may be displayed through a web page, for example, the interface after the AAA asset is put in storage is shown in fig. 8, where Name (Name), telephone (phone number), ID Card (identity Card number) and Email (mailbox) are the target CRM data.

The process of importing CRM data into an ES cluster is described below.

Referring to fig. 9 on the basis of fig. 3, before step S103, the asset data processing method provided in the embodiment of the present application further includes step S102.

S102, importing the CRM data into the ES cluster.

In this embodiment, in order to improve the data importing efficiency, in the CRM data importing process, a file to be imported including a large amount of CRM data is stored in the S3 cluster, and then is transferred to the ES cluster through the S3 cluster.

Referring to FIG. 10, the process of importing CRM data into an ES cluster may include sub-steps S1021-S1026.

S1021, obtaining a file to be imported and file name and file attribute information of the file to be imported, wherein the file to be imported comprises a plurality of pieces of CRM data to be imported.

S1022, judging whether the file to be imported has repeated import or exists in the S3 cluster according to the file name.

S1023, when the fact that the file to be imported does not exist in repeated import and does not exist in the S3 cluster is determined, the file qualification checking is conducted on the file to be imported based on the file attribute information.

S1024, after confirming that the file qualification check is passed, calling an uploading file interface of the S3 cluster, and storing the file to be imported into the S3 cluster.

S1025, calling a download file interface of the S3 cluster, and downloading the CRM data to be imported from the S3 cluster one by one and performing field qualification check to obtain each piece of CRM data passing the field qualification check.

S1026, writing all the CRM data passing the field qualification check into the kafka cluster, and calling the North Asian platform to transfer all the CRM data passing the field qualification check from the kafka cluster to the ES cluster.

In this embodiment, taking an electronic device as an example of a server, the electronic device may include a browser, a server interface and a server daemon, where the browser is used for displaying and importing CRM menus, the server interface is used for implementing importing CRM data, and the server daemon is used for analyzing and storing CRM data.

As shown in FIG. 11, clicking the CRM import button at the browser endpoint first selects the operator to import the file, and then further selects the CRM file to import locally.

Next, the server interface acquires the file to be imported and the file name and file attribute information of the file to be imported, firstly judges whether the file to be imported has repeated import or exists in the S3 cluster according to the file name, and if the file to be imported has repeated import or exists in the S3 cluster, the file import state is displayed as uploading failure through the web interface, and the import flow is exited; and if the file to be imported does not have repeated importation and does not exist in the S3 cluster, checking the file qualification of the file to be imported based on the file attribute information, and updating the file import state of the web page to be uploading.

Optionally, the file eligibility verification may include: the file type must be dat, txt, csv, these 3 types; the file name cannot have messy codes; the imported user information must be valid, e.g., the expiration time of the identification card information must be greater than the current time; the fields in the CRM data are separated by specific symbols (for example, |) so as to facilitate the extraction of imported data according to the symbols in subsequent processing; the size of the imported file cannot exceed 4GB; etc.

It should be noted that, the purpose of the file eligibility check is to screen false or malicious files and prevent such files from entering subsequent flows, so the above file eligibility check rule is only an example, and can be flexibly set in practice, so long as false or malicious files can be screened.

After the file qualification verification is passed, since a serious performance problem exists when 4GB of files are processed at a time, in order to improve the importing efficiency, the files to be imported are stored in an S3 cluster, and then transferred from the S3 cluster to an ES cluster.

That is, after confirming that the file qualification verification passes, the uploading file interface of the S3 cluster is called, parameters such as an encryption mode, an encryption key, a bucket, a key and the like are obtained, the file to be imported is stored in the S3 cluster in an encrypted mode based on the parameters, and meanwhile, the file importing state of the web page is updated to be uploading completion.

And then, the server daemon calls a download file interface of the S3 cluster, acquires parameters such as an encryption mode, an encryption key, a bucket, a key, a file name and the like, downloads CRM data to be imported from the S3 cluster one by one based on the parameters, and performs field qualification check on the downloaded CRM data. In this process, the file import status displayed by the web page is synchronously updated to be parsing.

Alternatively, the field eligibility check may be to check the eligibility of the fields according to rules pre-configured for each field. For example, the configuration rule of the identification card number is: it must be 18 bits, the first 17 bits must be digits, the 18 th bit must be digits of 0-9, or X, the 1 st bit not 0, etc. As another example, the configuration rule of the mobile phone number is: a number of 11 bits is necessary, a 1 st bit is necessary, and the like.

It should be noted that, the purpose of the field qualification check is to screen false or malicious CRM data, and prevent such CRM data from entering a subsequent flow, so the configuration rule used in the field qualification check is only an example, and may be flexibly defined according to the data characteristics of the field in practice, so long as the false or malicious CRM data can be screened.

After obtaining each piece of field qualification checking CRM data according to the above flow, the server daemon writes all pieces of field qualification checking CRM data into the kafka cluster, for example, topic is ty_crm, and then uses the flow of the Norian platform to consume from the topic and write into the ES cluster, for example, the index is skyeye-CRM, and at this time, the file import state displayed by the web page is synchronously updated to be resolved.

In this embodiment, after the CRM data is imported into the ES cluster, the CRM data may be displayed through a web page, for example, as shown in fig. 12, the interface after the CRM data is imported may include ISP (carrier), file Name, file Status, importer Time (Import Time), and the like.

Meanwhile, in order to improve the matching efficiency of AAA assets and CRM data, in the process of importing the CRM data, the field values of key fields can be extracted for each piece of CRM data passing through field qualification verification, and a matching file is generated and stored in a Redis cluster, so that the subsequent quick matching is facilitated.

Thus, referring again to FIG. 10, the process of importing CRM data into an ES cluster may further include sub-step S1027.

S1027, extracting field values corresponding to each key field from the CRM data according to a plurality of predefined key fields for each piece of CRM data, obtaining a matching file corresponding to the CRM data and storing the matching file into a Redis cluster, wherein the matching file comprises the plurality of key fields and the field values corresponding to each key field.

In this embodiment, the key field may be field information that is required in the matching process of the identification card number, the user name, the calling site ID, and the like. As shown in fig. 11, when CRM data passing through field eligibility check is written in kafka with topic being ty_crm, synchronization generates a matching file by using a pycedar.

On this basis, referring again to fig. 10, the process of acquiring target CRM data matching AAA assets from all customer relationship management CRM data pre-imported into the ES cluster and merging the target CRM data into the AAA assets may include sub-steps S105a to S105b.

S105a, determining a target matching file matched with the AAA asset from all matching files stored in the Redis cluster.

S105b, obtaining target CRM data corresponding to the target matching file from all CRM data, and merging the target CRM data into the AAA asset.

In this embodiment, the AAA asset includes an encapsulation protocol field, a username field, a call site ID field, and respective corresponding field values, i.e., a frame_ protocol, user _name, a call_station_id field, and respective corresponding field values, and the matching file includes a username information field, a call site ID information field, and respective corresponding field values.

Thus, the process of determining a target matching file that matches the AAA asset from all matching files stored in the Redis cluster may include:

if the encapsulation type is a fixed network type, acquiring target matching files with the same field values corresponding to the user name information fields and the user name fields from all the matching files;

And if the encapsulation type is the mobile network type, acquiring the target matching files with the same field value corresponding to the calling site ID information field and the same field value corresponding to the calling site ID field from all the matching files.

In this embodiment, in the process of matching the AAA asset with the matching file, the encapsulation type corresponding to the Radius protocol log may be determined according to the field value corresponding to the encapsulation protocol field in the AAA asset, that is, the encapsulation type is truly a fixed network or a mobile network through the value of the frame_protocol, and according to the encapsulation type, the target matching file matched with the AAA asset is found by adopting different methods.

When it is noted that, the process of finding the target matching file matching the AAA asset according to the encapsulation type is similar to the process of finding the target CRM file matching the AAA asset according to the encapsulation type described in sub-steps S1051 to S1053, and this will not be repeated in the embodiment of the present application.

In practice, security analysts may need to be able to locate the real responsible person directly through network traffic, in addition to by retrieving the AAA asset of any one network traffic and locating the real responsible person generating that network traffic through that AAA asset. Therefore, AAA assets incorporating target CRM data can also be enriched into the Radius protocol log stored by the ES cluster, facilitating locating the real responsible person through log retrieval.

Therefore, referring to fig. 13 on the basis of fig. 3, after step S105, the asset data processing method provided in the embodiment of the present application further includes steps S106 to S108.

S106, acquiring all AAA assets stored in the Redis cluster according to a preset time interval.

S107, for each AAA asset, determining each target Radius protocol log with the same IP address as the asset IP address of the AAA asset from all Radius protocol logs stored in the ES cluster.

S108, calling the Noah platform to enrich the target CRM data in the AAA assets into each target Radius protocol log stored in the ES cluster.

In this embodiment, the ES cluster stores all Radius protocol logs, which include IP addresses; the Redis cluster stores all AAA assets within a set period of time (e.g., 1 hour), including asset IP addresses and the target CRM data.

Therefore, the crontab file may be configured, the sync_asset2_noah timing task may be added to the analysis platform, the cron process restarted, the data enrichment task registered, then all AAA assets stored by the recent 1 hour dis cluster are read once at preset time intervals (e.g., every 1 hour), and the target CRM data such as identification card, operator information, etc. in each AAA asset may be updated into the Radius protocol log associated with that AAA asset.

It should be noted that the purpose of data enrichment is mainly to enable security analysts to know information such as an asset group and an asset responsible person identification card of a current log when the log is retrieved, so that in order to ensure safety of related user information, sensitive information (such as an identification card number and the like) is encrypted by an AES CBC mode by default in the data enrichment process, so that the information safety of the user can be ensured, and further research, judgment, analysis and tracing of the data can be conveniently carried out by the security analysts.

In this embodiment, after enriching AAA assets combined with target CRM data to Radius protocol logs stored in an ES cluster, the web page may be used to display, for example, an interface after enriching Radius protocol logs is shown in fig. 14, where Alert Log represents a Log list, a Log represented by a black frame in the figure is a Radius protocol Log, and "JSON" shown in the lower part of the figure is target CRM data enriched to the Radius protocol Log, such as group_id, group_name, id_card, isp respectively represents an asset group IP, an asset group name, an identification card number, operator information, and so on.

Compared with the prior art, the asset data processing method provided by the embodiment of the application has the following beneficial effects:

Firstly, AAA assets are extracted from the Radius protocol log, and the Radius protocol log usually records user authentication, authorization and accounting information, so that the AAA assets of network traffic can be accurately identified, meanwhile, target CRM data matched with the AAA assets are acquired and combined into the AAA assets, and specific responsible persons can be located through the AAA assets because the CRM data usually comprise various user information.

And secondly, in the CRM data importing process, firstly storing files to be imported containing a large amount of CRM data into an S3 cluster, and then transferring the files to be imported into an ES cluster through the S3 cluster, thereby improving the data importing efficiency.

Thirdly, in the process of importing the CRM data, extracting field values of key fields for each piece of CRM data passing through field qualification verification, generating a matching file and storing the matching file into a Redis cluster, so that the matching efficiency of subsequent AAA assets and the CRM data is improved.

Fourth, enriching AAA assets incorporating target CRM data to the Radius protocol log stored by the ES cluster facilitates locating true responsible persons through log retrieval.

Referring to fig. 15, fig. 15 is a block diagram of an asset data processing device 100 according to an embodiment of the present application. The asset data processing device 100 is applied to the electronic apparatus in fig. 1, including: log acquisition module 103, asset extraction module 104, and data merge module 105.

The log obtaining module 103 is configured to obtain the Radius protocol log in real time.

The asset extraction module 104 is configured to extract AAA assets from the Radius protocol log based on a pre-configured target profile for asset identification.

The data merging module 105 is configured to obtain target CRM data matching the AAA asset from all customer relationship management CRM data pre-imported into the ES cluster, and merge the target CRM data into the AAA asset.

Optionally, the target configuration file includes a plurality of target fields, and the Radius protocol log includes a field value corresponding to each target field;

the asset extraction module 104 is specifically configured to:

extracting a field value corresponding to each target field from the Radius protocol log based on a plurality of target fields;

and formatting the field value corresponding to each target field into a dictionary type to obtain an AAA asset, wherein the AAA asset comprises a plurality of target fields and the formatted field value corresponding to each target field.

the data merge module 105 performs a manner of obtaining target CRM data matching AAA assets from all customer relationship management CRM data pre-imported into the ES cluster, including:

if the packaging type is a fixed network type, acquiring target CRM data with the same user name information and field values corresponding to the user name fields from all CRM data;

and if the encapsulation type is the mobile network type, acquiring target CRM data with the same field value of the calling site ID information and the calling site ID field from all the CRM data.

Optionally, the data merging module 105 performs a manner of determining the encapsulation type corresponding to the Radius protocol log according to the field value corresponding to the encapsulation protocol field, including:

and if the field value corresponding to the encapsulation protocol field is a second set value, determining that the encapsulation type is the mobile network type.

Optionally, the asset data processing device 100 provided in the embodiment of the present application further includes a configuration module 101, where the configuration module 101 is configured to: configuring a target field required to be used as an asset identification in a Radius protocol log into an initial configuration file of an analysis platform; and instantiating the initial configuration file to generate a target configuration file.

Optionally, the electronic device is further in communication with the S3 cluster, the kafka cluster, and the nokia platform, and the asset data processing apparatus 100 provided in the embodiment of the present application further includes a CRM data importing module 102, where the CRM data importing module 102 is configured to import CRM data into the ES cluster, and the specific implementation manner includes:

acquiring a file to be imported and file names and file attribute information of the file to be imported, wherein the file to be imported comprises a plurality of pieces of CRM data to be imported;

judging whether the file to be imported is repeatedly imported or exists in the S3 cluster according to the file name;

when the fact that the file to be imported does not exist in the repeated import and does not exist in the S3 cluster is determined, checking the file qualification of the file to be imported based on the file attribute information;

calling a download file interface of the S3 cluster, and downloading CRM data to be imported from the S3 cluster one by one and performing field qualification check to obtain each piece of CRM data passing the field qualification check;

all field-eligible CRM data is written to the kafka cluster and the nokia platform is invoked to dump all field-eligible CRM data from the kafka cluster to the ES cluster.

Optionally, the electronic device is further in communication with the dis cluster, and the CRM data importing module 102 performs a manner of importing CRM data into the ES cluster, and further includes:

and extracting a field value corresponding to each key field from the CRM data according to a plurality of key fields defined in advance for each piece of CRM data, obtaining a matching file corresponding to the CRM data and storing the matching file into a Redis cluster, wherein the matching file comprises the plurality of key fields and the field value corresponding to each key field.

The data merge module 105 performs a manner of retrieving target CRM data matching AAA assets from all customer relationship management CRM data pre-imported into the ES cluster, and merging the target CRM data into the AAA assets, including:

The data merge module 105 performs a method of determining a target matching file matching the AAA asset from all matching files stored in the dis cluster, including:

Optionally, the asset data processing device 100 provided in the embodiment of the present application further includes a data enrichment module 106, where the data enrichment module 106 is configured to:

acquiring all AAA assets stored in a Redis cluster according to a preset time interval;

invoking the Norian platform to enrich the target CRM data in the AAA asset into each target Radius protocol log stored in the ES cluster.

It will be clear to those skilled in the art that, for convenience and brevity of description, the specific working process of the asset data processing device 100 described above may refer to the corresponding process in the foregoing method embodiment, which is not repeated herein.

Referring to fig. 16, fig. 16 is a block diagram of an electronic device 10 according to an embodiment of the disclosure. The electronic device 10 includes a processor 11, a memory 12, and a bus 13, and the processor 11 is connected to the memory 12 through the bus 13.

The memory 12 is used to store programs, such as the asset data processing device 100 shown in fig. 15. The asset data processing device 100 comprises at least one software functional module which may be stored in the memory 12 in the form of software or firmware (firmware), which, upon receiving an execution instruction, is executed by the processor 11 to implement the asset data processing method disclosed in the previous embodiments.

The memory 12 may include high-speed random access memory (Random Access Memory, RAM) and may also include non-volatile memory (NVM).

The processor 11 may be an integrated circuit chip with signal processing capabilities. In implementation, the steps of the above method may be performed by integrated logic circuits of hardware in the processor 11 or by instructions in the form of software. The processor 11 may be a general-purpose processor, including a central processing unit (Central Processing Unit, CPU), a micro control unit (Microcontroller Unit, MCU), a complex programmable logic device (Complex Programmable Logic Device, CPLD), a field programmable gate array (Field Programmable Gate Array, FPGA), an embedded ARM, and the like.

The present embodiment also provides a computer readable storage medium having stored thereon a computer program which, when executed by the processor 11, implements the asset data processing method disclosed by the foregoing embodiment.

In summary, according to the asset data processing method, device, electronic equipment and storage medium provided by the embodiments of the present application, by acquiring the Radius protocol log in real time and extracting the AAA asset from the Radius protocol log based on the pre-configured target configuration file for asset identification, since the Radius protocol log generally records the user authentication, authorization and accounting information, the AAA asset of the network traffic can be accurately identified through the Radius protocol log, and meanwhile, the target CRM data matched with the AAA asset is acquired from all the CRM data pre-imported into the ES cluster and is merged into the AAA asset, and because the CRM data generally includes various user information, the target CRM data matched with the AAA asset is merged into the AAA asset, so that a specific responsible person can be located through the AAA asset.

The foregoing description is only of the preferred embodiments of the present application and is not intended to limit the same, but rather, various modifications and variations may be made by those skilled in the art. Any modification, equivalent replacement, improvement, etc. made within the spirit and principles of the present application should be included in the protection scope of the present application.

Claims

1. An asset data processing method, applied to an electronic device, the electronic device being in communication with an ES cluster, the method comprising:

acquiring a Radius protocol log in real time;

2. The method of claim 1, wherein the destination profile includes a plurality of destination fields, the Radius protocol log including a field value corresponding to each of the destination fields;

3. The method of claim 1, wherein the AAA asset comprises an encapsulation protocol field, a username field, a call site ID field, and respective corresponding field values, the CRM data comprising username information and call site ID information;

4. The method of claim 1, wherein the electronic device is further in communication with an S3 cluster, a kafka cluster, and a nokia platform;

5. The method of claim 4, wherein the electronic device is further in communication with a dis cluster;

The step of importing CRM data into the ES cluster further includes:

6. The method of claim 5, wherein the AAA asset comprises an encapsulation protocol field, a username field, a call site ID field, and respective corresponding field values, and the matching file comprises a username information field and a call site ID information field, and respective corresponding field values;

7. The method according to claim 3 or 6, wherein the step of determining the encapsulation type corresponding to the Radius protocol log according to the field value corresponding to the encapsulation protocol field includes:

8. The method of claim 1, wherein the electronic device is further in communication with an ES cluster, a Redis cluster, and a nokia platform, the ES cluster storing all Radius protocol logs, the Radius protocol logs including IP addresses; the Redis cluster stores all AAA assets within a set time period, wherein the AAA assets comprise asset IP addresses and the target CRM data;

the method further comprises the steps of:

9. An asset data processing device for application to an electronic device in communication with an ES cluster, the device comprising:

10. An electronic device comprising a processor and a memory, the memory for storing a program, the processor for implementing the asset data processing method of any of claims 1-8 when the program is executed.

11. A computer readable storage medium, characterized in that a computer program is stored thereon, which computer program, when being executed by a processor, implements the asset data processing method of any of claims 1-8.