CN113438129B - Data acquisition method and device - Google Patents
Data acquisition method and device Download PDFInfo
- Publication number
- CN113438129B CN113438129B CN202110694581.3A CN202110694581A CN113438129B CN 113438129 B CN113438129 B CN 113438129B CN 202110694581 A CN202110694581 A CN 202110694581A CN 113438129 B CN113438129 B CN 113438129B
- Authority
- CN
- China
- Prior art keywords
- data
- acquired
- node
- acquisition
- type
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 67
- 230000002776 aggregation Effects 0.000 claims description 24
- 238000004220 aggregation Methods 0.000 claims description 24
- 230000002159 abnormal effect Effects 0.000 claims description 18
- 238000004891 communication Methods 0.000 claims description 14
- 238000004458 analytical method Methods 0.000 claims description 11
- 230000004931 aggregating effect Effects 0.000 claims description 9
- 238000012545 processing Methods 0.000 description 10
- 238000010586 diagram Methods 0.000 description 8
- 238000013480 data collection Methods 0.000 description 7
- 230000005540 biological transmission Effects 0.000 description 6
- 230000008569 process Effects 0.000 description 5
- 238000007726 management method Methods 0.000 description 4
- 230000004044 response Effects 0.000 description 4
- 238000013500 data storage Methods 0.000 description 3
- 238000005516 engineering process Methods 0.000 description 3
- 230000000903 blocking effect Effects 0.000 description 2
- 238000004590 computer program Methods 0.000 description 2
- 238000012544 monitoring process Methods 0.000 description 2
- 230000008439 repair process Effects 0.000 description 2
- 238000012546 transfer Methods 0.000 description 2
- 101100459917 Caenorhabditis elegans cnd-1 gene Proteins 0.000 description 1
- 230000005856 abnormality Effects 0.000 description 1
- 238000007792 addition Methods 0.000 description 1
- 230000001413 cellular effect Effects 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 230000036541 health Effects 0.000 description 1
- 230000008676 import Effects 0.000 description 1
- 230000007246 mechanism Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 238000011084 recovery Methods 0.000 description 1
- 239000004984 smart glass Substances 0.000 description 1
- 230000003068 static effect Effects 0.000 description 1
- 238000012360 testing method Methods 0.000 description 1
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L43/00—Arrangements for monitoring or testing data switching networks
- H04L43/08—Monitoring or testing based on specific metrics, e.g. QoS, energy consumption or environmental parameters
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L43/00—Arrangements for monitoring or testing data switching networks
- H04L43/16—Threshold monitoring
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L43/00—Arrangements for monitoring or testing data switching networks
- H04L43/50—Testing arrangements
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
- H04N21/23—Processing of content or additional data; Elementary server operations; Server middleware
- H04N21/24—Monitoring of processes or resources, e.g. monitoring of server load, available bandwidth, upstream requests
Landscapes
- Engineering & Computer Science (AREA)
- Signal Processing (AREA)
- Computer Networks & Wireless Communication (AREA)
- Multimedia (AREA)
- Environmental & Geological Engineering (AREA)
- Data Exchanges In Wide-Area Networks (AREA)
Abstract
The application provides a data acquisition method and a device, wherein the data acquisition method comprises the following steps: determining a data node to be acquired and a data type to be acquired corresponding to the data node to be acquired; determining the number of nodes of the data nodes to be acquired corresponding to each data type to be acquired based on the data nodes to be acquired and the data types to be acquired; determining an acquisition period corresponding to each type of the data to be acquired based on the number of the nodes; and acquiring data of the corresponding data type to be acquired from the data node to be acquired according to the acquisition period.
Description
Technical Field
The present application relates to the field of data processing technologies, and in particular, to a data acquisition method. The application also relates to a data acquisition device, a computing device, and a computer-readable storage medium.
Background
With the development of video services, in order to improve the video viewing experience of a user and reduce the situations of blocking, screen splash, offline and the like of the user in the video viewing process as much as possible, a video provider generally pushes live audio and video data to a CDN node close to the user in advance, so that the user obtains the audio and video data nearby, thereby improving the access speed and the viewing stability of the user.
Disclosure of Invention
In view of this, the embodiment of the present application provides a data acquisition method. The application also relates to a data acquisition device, a computing device and a computer readable storage medium, which are used for solving the defect of low acquisition efficiency in the prior art.
According to a first aspect of embodiments of the present application, there is provided a data acquisition method, including:
determining a data node to be acquired and a data type to be acquired corresponding to the data node to be acquired;
determining the number of nodes of the data nodes to be acquired corresponding to each data type to be acquired based on the data nodes to be acquired and the data types to be acquired;
determining an acquisition period corresponding to each type of the data to be acquired based on the number of the nodes;
and acquiring data of the corresponding type of the data to be acquired from the data node to be acquired according to the acquisition period.
According to a second aspect of embodiments of the present application, there is provided a data acquisition apparatus including:
the determining module is configured to determine a data node to be acquired and a data type to be acquired corresponding to the data node to be acquired;
the quantity determining module is configured to determine the node quantity of the data nodes to be acquired corresponding to each data type to be acquired based on the data nodes to be acquired and the data types to be acquired;
a period determining module configured to determine an acquisition period corresponding to each type of the data to be acquired based on the number of the nodes;
and the acquisition module is configured to acquire data of the corresponding data type to be acquired from the data node to be acquired according to the acquisition cycle.
According to a third aspect of embodiments of the present application, there is provided a computing device comprising a memory, a processor, and computer instructions stored on the memory and executable on the processor, the processor implementing the steps of the data acquisition method when executing the computer instructions.
According to a fourth aspect of embodiments herein, there is provided a computer readable storage medium storing computer instructions which, when executed by a processor, implement the steps of the data acquisition method.
According to the data acquisition method, data nodes to be acquired and data types to be acquired corresponding to the data nodes to be acquired are determined; determining the number of nodes of the data nodes to be acquired corresponding to each data type to be acquired based on the data nodes to be acquired and the data types to be acquired; determining an acquisition period corresponding to each type of the data to be acquired based on the number of the nodes; and acquiring data of the corresponding data type to be acquired from the data node to be acquired according to the acquisition period. The data acquisition method and the data acquisition device realize dynamic adjustment of the acquisition period of the data types to be acquired based on the node number of the data nodes to be acquired corresponding to different data types to be acquired, so that the timeliness of data acquisition is guaranteed, and the data acquisition efficiency is improved.
Drawings
Fig. 1 is a flowchart of a data acquisition method according to an embodiment of the present application;
fig. 2 is a schematic diagram of aggregated data in a data acquisition method according to an embodiment of the present application;
fig. 3 is a schematic diagram of an architecture of a data acquisition method according to an embodiment of the present application;
fig. 4 is a processing flow chart of a data acquisition method applied to a live broadcast scene according to an embodiment of the present application;
FIG. 5 is a flowchart illustrating a data collection method applied to an on-demand scenario according to an embodiment of the present application;
FIG. 6 is a schematic structural diagram of a data acquisition device according to an embodiment of the present application;
fig. 7 is a block diagram of a computing device according to an embodiment of the present application.
Detailed Description
In the following description, numerous specific details are set forth in order to provide a thorough understanding of the present application. This application is capable of implementation in many different ways than those herein set forth and of similar import by those skilled in the art without departing from the spirit of this application and is therefore not limited to the specific implementations disclosed below.
The terminology used in the one or more embodiments of the present application is for the purpose of describing particular embodiments only and is not intended to be limiting of the one or more embodiments of the present application. As used in one or more embodiments of the present application and the appended claims, the singular forms "a," "an," and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It should also be understood that the term "and/or" as used in one or more embodiments of the present application refers to and encompasses any and all possible combinations of one or more of the associated listed items.
It should be understood that, although the terms first, second, etc. may be used herein in one or more embodiments to describe various information, these information should not be limited by these terms. These terms are only used to distinguish one type of information from another. For example, a first can also be referred to as a second and, similarly, a second can also be referred to as a first without departing from the scope of one or more embodiments of the present application. The word "if" as used herein may be interpreted as "at … …" or "when … …" or "in response to a determination", depending on the context.
First, the noun terms to which one or more embodiments of the present application relate are explained.
CDN (Content Delivery Network): the intelligent virtual network is constructed on the basis of the existing network, and users can obtain required contents nearby by means of functional modules of load balancing, content distribution, scheduling and the like of a central platform by means of edge servers deployed in various places, so that network congestion is reduced, and the access response speed and hit rate of the users are improved. The key technology of the CDN is mainly content storage and distribution technology.
CDN node: server nodes at the edge of the network (different territories such as provinces, cities) that distribute content. The system is used for storing content, is positioned at a user access point, is a content providing device facing an end user, can cache static Web content and streaming media content, and realizes edge propagation and storage of the content so as to facilitate the near access of the user.
ICMP (Internet Control Message Protocol): is a subprotocol of TCP/IP protocol cluster, which is used to transfer control message between IP host and router. The control message refers to a message of the network itself, such as network access failure, whether a host is reachable, and whether a route is available.
SNMP (Simple Network Management Protocol): is a standard protocol for managing network nodes (servers, workstations, routers, switches, etc.) in an IP network, which is an application layer protocol. Can be used to collect node traffic to calculate bandwidth.
HTTP (Hypertext Transfer Protocol): is a simple request-response protocol that typically runs on top of TCP. It specifies what messages the client may send to the server and what responses it gets, and can be used to determine if the HTTP service of the device node is normal.
TCP (Transmission Control Protocol, TCP): the method is a connection-oriented, reliable and byte stream-based transport layer communication protocol transmission control protocol, and can be used for judging whether the communication state of the HTTP node in a transport layer is normal.
Bandwidth: network bandwidth refers to the amount of data transmitted in a unit of time (typically referred to as 1 second).
Live broadcast source station: and the central server receives the anchor audio and video data.
In the present application, a data acquisition method is provided, and the present application relates to a data acquisition apparatus, a computing device, and a computer-readable storage medium, which are described in detail in the following embodiments one by one.
Fig. 1 shows a flowchart of a data acquisition method according to an embodiment of the present application, which specifically includes the following steps:
step 102: and determining a data node to be acquired and a data type to be acquired corresponding to the data node to be acquired.
The data node to be acquired may be any device node, such as a switch node, a content distribution node, a terminal node, a server node, and the like, which is not limited herein. In practical application, the data node to be collected may store any data, such as state data, resource data, index data, and the like, without limitation, the data may be collected as the data to be collected, and the data may be further divided into different data types, for example, the state data may be divided into: network status data, communication status data, and the like, the resource data may be divided into: video resource data, text resource data, audio resource data and the like, and the index data can be divided into: performance index data, service index data, and the like, without limitation.
In specific implementation, because the data nodes to be acquired are variable, it is necessary to determine which data nodes to be acquired exist and which data types (i.e., data types to be acquired) need to be acquired from the data nodes to be acquired before data acquisition, so as to perform data acquisition from the data nodes to be acquired based on the information.
In a live broadcast scene, because the requirement of a live broadcast platform on a network is very high, in order to reduce the situations of blocking, screen splash, offline and the like as much as possible, live audio and video data can be pushed to a CDN node close to a user in advance by means of the CDN node, so that the user can obtain the audio and video data from the CDN node nearby, and the access speed and the watching stability of the user are improved.
The data is cached through the CDN nodes, the bandwidth and the access pressure of a live broadcast source station can be reduced, therefore, a live broadcast platform can access a large number of CDN nodes, the CDN nodes are used for receiving uplink audio and video data and receiving the request of nearby audiences to provide the task of watching the service, the stability of the CDN for providing the service directly influences the experience of users, once the network health state of the CDN is deteriorated and even the offline condition occurs, a monitoring system must capture the fault information in time and provide the fault information to a scheduling layer, the corresponding fault node is offline, the flow on the node is transferred to the similar node, and the quality of the service and the experience of the users can be guaranteed.
How to guarantee timely and reliable acquisition of network state data of a large number of CDN nodes and efficiently transmit the data to other systems (such as a scheduling system) through a network in a structured form is a difficult problem in managing the large number of CDN nodes, and because the number of CDN nodes is large and centralized deployment and delivery are inconvenient, a plurality of acquisition nodes can actively initiate requests of a general network protocol to the CDN nodes to collect data.
Based on this, the data node to be acquired in the present application may be a CDN node, and the data to be acquired may be state data of the CDN node, and further, the type of the data to be acquired (state data) of the data to be acquired may be: the ICMP is used for judging whether the network is not communicated and the host computer is reachable or not; the SNMP is used for acquiring the flow of the CDN node to calculate the bandwidth; the HTTP is used for judging whether the HTTP service of the CDN node is normal or not; the TCP is used for judging whether the communication state of the CDN node in the transmission layer is normal or not; the bandwidth is used for judging the network access amount of the CDN node.
In practical application, in order to facilitate management of the data node to be acquired and the data type to be acquired corresponding to the data node to be acquired, in an optional implementation manner provided in the embodiment of the present application, the determining of the data node to be acquired and the data type to be acquired corresponding to the data node to be acquired are specifically implemented in the following manner:
acquiring the acquisition configuration information of the data node to be acquired from a preset position, and determining the data node to be acquired and the type of the data to be acquired corresponding to the data node to be acquired based on the acquisition configuration information.
The preset position is used for storing acquisition configuration information of a data node to be acquired, and in practical application, the preset position can point to a configuration module independent of the acquisition node (a node for acquiring data of the data node to be acquired), and the acquisition configuration information stored in the preset position is maintained in the configuration module; in addition, the preset position may also point to any storage location in the acquisition node, and when there are multiple acquisition nodes, the preset position may point to any storage location of any acquisition node, or may point to the any location in each acquisition node, that is, the acquisition configuration information may be deployed in a configuration module independent of the acquisition nodes, or may be deployed in any acquisition node, or may be deployed in each acquisition node, which is not limited herein.
The collection configuration information may include which nodes (data nodes to be collected) need to be collected, and which types of data (data types to be collected) need to be collected for each data node to be collected. In specific implementation, the configuration information to be acquired can be configured according to acquisition requirements. For example, when a data node d to be acquired is newly added, the data node d to be acquired and the data type 4 to be acquired corresponding to the data node d to be acquired may be added to the acquisition configuration information; and if need not to treat data acquisition node c and carry out data acquisition, then with treat data acquisition node c and corresponding treat data type of gathering delete from gathering the configuration information can, just so guaranteed when having new data acquisition node (for example CDN node) of treating to gather to add, do not need manual intervention to gather, just can in time monitor this data acquisition node (for example CDN node) of treating to gather, correspondingly, when treating data acquisition node (for example CDN node) no longer need monitor, gather the node and will avoid the data acquisition to this node automatically, retrieve the resource that the collection corresponds node consumed.
Taking the collection node as an example to collect data, the collection configuration information stored in the preset position ad independent of the configuration module of the collection node includes 3 data nodes to be collected, which are respectively: a data node a to be acquired, a data node b to be acquired and a data node c to be acquired; in addition, the acquisition configuration information also comprises a data type 1 to be acquired and a data type 2 to be acquired of the data node a to be acquired; the data type 1, the data type 2 and the data type 3 of the data node b to be acquired; and the data type 1 to be acquired and the data type 3 to be acquired of the data node c to be acquired. Further, after the acquisition configuration information is acquired at the preset position ad, the data node to be acquired and the type of the data to be acquired corresponding to the data node to be acquired can be determined based on the acquisition configuration information.
Step 104: and determining the number of the nodes of the data nodes to be acquired corresponding to each data type to be acquired based on the data nodes to be acquired and the data types to be acquired.
In an actual scene, the number of nodes of the data node to be acquired is large, and the number of types of the data type to be acquired is also large, so that under the condition of determining the data node to be acquired and the data type to be acquired, the number of nodes of the data node to be acquired corresponding to each data type to be acquired is counted, so that the acquisition period of each data type to be acquired is further determined according to the number of nodes, and the acquisition performance of the acquisition node is more fully and reasonably utilized.
Along the above example, under the condition that the data node to be acquired and the data type to be acquired corresponding to the data node to be acquired are determined, the number of the nodes of the data node to be acquired corresponding to the data type 1 to be acquired can be further counted to be 3, the number of the nodes of the data node to be acquired corresponding to the data type 2 to be acquired is 2, and the number of the nodes of the data node to be acquired corresponding to the data type 3 to be acquired is 2.
Step 106: and determining the acquisition period corresponding to each type of the data to be acquired based on the number of the nodes.
On the basis of determining the number of nodes, in view of the problem that the number of nodes is not fixed, in order to further improve the acquisition efficiency, the acquisition cycle corresponding to each data type to be acquired may be determined based on the number of nodes, and in practical applications, the determination manner for determining the acquisition cycle corresponding to each data type to be acquired based on the number of nodes is various, for example: the larger the number of the nodes of the data node to be acquired corresponding to the type of the data to be acquired is, the larger acquisition resources occupied for acquiring the data corresponding to the type of the data to be acquired is, so that the acquisition period of the type of the acquired data can be properly increased. In addition, the larger the number of nodes of the data node to be acquired corresponding to the data type to be acquired is, the more important the data type to be acquired may be, and therefore, the acquisition period of the data type to be acquired may be appropriately reduced.
In addition, a corresponding relationship between the number of nodes and the acquisition period may be pre-established, so as to determine the acquisition period corresponding to the type of the data to be acquired based on the counted number of nodes of the type of the data to be acquired and the pre-established corresponding relationship.
Alternatively, the formula 1 may be calculated: p is a radical of i ＝n i M, calculating an acquisition period, wherein i is the type of data to be acquired, and p i Is to be treatedCollecting the collecting period corresponding to the data type i, wherein m is the performance reference value (performance parameter) of the collecting node, and n is the performance parameter of the collecting node i The node number of the data node to be collected aiming at the data type i to be collected. Based on the method, the acquisition performance of the acquisition nodes can be utilized to the maximum extent, and the acquisition period can be dynamically adjusted according to the number of the nodes of the data nodes to be acquired.
In practical applications, the performance reference value may be obtained through a pressure test, and specifically, may be a maximum data collection amount that can be completed in a unit time under the condition that only one type of data is collected.
Further, because the importance degree or the demand period of each type of data to be collected is different, in order to make the collection period more suitable for the use requirement of the type of data to be collected, and to utilize the machine performance of the collection node to the maximum extent, and improve the collection efficiency, in an optional implementation manner provided in the embodiment of the present application, the collection period corresponding to each type of data to be collected is determined based on the number of the nodes, and is specifically implemented in the following manner:
determining type coefficients corresponding to the types of the data to be acquired;
and calculating an acquisition period corresponding to the type of the data to be acquired according to the number of the nodes, the type coefficient and the performance parameter of the current equipment, wherein the number of the nodes is in direct proportion to the acquisition period.
Specifically, the type coefficient may be understood as an acquisition coefficient corresponding to a type of data to be acquired, the type coefficient may be preset according to different aspects such as an importance degree or a demand period of the type of data to be acquired, and the larger the type coefficient is, the larger the demand period of the type of data to be acquired is, or the less important the data corresponding to the type of data to be acquired is; the smaller the type coefficient is, the smaller the demand period of the type of the data to be acquired is, or the more important the data corresponding to the type of the data to be acquired is.
Further, in addition to the node number of the to-be-acquired data node corresponding to the to-be-acquired data type and the type coefficient of the to-be-acquired data type, performance parameters of the acquisition node (current device) need to be considered, the larger the performance parameter is, the stronger the data acquisition capability of the acquisition node is, that is, the larger the data amount/acquisition frequency that can be acquired in unit time is, the smaller the performance parameter is, the weaker the data acquisition capability of the acquisition node is, that is, the smaller the data amount/acquisition frequency that can be acquired in unit time is.
Therefore, in order to utilize the machine performance of the collection node to the maximum extent and improve the collection efficiency, the following formula 2 may be used to calculate the collection period:
p i ＝k i *n i equation 2 of/m
Wherein, i is the type of data to be collected, p i Is the acquisition period corresponding to the type i of the data to be acquired, m is the performance parameter of the acquisition node, k i Type coefficient, n, for type i of data to be acquired i The node number of the data node to be collected aiming at the data type i to be collected.
Step 108: and acquiring data of the corresponding data type to be acquired from the data node to be acquired according to the acquisition period.
On the basis of determining the acquisition period corresponding to the type of the data to be acquired, the data of the corresponding type of the data to be acquired can be acquired from the data node to be acquired according to the acquisition period. For example, if the acquisition period of the data type 1 to be acquired is 30 seconds, the data type 1 to be acquired is acquired from the data node to be acquired every 30 seconds.
Further, in an optional implementation manner provided in the embodiment of the present application, the acquiring, according to the acquisition period, data of a corresponding data type to be acquired from the data node to be acquired specifically adopts the following manner:
and acquiring data of the corresponding data type to be acquired from the data node to be acquired according to the acquisition period and the priority of the data type to be acquired.
In specific implementation, in order to ensure that more important data can be collected preferentially and the collection efficiency of the important data is ensured, priorities can be preset for various types of data to be collected so as to collect data of corresponding types of data to be collected from the nodes of the data to be collected according to the priorities.
In practical application, when an acquisition node executes an acquisition task, the multi-core performance of a machine (acquisition node) is fully utilized, the acquisition task is simultaneously initiated by a plurality of threads, different priorities can be configured for different types of data to be acquired before or when the acquisition task is initiated, wherein the higher the priority of the type of the data to be acquired is, it is indicated that the data of the type of the data to be acquired needs to be acquired preferentially, and then the data of the type of the data to be acquired with the higher priority is acquired preferentially by multiple threads, for example, the types of the data to be acquired include: an ICMP type and an SNMP type, wherein the priority of the ICMP type is high, and the priority of the SNMP type is low, the data of the ICMP type is collected preferentially.
In a specific implementation, in an optional implementation manner provided in this embodiment of the present application, the acquiring data of the corresponding data type to be acquired from the data node to be acquired is specifically implemented by:
sending a data acquisition request for acquiring data of a corresponding data type to be acquired to the data node to be acquired;
receiving the data returned by the data node to be acquired based on the acquisition request;
correspondingly, after sending a collection request for collecting data of a corresponding data type to be collected to the data node to be collected, the method further includes:
and under the condition that the corresponding data is not received within the preset acquisition time corresponding to the type of the data to be acquired, returning to execute the acquisition request of transmitting the data of the corresponding type of the data to be acquired to the node of the data to be acquired.
The acquisition request refers to an acquisition request for acquiring data of the type of the data to be acquired, and after receiving the acquisition request, the data node to be acquired acquires the data corresponding to the acquisition request and returns the data to the acquisition node.
In practical application, the data node to be collected may not return data for a long time, the collection node is empty or the data collection is unsuccessful due to the problems of network connection disconnection or network congestion, the longest acquisition time (i.e. the preset acquisition time) of the data of each type of the data to be acquired can be preset according to the historical acquisition condition, if the data is not received within the preset acquisition time, which indicates that the acquisition is unsuccessful, the data is acquired again, namely, the above-mentioned acquisition request for acquiring the data of the corresponding data type to be acquired is sent back to the data node to be acquired, thereby realizing that the overtime time is configured for the acquisition tasks of different types of data, and through a mechanism of overtime retry, when the data node to be collected (such as the CDN node) does not respond within the timeout period, a new collection task is immediately initiated.
If the data are received within the preset acquisition time, the acquisition is successful, and the data are not processed.
Further, a maximum timeout number may be set for the case of collection timeout, so as to avoid that, under the condition that the timeout problem is not solved, computing resources are continuously wasted to perform data collection, in an optional implementation manner provided in this application embodiment, in the case that corresponding data is not received within the preset collection duration corresponding to the data type to be collected, a collection request for sending data of the corresponding data type to be collected to the data node to be collected is returned to execute, where the collection request includes:
under the condition that corresponding data are not received within a preset acquisition time corresponding to the type of the data to be acquired, adding one to the acquired overtime times;
judging whether the overtime times reach preset times or not;
if not, returning to execute the acquisition request for transmitting the data of the corresponding data type to be acquired to the data node to be acquired;
if so, determining that the acquisition of the data node to be acquired fails, and recording acquisition failure information of the data node to be acquired.
The timeout times refer to the timeout times for acquiring the data of the corresponding data type to be acquired from the data node to be acquired, the initial value of the timeout times is 0, and the timeout times are increased by one under the condition of timeout.
The preset times refer to preset maximum overtime times; in practical application, when the acquisition is overtime, short-time acquisition failure possibly caused by problems such as network flash and the like can be caused, so that the problem recovery time is short or the problem can be automatically recovered; or the data to be acquired may fail to be acquired for a long time due to the problems of downtime of the data nodes to be acquired, and the like, so that the problems are repaired for a long time or manually.
When the overtime times do not reach the preset times, the acquisition request for acquiring the data of the corresponding type of the data to be acquired is sent to the data node to be acquired again, namely acquisition is retried; and when the overtime times reach the preset times, determining that the data to be acquired is failed to be acquired from the data node to be acquired, and recording acquisition failure information aiming at the data node to be acquired.
The acquisition failure information comprises node identification of the data node to be acquired which is failed to acquire, acquisition time, data type to be acquired and other information. And on the basis of recording the acquisition failure information, the acquisition failure information can be used for carrying out acquisition failure notification so as to repair the data node to be acquired.
Taking the initial timeout number as 0, the preset acquisition duration as 3 seconds and the preset number as 3 times as examples, sending an acquisition request for data of the type 1 of the data to be acquired to the node a of the data to be acquired, receiving the data of the type 1 of the data to be acquired returned by the node a of the data to be acquired within 3 seconds, adding 1 to the acquired timeout number, the timeout number is 1, the timeout number is less than the preset number 3, returning to execute sending the acquisition request for the data of the type 1 of the data to be acquired to the node a of the data to be acquired, if the data of the type 1 of the data to be acquired returned by the node a of the data to be acquired is still not received within 3 seconds, adding one more to the timeout number, the timeout number is 2, the timeout number is still less than the preset number 3, returning to execute sending the acquisition request for the data of the type 1 of the data to be acquired to the node a of the data to be acquired, and receiving the data of the data type 1 to be collected returned by the data node a to be collected based on the collection request, and not retrying the collection.
Further, in an optional implementation manner provided by the embodiment of the present application, after acquiring data of a corresponding data type to be acquired from the data node to be acquired, the method further includes:
and aggregating the acquired data according to a preset aggregation strategy and reporting.
In practical application, the data are collected according to the type of the data to be collected, so that the collected data are scattered, and the collected data can be aggregated and reported after the data are collected, so that the collected data can be analyzed and managed.
The preset aggregation policy refers to a policy for summarizing (aggregating) the acquired data, for example, aggregation is performed according to a dimension of a node identifier or aggregation is performed according to a dimension of a type of the data to be acquired, in addition, the aggregation may be used as a first aggregation, and the preset aggregation policy may further include a second aggregation policy, for example, aggregation according to a data amount or aggregation according to an acquisition time, and the like, without limitation.
In practical application, the second aggregation policy may dynamically adjust the aggregated data amount or the aggregation time value according to factors such as the current network transmission condition, the busy/idle condition of the system in the current time period, and the like, on the premise of guaranteeing the timeliness requirement of the data demand side, so as to achieve the purposes of adjusting load and saving resources.
Specifically, taking the data node to be acquired as the CDN node, for example, first, according to the same CDN node (the CDN nodes are distinguished by the node identifier, such as cnd _1, CDN _2 … CDN _ n, and the like), data of different data types to be acquired (such as ICMP, SNMP, HTTP, TCP, and BandWidth, and the like) are aggregated for the first time, and are aggregated for the second time according to 1000 data volumes and reported, that is, the total 1000 data volumes are reported once, and the specific aggregated data is as shown in fig. 2 (a).
In addition, data of different CDN nodes can be aggregated for the first time according to the same data type to be acquired, and according to an aggregation time interval, the data of 2020-01-0110: 10: 00-2020-01-0110: 10: and (3) performing second aggregation on the data acquired in the time period 03 and reporting, wherein the specific aggregated data is shown in fig. 2 (b).
In specific implementation, after aggregating the collected data, the collected data is reported, which may be understood as reporting (sending or uploading, etc.) the collected data to a data gathering service or a data gathering node, and aggregating and reporting the collected data, so that as much data as possible can be transmitted in a single report, the flow pressure of the data gathering service is reduced, and the transmission efficiency is improved.
Furthermore, the data gathering service or the data gathering node is used for collecting the collected data, so that each data demander can pull corresponding data from the data gathering service or the data gathering node and perform corresponding processing.
Specifically, taking a node to be acquired as a CDN node as an example, a schematic diagram of acquiring, aggregating and using data of the CDN node by an acquisition node is shown in fig. 3, where resource management is used to configure the CDN node to be acquired, and before the acquisition node acquires data of the CDN node, the acquisition node acquires a CDN list to be acquired from resource management, where the CDN list includes CDN nodes to be acquired (such as node identifiers) and data types (i.e., types of data to be acquired) that each CDN node to be acquired needs to acquire, so that the acquisition node aggregates and reports the acquired data to a data folding service based on state data corresponding to the CDN list acquisition node after the acquisition is completed, so as to schedule, store, monitor and/or other services/systems/platforms, and pull the aggregated data from the data folding service for corresponding processing, the scheduling service/system/platform may perform traffic scheduling on the CDN node based on the pulled data, the data storage service/system/platform (e.g., a data platform) is configured to perform data storage on the pulled data so as to track or analyze the data in the following process, and the monitoring service/system/platform is configured to monitor the pulled data so as to determine whether the CDN node has an abnormal state and perform alarm processing.
Further, in an optional implementation manner provided by the embodiment of the present application, the node to be acquired includes: a content distribution node, the data comprising: the state data, correspondingly, the data acquisition method, further includes:
performing state analysis on the state of the content distribution node based on the acquired state data;
and when the analysis result is abnormal, sending an alarm notice aiming at the abnormal content distribution node.
In practical applications, the purpose of collecting the state data of the content delivery node (CDN node) is to analyze the state of the CDN node, and specifically, the state analysis of the state of the content delivery node based on the state data may be understood as comparing the collected state data with a preset state data threshold interval in a normal state, if the obtained state data is in the preset state data threshold interval, it indicates that the node state of the content delivery node is normal (i.e., the analysis result is normal), and no processing is required, and if the obtained state data is outside the preset state data threshold interval, it indicates that the node state of the content delivery node is abnormal (i.e., the analysis result is abnormal), and an alarm notification needs to be sent to the abnormal content delivery node.
For example, a network state, a communication state, and the like of the CDN node are analyzed, and if an analysis result is abnormal, an alarm notification needs to be performed for a content delivery node (i.e., a faulty node) in which the abnormality occurs, specifically, the alarm notification may include: the abnormal content distribution node identification information and the abnormal state data, so that the receiver can repair the fault of the fault node as soon as possible based on the alarm information.
In addition to performing alarm processing based on the acquired state data, the content distribution node may also be scheduled based on the state data, so as to adjust the state of the content distribution node as soon as possible according to the state data and improve the access experience of the user, where in an optional implementation manner provided in an embodiment of the present application, the data node to be acquired includes: a content distribution node, the data comprising: status data;
correspondingly, after the data of the corresponding data type to be collected is collected from the data node to be collected, the method further includes:
according to the network state data in the state data, carrying out flow scheduling on the content distribution node; and/or
And under the condition that the communication state data in the state data is abnormal, removing the content distribution node.
Specifically, the network state data includes data such as SNMP, bandwidth, and the like, and the communication state data includes: ICMP, TCP data, etc.
The traffic scheduling may be understood as scheduling the access amount of the content distribution node with a large access amount to the content distribution node with a small access amount. For example, when the bandwidth in the network state data is greater than the preset bandwidth threshold, which indicates that the traffic of the corresponding content distribution node is large, the traffic for the content distribution node may be transferred to other content distribution nodes with smaller bandwidth.
The removing process may be understood as removing the CDN node from the external service, even if the CDN node with abnormal communication does not provide the external access service. For example, if the communication state data of a certain content distribution node is abnormal, it indicates that the content distribution node cannot communicate normally, and in order to avoid affecting the access experience of the user, the content distribution node is removed.
In summary, the data acquisition method provided by the application determines the data node to be acquired and the type of the data to be acquired corresponding to the data node to be acquired; determining the number of nodes of the data nodes to be acquired corresponding to each data type to be acquired based on the data nodes to be acquired and the data types to be acquired; determining an acquisition period corresponding to each type of the data to be acquired based on the number of the nodes; and acquiring data of the corresponding data type to be acquired from the data node to be acquired according to the acquisition period. The data acquisition method and the data acquisition device realize dynamic adjustment of the acquisition period of the data types to be acquired based on the node number of the data nodes to be acquired corresponding to different data types to be acquired, so that the timeliness of data acquisition is guaranteed, and the data acquisition efficiency is improved.
The following description further describes the data acquisition method with reference to fig. 4 by taking an application of the data acquisition method provided by the present application in a live broadcast scene as an example. Fig. 4 shows a processing flow chart of a data acquisition method applied to a live broadcast scene according to an embodiment of the present application, which specifically includes the following steps:
step 402: and collecting data from the CDN node through the collection node.
Specifically, the acquisition manner may refer to the above method embodiment, which is not described herein again. The number of CDN nodes is at least one, and the acquired data may include: : the system comprises state data such as ICMP, SNMP, HTTP, TCP, bandwidth and the like, wherein the ICMP is used for judging whether a network is not communicated or a host is reachable or not; the SNMP is used for acquiring the flow of the CDN node to calculate the bandwidth; the HTTP is used for judging whether the HTTP service of the CDN node is normal or not; the TCP is used for judging whether the communication state of the CDN node in the transmission layer is normal or not; the bandwidth is used for judging the network access amount of the CDN node.
Step 404: and aggregating data through the collection nodes and reporting the data to a data gathering service.
Specifically, the collection node aggregates the data collected from the CDN nodes, specifically, the aggregation may be performed according to an ID (identifier) of the CDN node and a data quantity (for example, 1000 pieces), and may also aggregate the data according to a data type (that is, a type of data to be collected in the foregoing method embodiment) and data collection time, and report the aggregated data to a data folding service.
Step 406: aggregating data through a data gathering service.
Specifically, the data gathering service may perform a second data aggregation on the basis of the data aggregation (the first data aggregation) according to the data requirements of different data demanders (such as scheduling, data platforms, or other systems or services), so as to form structured data.
Step 408: and caching the structured data from the storage into the memory at regular time through the data gathering service.
Specifically, a period may be preset, and the data after the second aggregation may be cached in the memory from the storage, so as to increase the acquisition efficiency for the structured data.
Step 410: and under the condition that the data folding service in the data demand direction sends a pulling request, returning corresponding data to the data demand direction through the data folding service based on the pulling request.
In practice, a data requestor (such as a scheduler, data platform, or other system or service) may periodically send a pull request to the data folding service.
In summary, according to the data acquisition method provided by the application, data acquisition is performed from the CDN node, and the acquired data is subjected to data aggregation and stored in the memory, so that the data storage efficiency is improved, a data demander can pull corresponding data from the memory conveniently, and the pull rate of the data demander is increased.
The following description will further explain the data acquisition method by taking the application of the data acquisition method provided by the present application in the on-demand scenario as an example with reference to fig. 5. Fig. 5 shows a processing flow chart of a data collection method applied to an on-demand scenario, which specifically includes the following steps:
step 502: acquiring configuration information of a content distribution node from a preset position, and determining the content distribution node and a to-be-acquired data type corresponding to the content distribution node based on the acquisition configuration information.
Specifically, in a video-on-demand scenario, the types of data to be collected may include: and the video-on-demand state of the content distribution node can be monitored and analyzed through the data types such as the access amount type, the loading duration type, the code rate type and the like.
Step 504: and determining the number of nodes of the content distribution node corresponding to each data type to be collected based on the content distribution node and the data type to be collected.
Step 506: and determining type coefficients corresponding to the types of the data to be acquired.
Step 508: and calculating the acquisition period corresponding to the type of the data to be acquired according to the number of the nodes, the type coefficient and the performance parameter of the current equipment.
Wherein the number of nodes is proportional to the acquisition period.
Step 510: and sending a collection request for collecting the data of the corresponding data type to be collected to the content distribution node according to the collection period.
Step 512: and under the condition that the corresponding data is not received within the preset acquisition time corresponding to the type of the data to be acquired, adding one to the acquired overtime times.
Step 514: and judging whether the overtime times reach preset times or not.
Step 516: if the preset number of times is not reached, the process returns to step 510.
And under the condition that the preset times are reached, the data collection is not carried out on the overtime content distribution node any more.
Step 518: and under the condition that the corresponding data is received within the preset acquisition time corresponding to the type of the data to be acquired, aggregating the acquired data according to a preset aggregation strategy and reporting the aggregated data.
In summary, the data acquisition method provided by the application determines the content distribution node and the type of the data to be acquired corresponding to the content distribution node; determining the number of nodes of the content distribution nodes corresponding to each type of the data to be acquired based on the content distribution nodes and the types of the data to be acquired; determining an acquisition period corresponding to each type of the data to be acquired based on the number of the nodes; and acquiring data of the corresponding data type to be acquired from the content distribution node according to the acquisition period. The method and the device realize dynamic adjustment of the acquisition period of the data types to be acquired based on the number of the nodes of the content distribution nodes corresponding to different data types to be acquired, so that the timeliness of data acquisition is guaranteed, and the data acquisition efficiency is improved.
Corresponding to the above method embodiment, the present application further provides an embodiment of a data acquisition device, and fig. 6 shows a schematic structural diagram of the data acquisition device provided in an embodiment of the present application. As shown in fig. 6, the apparatus includes:
a determining module 602 configured to determine a data node to be acquired and a data type to be acquired corresponding to the data node to be acquired;
a quantity determining module 604, configured to determine, based on the data nodes to be acquired and the data types to be acquired, the number of nodes of the data nodes to be acquired corresponding to each data type to be acquired;
a period determining module 606 configured to determine, based on the number of nodes, an acquisition period corresponding to each type of data to be acquired;
the acquisition module 608 is configured to acquire data of the corresponding data type to be acquired from the data node to be acquired according to the acquisition cycle.
Optionally, the determining period module 606 is further configured to:
determining type coefficients corresponding to the types of the data to be acquired;
and calculating an acquisition period corresponding to the type of the data to be acquired according to the number of the nodes, the type coefficient and the performance parameter of the current equipment, wherein the number of the nodes is in direct proportion to the acquisition period.
Optionally, the acquisition module 608 is further configured to:
sending a data acquisition request for acquiring data of a corresponding data type to be acquired to the data node to be acquired;
receiving the data returned by the data node to be acquired based on the acquisition request;
correspondingly, the data acquisition device further comprises:
and returning to execute the acquisition module 608 when the corresponding data is not received within the preset acquisition time corresponding to the type of the data to be acquired.
Optionally, the acquisition module 608 is further configured to:
under the condition that corresponding data are not received within a preset acquisition time corresponding to the type of the data to be acquired, adding one to the acquired overtime times;
judging whether the overtime times reach preset times or not;
if not, returning and executing the acquisition request for transmitting the data of the corresponding data type to be acquired to the data node to be acquired.
Optionally, the acquisition module 608 is further configured to:
and acquiring data of the corresponding data type to be acquired from the data node to be acquired according to the acquisition period and the priority of the data type to be acquired.
Optionally, the determining module 602 is further configured to:
acquiring the acquisition configuration information of the data node to be acquired from a preset position, and determining the data node to be acquired and the type of the data to be acquired corresponding to the data node to be acquired based on the acquisition configuration information.
Optionally, the data acquisition apparatus further includes:
and the aggregation module is configured to aggregate the acquired data and report the aggregated data according to a preset aggregation strategy.
Optionally, the data node to be collected includes: a content distribution node, the data comprising: status data;
correspondingly, the data acquisition device further comprises:
the scheduling module is configured to perform traffic scheduling on the content distribution node according to network state data in the state data; and/or
And the removing module is configured to remove the content distribution node under the condition that the communication state data in the state data is abnormal.
Optionally, the data node to be collected includes: a content distribution node, the data comprising: status data, correspondingly, the data acquisition device, further comprises:
an analysis module configured to perform a state analysis on a state of the content distribution node based on the collected state data;
and the sending module is configured to send an alarm notification aiming at the abnormal content distribution node when the analysis result is abnormal.
In summary, the data acquisition device provided by the application determines the data node to be acquired and the type of the data to be acquired corresponding to the data node to be acquired; determining the number of nodes of the data nodes to be acquired corresponding to each data type to be acquired based on the data nodes to be acquired and the data types to be acquired; determining an acquisition period corresponding to each type of the data to be acquired based on the number of the nodes; and acquiring data of the corresponding data type to be acquired from the data node to be acquired according to the acquisition period. The data acquisition method and the data acquisition device realize dynamic adjustment of the acquisition period of the data types to be acquired based on the node number of the data nodes to be acquired corresponding to different data types to be acquired, so that the timeliness of data acquisition is guaranteed, and the data acquisition efficiency is improved.
The above is a schematic configuration of a data acquisition apparatus of the present embodiment. It should be noted that the technical solution of the data acquisition apparatus and the technical solution of the data acquisition method belong to the same concept, and details that are not described in detail in the technical solution of the data acquisition apparatus can be referred to the description of the technical solution of the data acquisition method.
FIG. 7 illustrates a block diagram of a computing device 700 provided in accordance with one embodiment of the present description. The components of the computing device 700 include, but are not limited to, memory 710 and a processor 720. Processor 720 is coupled to memory 710 via bus 730, and database 750 is used to store data.
Computing device 700 also includes access device 740, access device 740 enabling computing device 700 to communicate via one or more networks 760. Examples of such networks include the Public Switched Telephone Network (PSTN), a Local Area Network (LAN), a Wide Area Network (WAN), a Personal Area Network (PAN), or a combination of communication networks such as the internet. Access device 740 may include one or more of any type of network interface (e.g., a Network Interface Card (NIC)) whether wired or wireless, such as an IEEE802.11 Wireless Local Area Network (WLAN) wireless interface, a worldwide interoperability for microwave access (Wi-MAX) interface, an ethernet interface, a Universal Serial Bus (USB) interface, a cellular network interface, a bluetooth interface, a Near Field Communication (NFC) interface, and so forth.
In one embodiment of the present description, the above-described components of computing device 700, as well as other components not shown in FIG. 7, may also be connected to each other, such as by a bus. It should be understood that the block diagram of the computing device architecture shown in FIG. 7 is for purposes of example only and is not limiting as to the scope of the present description. Those skilled in the art may add or replace other components as desired.
Computing device 700 may be any type of stationary or mobile computing device, including a mobile computer or mobile computing device (e.g., tablet, personal digital assistant, laptop, notebook, netbook, etc.), mobile phone (e.g., smartphone), wearable computing device (e.g., smartwatch, smartglasses, etc.), or other type of mobile device, or a stationary computing device such as a desktop computer or PC. Computing device 700 may also be a mobile or stationary server.
Wherein the steps of the data acquisition method are performed by processor 720 when executing the computer instructions.
The above is an illustrative scheme of a computing device of the present embodiment. It should be noted that the technical solution of the computing device and the technical solution of the data acquisition method belong to the same concept, and details that are not described in detail in the technical solution of the computing device can be referred to the description of the technical solution of the data acquisition method.
An embodiment of the present application further provides a computer readable storage medium, which stores computer instructions, and the computer instructions, when executed by a processor, implement the steps of the data acquisition method as described above.
The above is an illustrative scheme of a computer-readable storage medium of the embodiment. It should be noted that the technical solution of the storage medium belongs to the same concept as the technical solution of the data acquisition method, and details that are not described in detail in the technical solution of the storage medium can be referred to the description of the technical solution of the data acquisition method.
The foregoing description has been directed to specific embodiments of this application. Other embodiments are within the scope of the following claims. In some cases, the actions or steps recited in the claims may be performed in a different order than in the embodiments and still achieve desirable results. In addition, the processes depicted in the accompanying figures do not necessarily require the particular order shown, or sequential order, to achieve desirable results. In some embodiments, multitasking and parallel processing may also be possible or may be advantageous.
The computer instructions comprise computer program code which may be in the form of source code, object code, an executable file or some intermediate form, or the like. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, usb disk, removable hard disk, magnetic disk, optical disk, computer Memory, Read-Only Memory (ROM), Random Access Memory (RAM), electrical carrier wave signals, telecommunications signals, software distribution medium, and the like. It should be noted that the computer-readable medium may contain suitable additions or subtractions depending on the requirements of legislation and patent practice in jurisdictions, for example, in some jurisdictions, computer-readable media may not include electrical carrier signals or telecommunication signals in accordance with legislation and patent practice.
It should be noted that, for the sake of simplicity, the above-mentioned method embodiments are described as a series of acts or combinations, but those skilled in the art should understand that the present application is not limited by the described order of acts, as some steps may be performed in other orders or simultaneously according to the present application. Further, those skilled in the art should also appreciate that the embodiments described in the specification are preferred embodiments and that the acts and modules referred to are not necessarily required in this application.
In the above embodiments, the descriptions of the respective embodiments have respective emphasis, and for parts that are not described in detail in a certain embodiment, reference may be made to related descriptions of other embodiments.
The preferred embodiments of the present application disclosed above are intended only to aid in the explanation of the application. Alternative embodiments are not exhaustive and do not limit the invention to the precise embodiments described. Obviously, many modifications and variations are possible in light of the above teaching. The embodiments were chosen and described in order to best explain the principles of the application and its practical applications, to thereby enable others skilled in the art to best understand and utilize the application. The application is limited only by the claims and their full scope and equivalents.
Claims (11)
1. A method of data acquisition, comprising:
determining a data node to be acquired and a data type to be acquired corresponding to the data node to be acquired;
determining the number of nodes of the data nodes to be acquired corresponding to each data type to be acquired based on the data nodes to be acquired and the data types to be acquired;
determining an acquisition cycle corresponding to each type of the data to be acquired based on the number of the nodes, wherein the determination of the acquisition cycle corresponding to each type of the data to be acquired based on the number of the nodes comprises determining a type coefficient corresponding to each type of the data to be acquired, and calculating the acquisition cycle corresponding to the type of the data to be acquired according to the number of the nodes, the type coefficient and a performance parameter of current equipment, wherein the number of the nodes is in direct proportion to the acquisition cycle;
and acquiring data of the corresponding data type to be acquired from the data node to be acquired according to the acquisition period.
2. The data acquisition method according to claim 1, wherein the acquiring data of the corresponding data type to be acquired from the data node to be acquired comprises:
sending a data acquisition request for acquiring data of a corresponding data type to be acquired to the data node to be acquired;
receiving the data returned by the data node to be acquired based on the acquisition request;
correspondingly, after sending a collection request for collecting data of a corresponding data type to be collected to the data node to be collected, the method further includes:
and under the condition that the corresponding data is not received within the preset acquisition time corresponding to the type of the data to be acquired, returning to execute the acquisition request for sending the data of the corresponding type of the data to be acquired to the node of the data to be acquired.
3. The data acquisition method according to claim 2, wherein the returning and executing of the acquisition request for transmitting the data of the corresponding data type to be acquired to the data node to be acquired when the corresponding data is not received within the preset acquisition duration corresponding to the data type to be acquired includes:
adding one to the overtime times of the acquisition under the condition that the corresponding data is not received within the preset acquisition time corresponding to the type of the data to be acquired;
judging whether the overtime times reach preset times or not;
if not, returning and executing the acquisition request for acquiring the data of the corresponding data type to be acquired, which is sent to the data node to be acquired.
4. The data acquisition method according to claim 1, wherein the acquiring data of the corresponding data type to be acquired from the data node to be acquired according to the acquisition cycle comprises:
and acquiring data of the corresponding data type to be acquired from the data node to be acquired according to the acquisition period and the priority of the data type to be acquired.
5. The data acquisition method according to claim 1, wherein the determining of the data node to be acquired and the data type to be acquired corresponding to the data node to be acquired comprises:
acquiring the acquisition configuration information of the data node to be acquired from a preset position, and determining the data node to be acquired and the type of the data to be acquired corresponding to the data node to be acquired based on the acquisition configuration information.
6. The data acquisition method according to claim 1, wherein after acquiring the data of the corresponding data type to be acquired from the data node to be acquired, the method further comprises:
and aggregating the acquired data according to a preset aggregation strategy and reporting.
7. The data acquisition method according to any one of claims 1 to 6, wherein the data node to be acquired comprises: a content distribution node, the data comprising: status data;
correspondingly, after the data of the corresponding data type to be collected is collected from the data node to be collected, the method further includes:
according to the network state data in the state data, carrying out flow scheduling on the content distribution node; and/or
And under the condition that the communication state data in the state data is abnormal, removing the content distribution node.
8. The data acquisition method according to any one of claims 1 to 6, wherein the data node to be acquired comprises: a content distribution node, the data comprising: status data;
correspondingly, the data acquisition method further comprises the following steps:
performing state analysis on the state of the content distribution node based on the acquired state data;
and when the analysis result is abnormal, sending an alarm notice aiming at the abnormal content distribution node.
9. A data acquisition device, comprising:
the determining module is configured to determine a data node to be acquired and a data type to be acquired corresponding to the data node to be acquired;
the quantity determining module is configured to determine the node quantity of the data nodes to be acquired corresponding to each data type to be acquired based on the data nodes to be acquired and the data types to be acquired;
a period determining module configured to determine an acquisition period corresponding to each type of the data to be acquired based on the number of nodes, wherein the period determining module determines the acquisition period corresponding to each type of the data to be acquired based on the number of nodes, and includes determining a type coefficient corresponding to each type of the data to be acquired, and calculating the acquisition period corresponding to the type of the data to be acquired according to the number of nodes, the type coefficient, and a performance parameter of current equipment, and the number of nodes is in direct proportion to the acquisition period;
and the acquisition module is configured to acquire data of the corresponding data type to be acquired from the data node to be acquired according to the acquisition cycle.
10. A computing device comprising a memory, a processor, and computer instructions stored on the memory and executable on the processor, wherein the processor implements the steps of the method of any one of claims 1-8 when executing the computer instructions.
11. A computer-readable storage medium storing computer instructions, which when executed by a processor implement the steps of the method of any one of claims 1 to 8.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110694581.3A CN113438129B (en) | 2021-06-22 | 2021-06-22 | Data acquisition method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110694581.3A CN113438129B (en) | 2021-06-22 | 2021-06-22 | Data acquisition method and device |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN113438129A CN113438129A (en) | 2021-09-24 |
| CN113438129B true CN113438129B (en) | 2022-08-26 |
Family
ID=77757176
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110694581.3A Active CN113438129B (en) | 2021-06-22 | 2021-06-22 | Data acquisition method and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113438129B (en) |
Families Citing this family (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114070667A (en) * | 2021-11-16 | 2022-02-18 | 张家口安智科为新能源有限公司 | Method for calling data of Modbus protocol equipment |
| CN114415560A (en) * | 2021-12-24 | 2022-04-29 | 浙江清华柔性电子技术研究院 | Data synchronous acquisition and processing method and device and storage medium |
| CN114301942B (en) * | 2021-12-29 | 2024-07-26 | 杭州涂鸦信息技术有限公司 | Data reporting method, data reporting device and computer readable storage medium |
| CN114640632B (en) * | 2022-03-25 | 2023-12-15 | 北京奇艺世纪科技有限公司 | Data aggregation method, system, device, equipment and storage medium |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108768791A (en) * | 2018-07-04 | 2018-11-06 | 山东汇贸电子口岸有限公司 | A kind of information collection configuration management system and method |
| CN112565035A (en) * | 2020-11-20 | 2021-03-26 | 珠海格力电器股份有限公司 | Data transmission method, device and storage medium |
Family Cites Families (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103236949B (en) * | 2013-04-27 | 2016-12-28 | 北京搜狐新媒体信息技术有限公司 | Monitoring method, device and the system of a kind of server cluster |
| CN106936888A (en) * | 2015-12-30 | 2017-07-07 | 五八同城信息技术有限公司 | The method of Data Collection, apparatus and system |
| CN110347557A (en) * | 2019-07-16 | 2019-10-18 | 深圳市网心科技有限公司 | Node resource monitoring method, electronic equipment, medium and acquisition method |
-
2021
- 2021-06-22 CN CN202110694581.3A patent/CN113438129B/en active Active
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108768791A (en) * | 2018-07-04 | 2018-11-06 | 山东汇贸电子口岸有限公司 | A kind of information collection configuration management system and method |
| CN112565035A (en) * | 2020-11-20 | 2021-03-26 | 珠海格力电器股份有限公司 | Data transmission method, device and storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN113438129A (en) | 2021-09-24 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN113438129B (en) | Data acquisition method and device | |
| US11089516B2 (en) | Systems and methods for network performance monitoring, event detection, and remediation | |
| EP2815538B1 (en) | Method and apparatus for rapid disaster recovery preparation in a cloud network | |
| CA2747336C (en) | Dynamic mobile network traffic control | |
| JP2022547748A (en) | Methods and apparatus for transmitting data in IoT systems, gateway devices and their storage media | |
| CN113055693B (en) | Data processing method and device | |
| JP7743629B2 (en) | Method, device and readable storage medium for analyzing model transmission status in a subscription network | |
| EP3447970A1 (en) | Live broadcast system in peer-to-peer network and node management method | |
| US10931529B2 (en) | Terminal device management method, server, and terminal device for managing terminal devices in local area network | |
| US9800662B2 (en) | Generic network trace with distributed parallel processing and smart caching | |
| CN113438494A (en) | Data processing method and device | |
| CN105554099A (en) | Method and device for balancing load of acquisition servers | |
| CN112491961A (en) | Scheduling system and method and CDN system | |
| WO2017084618A1 (en) | Method for monitoring node communication of shared storage cluster file system, and monitoring node | |
| US20130128729A1 (en) | Communication network operator traffic regulation manager and data collection manager and method of operation thereof | |
| CN109257335B (en) | Method for maintaining back source link, back source method, related device and storage medium | |
| CN101917406A (en) | Data transmission method and data transmission system | |
| CN101695049A (en) | Method and device for processing businesses in monitoring system | |
| US10103950B2 (en) | Hub filtering | |
| WO2013189421A2 (en) | Distributed call ticket statistical method, device and system | |
| Lee et al. | Monitoring of IoT data for reducing network traffic | |
| WO2025017363A1 (en) | Data analytics for energy management | |
| CN119030972A (en) | Processing method, device and storage medium for computing power and network services | |
| CN107147867B (en) | Distributed transcoding method and system | |
| CN116455686B (en) | Message broadcasting method and device of emergency broadcasting platform |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |