CN110826001A - Data acquisition request processing method and device, computer equipment and storage medium - Google Patents
Data acquisition request processing method and device, computer equipment and storage medium Download PDFInfo
- Publication number
- CN110826001A CN110826001A CN201911047636.0A CN201911047636A CN110826001A CN 110826001 A CN110826001 A CN 110826001A CN 201911047636 A CN201911047636 A CN 201911047636A CN 110826001 A CN110826001 A CN 110826001A
- Authority
- CN
- China
- Prior art keywords
- transmission link
- information
- client
- data
- acquisition request
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/90—Details of database functions independent of the retrieved data types
- G06F16/95—Retrieval from the web
- G06F16/958—Organisation or management of web site content, e.g. publishing, maintaining pages or automatic linking
- G06F16/972—Access to data in other repository systems, e.g. legacy data or dynamic Web page generation
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/90—Details of database functions independent of the retrieved data types
- G06F16/95—Retrieval from the web
- G06F16/958—Organisation or management of web site content, e.g. publishing, maintaining pages or automatic linking
- G06F16/986—Document structures and storage, e.g. HTML extensions
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/50—Network services
- H04L67/56—Provisioning of proxy services
Landscapes
- Engineering & Computer Science (AREA)
- Databases & Information Systems (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a data acquisition request processing method, a data acquisition request processing device, computer equipment and a storage medium, wherein the method comprises the following steps: the method comprises the steps of obtaining agent description information and transmission link information of a client and storing the agent description information and the transmission link information into an agent database, obtaining a target transmission link corresponding to a data obtaining request in the agent database if the data obtaining request is received through a request receiving interface, generating corresponding obtaining information according to the agent description information and the data obtaining request corresponding to the target transmission link, obtaining a data result according to the target transmission link, storing the data result into the agent database and feeding back the data result according to the request receiving interface. By the method, the terminal address of the client can be used as the proxy terminal address to acquire the data information, the stability of the proxy terminal address is improved, the plurality of data acquisition requests can be processed concurrently to acquire data results in batches, the requirement of large-scale enterprises on acquiring large-batch data is met, and the efficiency of acquiring the data is greatly improved.
Description
Technical Field
The present invention relates to the field of computer technologies, and in particular, to a method and an apparatus for processing a data acquisition request, a computer device, and a storage medium.
Background
When data is acquired, a client sends a data acquisition request to acquire data information stored in a website, the current website limits the number of times that the same terminal address (IP address) sends the data acquisition request in a certain time period, if the number of times exceeds the limited number of times, the subsequent data acquisition request is intercepted, and the data information cannot be acquired by using the same terminal address in the time period. In the conventional technical method, a VPS server (virtual private server) which is provided by a cloud service provider and can dynamically switch an IP address is used to avoid interception of a sent data acquisition request, but the method has the problem that part of IP addresses are unavailable, so that the stability of the IP addresses is not high, the data acquisition efficiency is greatly influenced, and the implementation cost is high due to frequent dialing through a VPS server. Therefore, the prior art method has the problem of low data acquisition efficiency when processing the data acquisition request.
Disclosure of Invention
The embodiment of the invention provides a method and a device for processing a data acquisition request, computer equipment and a storage medium, aiming at solving the problem of low data acquisition efficiency when the data acquisition request is processed in the prior art.
In a first aspect, an embodiment of the present invention provides a data acquisition request processing method, including:
acquiring agent description information and transmission link information of each client and storing the agent description information and the transmission link information into a preset agent database, wherein the transmission link information at least comprises one transmission link corresponding to the client;
if a data acquisition request from any one of the request receiving interfaces is received, acquiring a transmission link corresponding to the data acquisition request in the proxy database as a target transmission link;
acquiring agent description information of a client corresponding to the target transmission link as target agent description information, and generating acquisition information corresponding to the target agent description information according to the data acquisition request;
acquiring a data result corresponding to the acquired information according to the target transmission link;
and storing the data result to the proxy database, and feeding back the data result according to the request receiving interface.
In a second aspect, an embodiment of the present invention provides a data acquisition request processing apparatus, including:
the client information storage unit is used for acquiring the agent description information and the transmission link information of each client and storing the agent description information and the transmission link information into a preset agent database, wherein the transmission link information at least comprises one transmission link corresponding to the client;
a target transmission link obtaining unit, configured to, if a data obtaining request from any one of the request receiving interfaces is received, obtain a transmission link, corresponding to the data obtaining request, in the proxy database as a target transmission link;
an acquisition information generating unit, configured to acquire proxy description information of a client corresponding to the target transmission link as target proxy description information, and generate acquisition information corresponding to the target proxy description information according to the data acquisition request;
the data result acquisition unit is used for acquiring a data result corresponding to the acquisition information according to the target transmission link;
and the data result storage and feedback unit is used for storing the data result to the proxy database and feeding back the data result according to the request receiving interface.
In a third aspect, an embodiment of the present invention further provides a computer device, which includes a memory, a processor, and a computer program stored on the memory and executable on the processor, where the processor implements the data acquisition request processing method according to the first aspect when executing the computer program.
In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium, where the computer-readable storage medium stores a computer program, and the computer program, when executed by a processor, causes the processor to execute the data acquisition request processing method according to the first aspect.
The embodiment of the invention provides a data acquisition request processing method, a data acquisition request processing device, computer equipment and a storage medium. The method comprises the steps of obtaining agent description information and transmission link information of a client and storing the agent description information and the transmission link information into an agent database, obtaining a target transmission link corresponding to a data obtaining request in the agent database if the data obtaining request is received through a request receiving interface, generating corresponding obtaining information according to the agent description information and the data obtaining request corresponding to the target transmission link, obtaining a data result according to the target transmission link, storing the data result into the agent database and feeding back the data result through the request receiving interface. By the method, the terminal address of the client can be used as the proxy terminal address to acquire the data information, the stability of the proxy terminal address is improved, the plurality of data acquisition requests can be processed concurrently to acquire data results in batches, the requirement of large-scale enterprises on acquiring large-batch data is met, and the efficiency of acquiring the data is greatly improved.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings needed to be used in the description of the embodiments are briefly introduced below, and it is obvious that the drawings in the following description are some embodiments of the present invention, and it is obvious for those skilled in the art to obtain other drawings based on these drawings without creative efforts.
Fig. 1 is a schematic flowchart of a data acquisition request processing method according to an embodiment of the present invention;
fig. 2 is a schematic view of an application scenario of a data acquisition request processing method according to an embodiment of the present invention;
fig. 3 is a schematic sub-flow diagram of a data acquisition request processing method according to an embodiment of the present invention;
fig. 4 is another schematic flow chart of a data acquisition request processing method according to an embodiment of the present invention;
fig. 5 is a schematic sub-flow chart of a data acquisition request processing method according to an embodiment of the present invention;
fig. 6 is a schematic block diagram of a data acquisition request processing apparatus according to an embodiment of the present invention;
FIG. 7 is a schematic block diagram of a computer device provided by an embodiment of the present invention.
Detailed Description
The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are some, not all, embodiments of the present invention. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
It will be understood that the terms "comprises" and/or "comprising," when used in this specification and the appended claims, specify the presence of stated features, integers, steps, operations, elements, and/or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and/or groups thereof.
It is also to be understood that the terminology used in the description of the invention herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. As used in the specification of the present invention and the appended claims, the singular forms "a," "an," and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise.
It should be further understood that the term "and/or" as used in this specification and the appended claims refers to and includes any and all possible combinations of one or more of the associated listed items.
Referring to fig. 1 and fig. 2, fig. 1 is a schematic flow chart of a data acquisition request processing method according to an embodiment of the present invention, and fig. 2 is a schematic application scenario diagram of the data acquisition request processing method according to the embodiment of the present invention. The data acquisition request processing method is applied to a management server 10, and is executed through application software installed in the management server 10, and a user terminal 20 performs data information transmission by establishing a network connection with the management server 10, wherein the user terminal 20 is a terminal device capable of sending data information to the management server, such as a desktop computer, a notebook computer, a tablet computer or a mobile phone, and the management server 10 is an enterprise terminal for processing a data acquisition request to acquire data information. In addition, in the embodiment, only one user terminal 20 is illustrated to establish network connection with the management server 10, and in an actual application process, multiple user terminals 20 can establish network connection with the management server 10 at the same time.
As shown in fig. 1, the method includes steps S110 to S150.
And S110, acquiring the agent description information and the transmission link information of each client and storing the agent description information and the transmission link information into a preset agent database.
And acquiring the agent description information and the transmission link information of each client and storing the agent description information and the transmission link information into a preset agent database, wherein the transmission link information at least comprises one transmission link corresponding to the client. At least one client is connected with a management server through a network, the client acquires data information stored in a website through one or more transmission links, the management server acquires the data information from the website by taking a terminal address (IP address) of the client as an agent IP address after receiving a data acquisition request, the agent IP addresses of the clients form an IP agent pool at the moment, and the data information is acquired from the website through the agent IP addresses contained in the IP agent pool, so that each agent IP address can be ensured to be really available, the stability of the agent IP addresses is improved, and the management server can acquire the data efficiently. The agent description information is information for describing the information related to the client, after the management server obtains the agent description information of each client, the client can be classified according to the agent description information, the transmission link information is information for recording a transmission link between the client and the management server, and the client can communicate with the management server through one or more transmission links. Wherein the proxy database may be a database suitable for high concurrency processing, such as a redis database.
In an embodiment, as shown in fig. 3, step S110 includes sub-steps S111, S112 and S113.
And S111, sending a terminal address acquisition request to each client to acquire the terminal address information fed back by each client according to the terminal address.
And sending a terminal address acquisition request to each client to acquire the terminal address information fed back by each client according to the terminal address. The client side is pre-provided with an agent application program, the IP address of the client side is a stable IP address, the client side establishes network connection with the management server through the agent application program, the management server sends a terminal address acquisition request to the client side based on the network connection, the client side correspondingly feeds back terminal address information according to the terminal address acquisition request, and the management server can acquire the terminal address information corresponding to each client side.
And S112, establishing a transmission link with each client according to the terminal address information.
And establishing a transmission link with each client according to the terminal address information. After the terminal address information of each client is obtained, one or more Transmission links can be correspondingly established with each client according to the terminal address information, the Transmission links can be used for transmitting information between the client and the management server, specifically, the Transmission links are used for long-time connection communication between the management server and the client, the Transmission links can be WebSocket connection links established based on the golang Protocol, WebSocket is a Protocol for performing full-duplex communication on a single TCP (Transmission Control Protocol) connection provided by the HTML5 standard (an execution standard of an HTML web page editor), the connection links established by the WebSocket are WebSocket connection links, the WebSocket enables data exchange between the client and the management server to be simpler, allows the management server to actively push data to the client, and the management server can maintain a read Protocol and a write Protocol corresponding to the Transmission links respectively based on each Transmission link. The "write protocol" is used to transmit the acquisition information from the management server to the client, and the "read protocol" is used to acquire the data information transmitted from the client to the management server.
And S113, acquiring corresponding agent description information and transmission link information from each client and storing the agent description information and the transmission link information into a preset agent database.
And acquiring corresponding agent description information and transmission link information from each client and storing the corresponding agent description information and transmission link information in a preset agent database. The corresponding agent description information and the transmission link information can be obtained from each client based on the established transmission link, and the agent description information can also include the hardware information of the corresponding client.
For example, the proxy description information corresponding to a certain client is "area: shandong, IP: 103.10.xx.xxx, system version: win7, network latency: 75ms ", the corresponding transmission link information of the client comprises two transmission links, and the transmission link 1 is 103.10. XX.XXX: 1; transmission link 2 is 103.10. xx.xxx: 2.
and S120, if a data acquisition request from any one of the request receiving interfaces is received, acquiring a transmission link corresponding to the data acquisition request in the proxy database as a target transmission link.
And if a data acquisition request from any one of the request receiving interfaces is received, acquiring a transmission link corresponding to the data acquisition request in the proxy database as a target transmission link. Specifically, the request receiving interface is an interface used by the management server to obtain the data obtaining request, and the request receiving interface may be an http interface or a network interface in other forms; the data acquisition request comprises request parameters, one or more clients corresponding to the data acquisition request in the proxy database can be acquired based on the request parameters, and a transmission link corresponding to each client is used as a target transmission link.
For example, the request parameter of a certain data acquisition request is "area: henan, network latency: and if the time is less than or equal to 50 ms', the transmission link of the corresponding client in the proxy database can be obtained according to the request parameter, so as to obtain the target transmission link.
In an embodiment, as shown in fig. 4, step S121 is further included after step S120.
And S121, storing the data acquisition request into a storage area corresponding to the storage address in the proxy database according to the storage address of the client to which the target transmission link belongs.
And storing the data acquisition request into a storage area corresponding to the storage address in the proxy database according to the storage address of the client to which the target transmission link belongs. After receiving the data acquisition request, the management server may correspondingly store the data acquisition request in the proxy database, specifically, each acquired target storage link corresponds to one client, and the data acquisition request is stored according to a storage address of proxy description information of the client to which the target storage link belongs. If the plurality of target storage links all belong to the same client, only one data acquisition request is stored in the storage address of the agent description information of the client.
S130, acquiring the agent description information of the client corresponding to the target transmission link as target agent description information, and generating acquisition information corresponding to the target agent description information according to the data acquisition request.
And acquiring agent description information of a client corresponding to the target transmission link as target agent description information, and generating acquisition information corresponding to the target agent description information according to the data acquisition request. The method comprises the steps of obtaining agent description information of a client of each target transmission link to obtain target agent description information, and correspondingly generating obtaining information according to a data obtaining request and the target agent description information, wherein the obtaining information can be a computer instruction for obtaining data information.
And S140, acquiring a data result corresponding to the acquired information according to the target transmission link.
And acquiring a data result corresponding to the acquired information according to the target transmission link. The data result corresponding to the acquisition information, that is, the data result corresponding to the data acquisition request, may be acquired based on the target transmission link. Specifically, the data result corresponding to the acquired information may be acquired by any one of the target transmission links, or may be acquired simultaneously by a plurality of target transmission links, and the data result corresponding to the acquired information may be acquired from the one target transmission link with the highest feedback speed. The target transmission link comprises a write protocol process and a read protocol process.
In an embodiment, as shown in fig. 5, step S140 includes sub-steps S141 and S142.
And S141, sending the acquired information to a client corresponding to the target transmission link through the write protocol of the target transmission link.
And sending the acquired information to a client corresponding to the target transmission link through a write protocol of the target transmission link, wherein the write protocol can be used for transmitting the acquired information from the management server to the client, so that the acquired information can be sent to the corresponding client through the write protocol.
And S142, receiving a data result fed back by the user side according to the acquisition information according to the read protocol of the target transmission link.
And receiving a data result fed back by the user side according to the acquisition information according to the reading protocol of the target transmission link, wherein the reading protocol is used for acquiring data information transmitted from the client side to the management server. After receiving the acquisition information, the client can acquire a corresponding data result from the website according to the acquisition information, and feed the data result back to the management server through the protocol reading process, so that the management server can acquire the fed-back data result through the protocol reading process.
S150, storing the data result to the proxy database, and feeding back the data result according to the request receiving interface.
And storing the data result to the proxy database, and feeding back the data result according to the request receiving interface. And correspondingly storing the data result into the proxy database, and feeding back the data result according to the request receiving interface for receiving the data acquisition request, namely finishing the processing process of one data acquisition request. The processing process of the data acquisition request can be performed in a multi-concurrent mode, namely, the data acquisition request is received through the plurality of request receiving interfaces simultaneously, and the data acquisition requests are concurrently processed through the management server to acquire data results in batches, so that the requirement of large-scale enterprises on acquiring mass data can be met, and the data acquisition efficiency is greatly improved.
In an embodiment, step S150 includes sub-step S151.
And S151, storing the data result into a storage area corresponding to the storage address of the data acquisition request in the proxy database.
And storing the data result into a storage area corresponding to the storage address of the data acquisition request in the proxy database. If the data acquisition request is stored in at least one storage area in the proxy database, the data result corresponding to the data acquisition request can be stored in the storage space of the corresponding storage area according to the storage address of the data acquisition request. If the data acquisition request is stored in a plurality of storage addresses of the proxy database, the data result needs to be stored in a plurality of storage areas corresponding to the plurality of storage addresses.
The data acquisition request processing method provided by the embodiment of the invention is characterized in that the proxy description information and the transmission link information of the client are acquired and stored in the proxy database, if the data acquisition request is received through the request receiving interface, the target transmission link corresponding to the data acquisition request in the proxy database is acquired, the corresponding acquisition information is generated according to the proxy description information and the data acquisition request corresponding to the target transmission link, the data result is acquired according to the target transmission link, and the data result is stored in the proxy database and fed back through the request receiving interface. By the method, the terminal address of the client can be used as the proxy terminal address to acquire the data information, the stability of the proxy terminal address is improved, the plurality of data acquisition requests can be processed concurrently to acquire data results in batches, the requirement of large-scale enterprises on acquiring large-batch data is met, and the efficiency of acquiring the data is greatly improved.
An embodiment of the present invention further provides a data acquisition request processing apparatus, where the data acquisition request processing apparatus is configured to execute any one of the embodiments of the data acquisition request processing method. Specifically, referring to fig. 6, fig. 6 is a schematic block diagram of a data acquisition request processing apparatus according to an embodiment of the present invention. The data acquisition request processing device may be disposed in the management server 10.
As shown in fig. 6, the data acquisition request processing apparatus 100 includes a client information storage unit 110, a target transmission link acquisition unit 120, an acquisition information generation unit 130, a data result acquisition unit 140, and a data result storage feedback unit 150.
The client information storage unit 110 is configured to obtain agent description information and transmission link information of each client and store the agent description information and the transmission link information into a preset agent database, where the transmission link information at least includes one transmission link corresponding to the client.
In other embodiments of the present invention, the client information storage unit 110 includes sub-units: a terminal address information acquisition unit 111, a transmission link establishment unit 112, and a client information acquisition unit 113.
A terminal address information obtaining unit 111, configured to send a terminal address obtaining request to each client, so as to obtain terminal address information fed back by each client according to the terminal address; a transmission link establishing unit 112, configured to establish a transmission link with each client according to the terminal address information; the client information obtaining unit 113 is configured to obtain corresponding agent description information and transmission link information from each client and store the obtained information in a preset agent database.
A target transmission link obtaining unit 120, configured to, if a data obtaining request from any one of the request receiving interfaces is received, obtain a transmission link, as a target transmission link, in the proxy database, corresponding to the data obtaining request.
In another embodiment of the present invention, the data acquisition request processing apparatus 100 further includes a sub-unit: the data acquisition request storage unit 121.
A data obtaining request storage unit 121, configured to store the data obtaining request into a storage area, corresponding to the storage address, in the proxy database according to the storage address of the client to which the target transmission link belongs.
An obtaining information generating unit 130, configured to obtain, as target proxy description information, proxy description information of a client corresponding to the target transmission link, and generate obtaining information corresponding to the target proxy description information according to the data obtaining request.
A data result obtaining unit 140, configured to obtain a data result corresponding to the obtained information according to the target transmission link.
In other embodiments of the present invention, the data result obtaining unit 140 includes sub-units: an acquisition information transmitting unit 141 and a data result receiving unit 142.
An acquired information sending unit 141, configured to send the acquired information to a client corresponding to the target transmission link through a write protocol of the target transmission link; and a data result receiving unit 142, configured to receive, according to the read protocol of the target transmission link, a data result fed back by the user end according to the acquisition information.
And a data result storage and feedback unit 150, configured to store the data result in the proxy database, and feed back the data result according to the request receiving interface.
In other embodiments of the present invention, the data result storage feedback unit 150 includes sub-units: a data result storage unit 151.
A data result storage unit 151, configured to store the data result into a storage area corresponding to the storage address of the data obtaining request in the proxy database.
The data acquisition request processing device provided by the embodiment of the invention is used for executing the data acquisition request processing method, acquiring proxy description information and transmission link information of a client and storing the proxy description information and the transmission link information into a proxy database, acquiring a target transmission link corresponding to a data acquisition request in the proxy database if the data acquisition request is received through a request receiving interface, generating corresponding acquisition information according to the proxy description information and the data acquisition request corresponding to the target transmission link, acquiring a data result according to the target transmission link, storing the data result into the proxy database and feeding back the data result through the request receiving interface. By the method, the terminal address of the client can be used as the proxy terminal address to acquire the data information, the stability of the proxy terminal address is improved, the plurality of data acquisition requests can be processed concurrently to acquire data results in batches, the requirement of large-scale enterprises on acquiring large-batch data is met, and the efficiency of acquiring the data is greatly improved.
The data acquisition request processing means described above may be implemented in the form of a computer program that can be run on a computer device as shown in fig. 7.
Referring to fig. 7, fig. 7 is a schematic block diagram of a computer device according to an embodiment of the present invention.
Referring to fig. 7, the computer device 500 includes a processor 502, memory, and a network interface 505 connected by a system bus 501, where the memory may include a non-volatile storage medium 503 and an internal memory 504.
The non-volatile storage medium 503 may store an operating system 5031 and a computer program 5032. The computer program 5032, when executed, may cause the processor 502 to perform a data acquisition request processing method.
The processor 502 is used to provide computing and control capabilities that support the operation of the overall computer device 500.
The internal memory 504 provides an environment for the operation of the computer program 5032 in the nonvolatile storage medium 503, and when the computer program 5032 is executed by the processor 502, the processor 502 may be caused to execute the data acquisition request processing method.
The network interface 505 is used for network communication, such as providing transmission of data information. Those skilled in the art will appreciate that the configuration shown in fig. 7 is a block diagram of only a portion of the configuration associated with aspects of the present invention and is not intended to limit the computing device 500 to which aspects of the present invention may be applied, and that a particular computing device 500 may include more or less components than those shown, or may combine certain components, or have a different arrangement of components.
Wherein the processor 502 is configured to run the computer program 5032 stored in the memory to implement the following functions: acquiring agent description information and transmission link information of each client and storing the agent description information and the transmission link information into a preset agent database, wherein the transmission link information at least comprises one transmission link corresponding to the client; if a data acquisition request from any one of the request receiving interfaces is received, acquiring a transmission link corresponding to the data acquisition request in the proxy database as a target transmission link; acquiring agent description information of a client corresponding to the target transmission link as target agent description information, and generating acquisition information corresponding to the target agent description information according to the data acquisition request; acquiring a data result corresponding to the acquired information according to the target transmission link; and storing the data result to the proxy database, and feeding back the data result according to the request receiving interface.
In an embodiment, the processor 502 performs the following operations when performing the step of obtaining the agent description information and the transmission link information of each client and storing the obtained information in a preset agent database: sending a terminal address acquisition request to each client to acquire terminal address information fed back by each client according to the terminal address; establishing a transmission link with each client according to the terminal address information; and acquiring corresponding agent description information and transmission link information from each client and storing the corresponding agent description information and transmission link information in a preset agent database.
In an embodiment, after the processor 502 performs the step of acquiring, if a data acquisition request from any one of the request receiving interfaces is received, a transmission link corresponding to the data acquisition request in the proxy database as a target transmission link, the following operations are further performed: and storing the data acquisition request into a storage area corresponding to the storage address in the proxy database according to the storage address of the client to which the target transmission link belongs.
In an embodiment, when the processor 502 executes the steps of storing the data result in the proxy database and feeding back the data result according to the request receiving interface, the following operations are performed: and storing the data result into a storage area corresponding to the storage address of the data acquisition request in the proxy database.
In an embodiment, when the processor 502 executes the step of acquiring the data result corresponding to the acquisition information according to the target transmission link, the following operations are performed: sending the acquired information to a client corresponding to the target transmission link through a write protocol of the target transmission link; and receiving a data result fed back by the user side according to the acquired information according to the read protocol of the target transmission link.
Those skilled in the art will appreciate that the embodiment of a computer device illustrated in fig. 7 does not constitute a limitation on the specific construction of the computer device, and that in other embodiments a computer device may include more or fewer components than those illustrated, or some components may be combined, or a different arrangement of components. For example, in some embodiments, the computer device may only include a memory and a processor, and in such embodiments, the structures and functions of the memory and the processor are consistent with those of the embodiment shown in fig. 7, and are not described herein again.
It should be understood that, in the embodiment of the present invention, the Processor 502 may be a Central Processing Unit (CPU), and the Processor 502 may also be other general purpose processors, Digital Signal Processors (DSPs), Application Specific Integrated Circuits (ASICs), Field-Programmable gate arrays (FPGAs) or other Programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, and the like. Wherein a general purpose processor may be a microprocessor or the processor may be any conventional processor or the like.
In another embodiment of the invention, a computer-readable storage medium is provided. The computer readable storage medium may be a non-volatile computer readable storage medium. The computer-readable storage medium stores a computer program, wherein the computer program when executed by a processor implements the steps of: acquiring agent description information and transmission link information of each client and storing the agent description information and the transmission link information into a preset agent database, wherein the transmission link information at least comprises one transmission link corresponding to the client; if a data acquisition request from any one of the request receiving interfaces is received, acquiring a transmission link corresponding to the data acquisition request in the proxy database as a target transmission link; acquiring agent description information of a client corresponding to the target transmission link as target agent description information, and generating acquisition information corresponding to the target agent description information according to the data acquisition request; acquiring a data result corresponding to the acquired information according to the target transmission link; and storing the data result to the proxy database, and feeding back the data result according to the request receiving interface.
In an embodiment, the step of obtaining and storing the agent description information and the transmission link information of each client in a preset agent database includes: sending a terminal address acquisition request to each client to acquire terminal address information fed back by each client according to the terminal address; establishing a transmission link with each client according to the terminal address information; and acquiring corresponding agent description information and transmission link information from each client and storing the corresponding agent description information and transmission link information in a preset agent database.
In an embodiment, after the step of acquiring a transmission link corresponding to the data acquisition request in the proxy database as a target transmission link if the data acquisition request from any of the request receiving interfaces is received, the method further includes: and storing the data acquisition request into a storage area corresponding to the storage address in the proxy database according to the storage address of the client to which the target transmission link belongs.
In an embodiment, the step of storing the data result in the proxy database and feeding back the data result according to the request receiving interface includes: and storing the data result into a storage area corresponding to the storage address of the data acquisition request in the proxy database.
In an embodiment, the step of acquiring the data result corresponding to the acquisition information according to the target transmission link includes: sending the acquired information to a client corresponding to the target transmission link through a write protocol of the target transmission link; and receiving a data result fed back by the user side according to the acquired information according to the read protocol of the target transmission link.
It is clear to those skilled in the art that, for convenience and brevity of description, the specific working processes of the above-described apparatuses, devices and units may refer to the corresponding processes in the foregoing method embodiments, and are not described herein again. Those of ordinary skill in the art will appreciate that the elements and algorithm steps of the examples described in connection with the embodiments disclosed herein may be embodied in electronic hardware, computer software, or combinations of both, and that the components and steps of the examples have been described in a functional general in the foregoing description for the purpose of illustrating clearly the interchangeability of hardware and software. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the implementation. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present invention.
In the embodiments provided by the present invention, it should be understood that the disclosed apparatus, device and method can be implemented in other ways. For example, the above-described embodiments of the apparatus are merely illustrative, and for example, the division of the units is only a logical division, and there may be other divisions when the actual implementation is performed, or units having the same function may be grouped into one unit, for example, a plurality of units or components may be combined or may be integrated into another system, or some features may be omitted, or not executed. In addition, the shown or discussed mutual coupling or direct coupling or communication connection may be an indirect coupling or communication connection through some interfaces, devices or units, and may also be an electric, mechanical or other form of connection.
The units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of the embodiment of the present invention.
In addition, functional units in the embodiments of the present invention may be integrated into one processing unit, or each unit may exist alone physically, or two or more units are integrated into one unit. The integrated unit can be realized in a form of hardware, and can also be realized in a form of a software functional unit.
The integrated unit, if implemented in the form of a software functional unit and sold or used as a stand-alone product, may be stored in a computer readable storage medium. Based on such understanding, the technical solution of the present invention essentially contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product stored in a computer-readable storage medium, which includes several instructions for causing a computer device (which may be a personal computer, a server, or a network device) to execute all or part of the steps of the method according to the embodiments of the present invention. And the aforementioned computer-readable storage media comprise: various media capable of storing program codes, such as a usb disk, a removable hard disk, a Read-Only Memory (ROM), a magnetic disk, or an optical disk.
While the invention has been described with reference to specific embodiments, the invention is not limited thereto, and various equivalent modifications and substitutions can be easily made by those skilled in the art within the technical scope of the invention. Therefore, the protection scope of the present invention shall be subject to the protection scope of the claims.
Claims (10)
1. A data acquisition request processing method is applied to a management server, the management server is communicated with at least one client and a request receiving interface, and the method is characterized by comprising the following steps:
acquiring agent description information and transmission link information of each client and storing the agent description information and the transmission link information into a preset agent database, wherein the transmission link information at least comprises one transmission link corresponding to the client;
if a data acquisition request from any one of the request receiving interfaces is received, acquiring a transmission link corresponding to the data acquisition request in the proxy database as a target transmission link;
acquiring agent description information of a client corresponding to the target transmission link as target agent description information, and generating acquisition information corresponding to the target agent description information according to the data acquisition request;
acquiring a data result corresponding to the acquired information according to the target transmission link;
and storing the data result to the proxy database, and feeding back the data result according to the request receiving interface.
2. The method according to claim 1, wherein the acquiring and storing the agent description information and the transmission link information of each client in a preset agent database comprises:
sending a terminal address acquisition request to each client to acquire terminal address information fed back by each client according to the terminal address;
establishing a transmission link with each client according to the terminal address information;
and acquiring corresponding agent description information and transmission link information from each client, storing the corresponding agent description information and transmission link information into a preset agent database, and storing the corresponding agent description information and transmission link information into the preset agent database.
3. The method according to claim 1, wherein after the obtaining of the transmission link corresponding to the data obtaining request in the proxy database is taken as a target transmission link, the method further comprises:
and storing the data acquisition request into a storage area corresponding to the storage address in the proxy database according to the storage address of the client to which the target transmission link belongs.
4. The method of claim 3, wherein the storing the data results to the proxy database comprises:
and storing the data result into a storage area corresponding to the storage address of the data acquisition request in the proxy database.
5. The method of claim 1, wherein the target transmission link includes a write protocol process and a read protocol process, and the obtaining the data result corresponding to the obtaining information according to the target transmission link includes:
sending the acquired information to a client corresponding to the target transmission link through a write protocol of the target transmission link;
and receiving a data result fed back by the user side according to the acquired information according to the read protocol of the target transmission link.
6. A data acquisition request processing apparatus, comprising:
the client information storage unit is used for acquiring the agent description information and the transmission link information of each client and storing the agent description information and the transmission link information into a preset agent database, wherein the transmission link information at least comprises one transmission link corresponding to the client;
a target transmission link obtaining unit, configured to, if a data obtaining request from any one of the request receiving interfaces is received, obtain a transmission link, corresponding to the data obtaining request, in the proxy database as a target transmission link;
an acquisition information generating unit, configured to acquire proxy description information of a client corresponding to the target transmission link as target proxy description information, and generate acquisition information corresponding to the target proxy description information according to the data acquisition request;
the data result acquisition unit is used for acquiring a data result corresponding to the acquisition information according to the target transmission link;
and the data result storage and feedback unit is used for storing the data result to the proxy database and feeding back the data result according to the request receiving interface.
7. The data acquisition request processing apparatus according to claim 6, wherein the client information storage unit includes:
a terminal address information obtaining unit, configured to send a terminal address obtaining request to each client, so as to obtain terminal address information fed back by each client according to the terminal address;
a transmission link establishing unit, configured to establish a transmission link with each client according to the terminal address information;
and the client information acquisition unit is used for acquiring corresponding agent description information and transmission link information from each client and storing the corresponding agent description information and transmission link information into a preset agent database.
8. The data acquisition request processing apparatus according to claim 6, further comprising:
and the data acquisition request storage unit is used for storing the data acquisition request into a storage area corresponding to the storage address in the proxy database according to the storage address of the client to which the target transmission link belongs.
9. A computer device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, characterized in that the processor implements the data acquisition request processing method according to any one of claims 1 to 5 when executing the computer program.
10. A computer-readable storage medium, characterized in that the computer-readable storage medium stores a computer program which, when executed by a processor, causes the processor to execute a data acquisition request processing method according to any one of claims 1 to 5.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911047636.0A CN110826001A (en) | 2019-10-30 | 2019-10-30 | Data acquisition request processing method and device, computer equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911047636.0A CN110826001A (en) | 2019-10-30 | 2019-10-30 | Data acquisition request processing method and device, computer equipment and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN110826001A true CN110826001A (en) | 2020-02-21 |
Family
ID=69551598
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201911047636.0A Pending CN110826001A (en) | 2019-10-30 | 2019-10-30 | Data acquisition request processing method and device, computer equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110826001A (en) |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112822284A (en) * | 2021-01-22 | 2021-05-18 | 平安壹钱包电子商务有限公司 | Data processing method and device based on dynamic proxy and computer equipment |
| CN113709232A (en) * | 2021-08-24 | 2021-11-26 | 广州文远知行科技有限公司 | Data packet loading method, client agent, device and storage medium |
| CN114356335A (en) * | 2021-12-24 | 2022-04-15 | 北京字跳网络技术有限公司 | Data processing method, device, equipment and medium |
| WO2024092944A1 (en) * | 2022-11-03 | 2024-05-10 | 之江实验室 | Data processing method and apparatus, storage medium, and electronic device |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CA2267755A1 (en) * | 1996-09-24 | 1998-04-02 | Intervoice Limited Partnership | System and method for establishing a real-time agent pool between computer systems |
| CN105915658A (en) * | 2016-07-04 | 2016-08-31 | 上海优刻得信息科技有限公司 | Data calling method for acquiring client IP address and data transmission method |
| CN106254579A (en) * | 2016-09-30 | 2016-12-21 | 携程计算机技术（上海）有限公司 | A kind of dynamic IP agent pool and building and management method |
| CN107395782A (en) * | 2017-07-19 | 2017-11-24 | 北京理工大学 | A kind of IP limitation controlled source information extraction methods based on agent pool |
| CN109474710A (en) * | 2018-10-30 | 2019-03-15 | 北京网众共创科技有限公司 | Obtain the method and device of information |
| CN110287395A (en) * | 2019-07-01 | 2019-09-27 | 杭州安恒信息技术股份有限公司 | A kind of crawler method, apparatus, system, equipment and readable storage medium storing program for executing |
-
2019
- 2019-10-30 CN CN201911047636.0A patent/CN110826001A/en active Pending
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CA2267755A1 (en) * | 1996-09-24 | 1998-04-02 | Intervoice Limited Partnership | System and method for establishing a real-time agent pool between computer systems |
| CN105915658A (en) * | 2016-07-04 | 2016-08-31 | 上海优刻得信息科技有限公司 | Data calling method for acquiring client IP address and data transmission method |
| CN106254579A (en) * | 2016-09-30 | 2016-12-21 | 携程计算机技术（上海）有限公司 | A kind of dynamic IP agent pool and building and management method |
| CN107395782A (en) * | 2017-07-19 | 2017-11-24 | 北京理工大学 | A kind of IP limitation controlled source information extraction methods based on agent pool |
| CN109474710A (en) * | 2018-10-30 | 2019-03-15 | 北京网众共创科技有限公司 | Obtain the method and device of information |
| CN110287395A (en) * | 2019-07-01 | 2019-09-27 | 杭州安恒信息技术股份有限公司 | A kind of crawler method, apparatus, system, equipment and readable storage medium storing program for executing |
Non-Patent Citations (1)
| Title |
|---|
| 梁银亮: "《全图解黑客攻防实战图解视听版》", 31 March 2019, 中国铁道出版社 * |
Cited By (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112822284A (en) * | 2021-01-22 | 2021-05-18 | 平安壹钱包电子商务有限公司 | Data processing method and device based on dynamic proxy and computer equipment |
| CN113709232A (en) * | 2021-08-24 | 2021-11-26 | 广州文远知行科技有限公司 | Data packet loading method, client agent, device and storage medium |
| CN113709232B (en) * | 2021-08-24 | 2024-05-17 | 广州文远知行科技有限公司 | Data packet loading method, client agent, device and storage medium |
| CN114356335A (en) * | 2021-12-24 | 2022-04-15 | 北京字跳网络技术有限公司 | Data processing method, device, equipment and medium |
| WO2024092944A1 (en) * | 2022-11-03 | 2024-05-10 | 之江实验室 | Data processing method and apparatus, storage medium, and electronic device |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110826001A (en) | Data acquisition request processing method and device, computer equipment and storage medium | |
| CN108833521B (en) | Message pushing method, device, system, computer equipment and storage medium | |
| CN106850402B (en) | Message transmission method and device | |
| CN110928491B (en) | Storage partition dynamic selection method, system, computer equipment and storage medium | |
| CN109586969B (en) | Content distribution network disaster tolerance method and device, computer equipment and storage medium | |
| RU2498520C2 (en) | Method of providing peer-to-peer communication on web page | |
| CN108377247B (en) | Message pushing method and device | |
| CN111917900B (en) | Domain name agent request processing method and device | |
| CN108965203A (en) | A kind of resource access method and server | |
| KR102208935B1 (en) | Messaging api over http protocol to establish context for data exchange | |
| CN111625301A (en) | Idempotent processing method, apparatus, device and storage medium | |
| CN109154968B (en) | System and method for secure and efficient communication within an organization | |
| US20170019484A1 (en) | System and method for aggregating communication connections | |
| CN105100263A (en) | Reverse proxy method and apparatus | |
| US11843642B1 (en) | Serverless signaling in peer-to-peer session initialization | |
| US10320951B2 (en) | Systems and methods for establishing a virtual local area network | |
| CN108880972B (en) | Information processing method, server and terminal | |
| CN104320328A (en) | Message synchronization method, terminal and server | |
| CN114500630A (en) | Method, apparatus, system, storage medium and electronic device for message push | |
| CN113190452B (en) | Dial test service system, dial test service method, dial test service equipment and storage medium | |
| CN116760579A (en) | Data transmission methods, devices, storage media and systems based on industrial Internet | |
| US9106596B2 (en) | Method and apparatus of configuring a data broadcast service | |
| US20130226984A1 (en) | Method and apparatus of providing optimized web browser communications | |
| CN110266705B (en) | Control method and system | |
| US20170223136A1 (en) | Any Web Page Reporting and Capture |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20200221 |