CN118041974A - Flume server connection safety reconnection method based on exponential backoff implementation - Google Patents
Flume server connection safety reconnection method based on exponential backoff implementation Download PDFInfo
- Publication number
- CN118041974A CN118041974A CN202410112985.0A CN202410112985A CN118041974A CN 118041974 A CN118041974 A CN 118041974A CN 202410112985 A CN202410112985 A CN 202410112985A CN 118041974 A CN118041974 A CN 118041974A
- Authority
- CN
- China
- Prior art keywords
- connection
- detection
- exponential backoff
- reconnection
- server
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 25
- 238000001514 detection method Methods 0.000 claims abstract description 54
- 230000007246 mechanism Effects 0.000 claims description 11
- 239000002699 waste material Substances 0.000 abstract description 4
- 238000004364 calculation method Methods 0.000 abstract description 3
- 230000008569 process Effects 0.000 description 4
- 210000003813 thumb Anatomy 0.000 description 3
- 239000003795 chemical substances by application Substances 0.000 description 2
- 230000002776 aggregation Effects 0.000 description 1
- 238000004220 aggregation Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 238000005259 measurement Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000012546 transfer Methods 0.000 description 1
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/14—Session management
- H04L67/141—Setup of application sessions
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L43/00—Arrangements for monitoring or testing data switching networks
- H04L43/08—Monitoring or testing based on specific metrics, e.g. QoS, energy consumption or environmental parameters
- H04L43/0805—Monitoring or testing based on specific metrics, e.g. QoS, energy consumption or environmental parameters by checking availability
- H04L43/0811—Monitoring or testing based on specific metrics, e.g. QoS, energy consumption or environmental parameters by checking availability by checking connectivity
Landscapes
- Engineering & Computer Science (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Environmental & Geological Engineering (AREA)
- Computer And Data Communications (AREA)
Abstract
The invention discloses a method for realizing secure reconnection of a Flume server based on exponential backoff, which comprises the following steps: step S1: starting an application, and initializing connection to a Flume server; step S2: detecting the connection state by connecting the detection thread through the connection pool; step S3: if the connection is successful, presetting detection interval time for repeated cycle detection; if the connection is unsuccessful, closing the detection thread connected before, and newly establishing connection detection thread to reconnect. The technical scheme of the invention reduces the pressure on the server, improves the stability of connection, carries out exponential delay according to the number of connection failures, improves the success rate of connection, reduces invalid connection attempts, avoids the waste of resources, only needs to carry out exponential delay according to the number of connection failures, does not need complex logic and calculation, and reduces the complexity of realization.
Description
Technical Field
The invention relates to the technical field of server connection, in particular to a method for realizing safe reconnection of a Flume server connection based on exponential backoff.
Background
The jume is a distributed, reliable, highly available log collection and aggregation system. It was developed and maintained by the Apache open source software Foundation. Its objective is to provide a simple, reliable, flexible method to collect, aggregate and move large-scale data.
The design idea of the jume is to process the data by means of a data stream. The user may define a data stream and then connect the source and destination to this data stream, and the jump will automatically transmit data from the source to the destination. The jume has high reliability and expandability, can process a large amount of data, and can automatically perform fault transfer and load balancing.
The jume client is part of a jume system for collecting and sending log data from a source (e.g., file, network, etc.) to a jume Agent. It is responsible for transmitting data to the flight Agent in a specified manner.
The thumb client may be configured and used in different ways, including writing a custom client using the Java API of the thumb, or using some predefined client component provided by the thumb, such as Avro, thrift, HTTP, etc.
The method of integrating with service application is the simplest and effective method when collecting service logs, the service can complete the collection of the service logs by integrating jar packets of the Flume client and carrying out corresponding configuration in log4j, but in actual large-scale application scenes, the service logs often have mass scale, and connection management of the service measurement Flume client and the Flume server often becomes very important. In the case of continuous reconnection of a client such as a connection failure of a service end node, precious resources of service application may be impacted, and normal operation of the service is finally affected.
Disclosure of Invention
Based on the problems, the invention provides a method for realizing the safe reconnection of the connection of the Flume server based on exponential backoff, which comprehensively considers that the service can be ensured to stably run under any condition, and the specific technical scheme is as follows.
The method for realizing the secure reconnection of the connection of the Flume server based on the exponential backoff comprises the following steps:
Step S1: starting an application, and initializing connection to a Flume server;
step S2: detecting the connection state by connecting the detection thread through the connection pool;
Step S3: if the connection is successful, presetting detection interval time for repeated cycle detection; if the connection is unsuccessful, closing the detection thread connected before, and newly establishing connection detection thread to reconnect.
Further, the method further comprises the following steps: if the newly-built connection detection thread is successfully reconnected, carrying out repeated cycle detection according to the preset detection interval time; and if the newly-built connection detection thread is not successfully reconnected, an exponential backoff detection mechanism is started.
Further, the exponential backoff detection mechanism performs exponential delay according to the number of connection failures, avoiding frequent connection attempts.
Further, in the exponential backoff detection mechanism, the preset detection interval time is extended with the exponential backoff interval, but does not exceed the preset maximum time interval.
Further, the step S1 further includes: when the application is started, a plurality of server addresses are configured, corresponding connections are established and put into a connection pool.
Further, each connection in the connection pool is established with a connection detection thread.
The invention has the beneficial effects that: the invention provides a method for realizing safe reconnection of a Flame server connection based on exponential backoff, which constructs a unified engine adapter, effectively reduces the frequency of connection retry, avoids frequent connection attempt, reduces the pressure on a server, improves the stability of connection, and carries out exponential delay according to the number of connection failures, so that a certain buffer time can be given to the server, the server has more opportunities to process, thereby improving the success rate of connection, reducing invalid connection attempt, avoiding resource waste, only needing to carry out exponential delay according to the number of connection failures, needing no complex logic and calculation, and reducing the realization complexity.
Drawings
Fig. 1 is a flowchart of a method for realizing connection security reconnection of a jume server based on exponential backoff in an embodiment of the present invention.
Detailed Description
For a clearer understanding of technical features, objects, and effects of the present invention, a specific embodiment of the present invention will be described with reference to the accompanying drawings.
The invention provides a method for realizing the safe reconnection of a Flume server based on exponential back-off, which comprises the following steps:
Step S1: starting an application, and initializing connection to a Flume server;
step S2: detecting the connection state by connecting the detection thread through the connection pool;
Step S3: if the connection is successful, presetting detection interval time for repeated cycle detection; if the connection is unsuccessful, closing the detection thread connected before, and newly establishing connection detection thread to reconnect.
In this embodiment, the secure reconnection method further includes: if the newly-built connection detection thread is successfully reconnected, carrying out repeated cycle detection according to the preset detection interval time; and if the newly-built connection detection thread is not successfully reconnected, an exponential backoff detection mechanism is started.
In this embodiment, the exponential backoff detection mechanism performs exponential delay according to the number of connection failures, avoiding frequent connection attempts. Under the exponential backoff detection mechanism, the preset detection interval time expands with the exponential backoff interval, but does not exceed the preset maximum time interval.
In this embodiment, step S1 further includes: when the application is started, a plurality of server addresses are configured, corresponding connections are established and put into a connection pool. Each connection in the connection pool is established with a connection detection thread.
In one embodiment, as shown in fig. 1, the implementation steps of the method for realizing the secure reconnection of the Flume server based on the exponential backoff are as follows:
(1) Initializing connection to a Flume server when an application is started, and setting up a plurality of connections to a connection pool after a plurality of server addresses are configured;
(2) Each connection in the connection pool establishes a connection detection thread;
(3) The detection thread detects the connection state once every 2 minutes;
(4) Reestablishing the connection when the detection fails;
(5) If the reestablishment connection fails, an exponential backoff detection mechanism is enabled, and then connection state detection is performed every 2 minutes, 4 minutes, 6 minutes; in this embodiment, the maximum interval is set to 6 minutes, and no index-based backoff interval is infinitely enlarged, so that the acquisition of service logs is affected when the interval is too large, and the use of service resources is affected when the interval is too small;
(6) The connection state is successfully recovered by the connection state detection;
(7) The exponential backoff detection mechanism is exited, and the connection state is detected once after 2 minutes.
In this embodiment, the exponential backoff algorithm may effectively reduce the frequency of connection retries, and avoid frequent connection attempts, thereby reducing the pressure on the server and improving the stability of connection. The following advantages are also provided:
The connection success rate is improved: the exponential backoff algorithm can delay exponentially according to the connection failure times, so that a certain buffer time can be given to the server, and the server has more opportunities to process, thereby improving the connection success rate.
The resource waste is reduced: the exponential backoff algorithm gradually increases the interval time of retry according to the number of connection failures, so that invalid connection attempts can be reduced, and waste of resources is avoided.
Adaptivity: the exponential backoff algorithm can automatically adjust the retry time interval according to the connection failure condition, and adapt to different network environments and server load conditions, thereby improving the connection adaptability.
Simple and easy to realize: the exponential backoff algorithm is a simple and easy-to-implement algorithm, and only needs to perform exponential delay according to the number of connection failures, so that complex logic and calculation are not needed, and the implementation complexity is reduced.
The foregoing has shown and described the basic principles and features of the invention and the advantages of the invention. It will be understood by those skilled in the art that the present invention is not limited to the embodiments described above, and that the above embodiments and descriptions are merely illustrative of the principles of the present invention, and various changes and modifications may be made without departing from the spirit and scope of the invention, which is defined in the appended claims. The scope of the invention is defined by the appended claims and equivalents thereof.
Claims (6)
1. The method for realizing the secure reconnection of the connection of the Flume server based on the exponential backoff is characterized by comprising the following steps:
Step S1: starting an application, and initializing connection to a Flume server;
step S2: detecting the connection state by connecting the detection thread through the connection pool;
Step S3: if the connection is successful, presetting detection interval time for repeated cycle detection; if the connection is unsuccessful, closing the detection thread connected before, and newly establishing connection detection thread to reconnect.
2. The method for realizing secure reconnection of a Flume server connection based on exponential backoff according to claim 1, further comprising: if the newly-built connection detection thread is successfully reconnected, carrying out repeated cycle detection according to the preset detection interval time; and if the newly-built connection detection thread is not successfully reconnected, an exponential backoff detection mechanism is started.
3. The method for realizing safe reconnection of a Flume server connection based on exponential backoff according to claim 2, wherein the exponential backoff detection mechanism performs exponential delay according to the number of connection failures, avoiding frequent connection attempts.
4. The method for realizing safe reconnection of a Flume server based on exponential backoff according to claim 3, wherein the preset detection interval time is extended with the exponential backoff interval but does not exceed a preset maximum time interval under the exponential backoff detection mechanism.
5. The method for realizing secure reconnection of a Flume server connection based on exponential backoff according to claim 1, wherein the step S1 further comprises: when the application is started, a plurality of server addresses are configured, corresponding connections are established and put into a connection pool.
6. The method for achieving secure reconnection of a Flume server connection based on exponential backoff according to claim 5, wherein each connection in the connection pool is established with a connection detection thread.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202410112985.0A CN118041974A (en) | 2024-01-26 | 2024-01-26 | Flume server connection safety reconnection method based on exponential backoff implementation |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202410112985.0A CN118041974A (en) | 2024-01-26 | 2024-01-26 | Flume server connection safety reconnection method based on exponential backoff implementation |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN118041974A true CN118041974A (en) | 2024-05-14 |
Family
ID=90999555
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202410112985.0A Pending CN118041974A (en) | 2024-01-26 | 2024-01-26 | Flume server connection safety reconnection method based on exponential backoff implementation |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN118041974A (en) |
-
2024
- 2024-01-26 CN CN202410112985.0A patent/CN118041974A/en active Pending
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| JP7234402B2 (en) | Secondary link monitoring method for vehicle communication, vehicle communication terminal, electronic device and computer program | |
| US20070130324A1 (en) | Method for detecting non-responsive applications in a TCP-based network | |
| CN110581852A (en) | Efficient mimicry defense system and method | |
| WO2016053561A1 (en) | Systems and methods for monitoring globally distributed remote storage devices | |
| CN111917451B (en) | Data transmission method and device, satellite module and storage medium | |
| CN111726413B (en) | Device connection method and device | |
| CN106598633B (en) | Configuration file updating method, client and server | |
| CN112769652B (en) | Node service monitoring method, device, equipment and medium | |
| CN114640703A (en) | Data communication method and device, electronic equipment and storage medium | |
| CN111711680A (en) | File breakpoint continuous transmission method and device based on UDP (user Datagram protocol) | |
| CN110912759A (en) | Automatic connection method and system for VPN network abnormity | |
| CN101594308B (en) | Method and system for transmitting message | |
| CN114500243B (en) | Data communication method, device, computer readable medium and electronic equipment | |
| CN110413435A (en) | A communication fault recovery method, system and related components | |
| CN108614750A (en) | The restoration methods and device of delay machine data | |
| CN112969244B (en) | Session recovery method and device | |
| US8972543B1 (en) | Managing clients utilizing reverse transactions | |
| CN114422428A (en) | Restarting method, device, electronic device and storage medium of service node | |
| CN118041974A (en) | Flume server connection safety reconnection method based on exponential backoff implementation | |
| CN101448022B (en) | Control method of system interface iSCSI receiving end window of internet small computer and device | |
| CN111918236B (en) | Internet of things security sensor network searching method, device, equipment and storage medium | |
| CN106123476A (en) | Intelligent detecting method for refrigerator | |
| CN107896176B (en) | Processing method of computing node, intelligent terminal and storage medium | |
| US20130036220A1 (en) | Apparatus and Method for Implementing Point to Point Protocol Data Service Link-Disconnect | |
| CN116260747A (en) | Monitoring method, device and electronic equipment of terminal testing equipment |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |