CN119520512B - A distributed cross-segment file transmission method - Google Patents
A distributed cross-segment file transmission method Download PDFInfo
- Publication number
- CN119520512B CN119520512B CN202510058883.XA CN202510058883A CN119520512B CN 119520512 B CN119520512 B CN 119520512B CN 202510058883 A CN202510058883 A CN 202510058883A CN 119520512 B CN119520512 B CN 119520512B
- Authority
- CN
- China
- Prior art keywords
- file
- transmission
- node
- module
- network
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/06—Protocols specially adapted for file transfer, e.g. file transfer protocol [FTP]
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L1/00—Arrangements for detecting or preventing errors in the information received
- H04L1/12—Arrangements for detecting or preventing errors in the information received by using return channel
- H04L1/16—Arrangements for detecting or preventing errors in the information received by using return channel in which the return channel carries supervisory signals, e.g. repetition request signals
- H04L1/18—Automatic repetition systems, e.g. Van Duuren systems
- H04L1/1867—Arrangements specially adapted for the transmitter end
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L12/00—Data switching networks
- H04L12/28—Data switching networks characterised by path configuration, e.g. LAN [Local Area Networks] or WAN [Wide Area Networks]
- H04L12/2801—Broadband local area networks
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L63/00—Network architectures or network communication protocols for network security
- H04L63/04—Network architectures or network communication protocols for network security for providing a confidential data exchange among entities communicating through data packet networks
- H04L63/0428—Network architectures or network communication protocols for network security for providing a confidential data exchange among entities communicating through data packet networks wherein the data content is protected, e.g. by encrypting or encapsulating the payload
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L9/00—Cryptographic mechanisms or cryptographic arrangements for secret or secure communications; Network security protocols
- H04L9/40—Network security protocols
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D30/00—Reducing energy consumption in communication networks
- Y02D30/50—Reducing energy consumption in communication networks in wire-line communication networks, e.g. low power modes or reduced link rate
Landscapes
- Engineering & Computer Science (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Computer Security & Cryptography (AREA)
- Computer Hardware Design (AREA)
- Computing Systems (AREA)
- General Engineering & Computer Science (AREA)
- Data Exchanges In Wide-Area Networks (AREA)
- Computer And Data Communications (AREA)
Abstract
The application discloses a distributed-based cross-network-segment file transmission method, which comprises the following steps of S1, distributing file transmission service in a local area network; the method comprises the steps of S2, registering transmission nodes among local area networks through file transmission service, S3, determining a transmission path according to whether a target node is included in the local area networks, S4, encrypting file fragments according to path information after the transmission path is acquired and sending the file fragments to the next transmission node, S5, decrypting data packets by a receiving node according to source node information and continuing to transmit the data packets according to route information until reaching a destination, S6, after receiving all fragment data, returning acknowledgement information of successful receiving to a sender by a receiving party, and S7, retransmitting the fragment data packets which are not acknowledged if the sender does not receive acknowledgement for a long time. The application not only improves the efficiency and the safety of file transmission, but also enhances the reliability and the stability of the system.
Description
Technical Field
The application relates to the field of file transmission, in particular to a distributed-based cross-network-segment file transmission method.
Background
With the development of informatization, trans-segment file transmission plays a vital role in enterprises and data centers. Common file transfer protocols include FTP, SFTP, HTTP, etc., which are mainly used for file transfer in the same network segment or in a close-range network environment. However, as the demand for trans-regional, trans-network segments increases, conventional transport protocols expose problems in trans-network segment transmission.
Aiming at the related technology, the inventor considers that the following defects are that the network configuration is complex, the traditional file transmission protocol realizes file transmission from one node to another node by establishing point-to-point connection, when the number of nodes in the network is increased, the complex network configuration needs to be established, the operation and maintenance cost is increased, the transmission reliability is insufficient, the stability of transmission is difficult to ensure due to network delay and packet loss, the trigger time and mechanism of a breakpoint continuous transmission function after the transmission failure are not flexible enough, the transmission efficiency is lower, and therefore, the improvement is needed.
Disclosure of Invention
Aiming at the defects of the prior art, the application provides a distributed-based cross-network-segment file transmission method, which aims to solve the problems.
A transmission method of a cross-network section file based on distribution comprises the following steps:
The method comprises the steps that S1, file transmission service is arranged in a local area network, so that the service can be communicated with all node networks in the local area network, and the file transmission service network communication with other nodes is required to be ensured;
step S2, before the file transmission authentication configuration is carried out, transmission node registration is carried out among all local area networks through file transmission service;
step S3, when the file is required to be transmitted from one node to another node, determining a transmission path according to whether the local area network contains a target node or not;
S4, after the transmission path is acquired, encrypting the file fragments according to the path information, and sending the file fragments to a next transmission node;
S5, the receiving node decrypts the data packet according to the source node information and continues to transmit according to the route information until reaching the destination;
Step S6, after receiving all the fragment data, the receiving party combines the fragment data into a complete file and returns confirmation information of successful receiving to the sending party;
and S7, if the sender does not receive the receiving confirmation for a long time, retransmitting the unacknowledged fragmented data packet.
By adopting the technical scheme, file transmission service is arranged in a local area network, the service can be communicated with all node networks in the local area network, meanwhile, the communication with file transmission service networks of other nodes is ensured, before file transmission authentication configuration is carried out, transmission node registration is carried out among all local area networks through the file transmission service, complex network configuration required by cross-network segment file transmission is simplified, operation and maintenance cost is reduced, the safety and the trust degree of a system are enhanced, when the file is required to be transmitted from one node to another node, an optimal transmission path is determined according to whether a target node is contained in the local area network, then the file is subjected to fragmentation processing and encryption, and then is sent to the next transmission node, and a receiving node decrypts a data packet according to source node information and continues to transmit according to route information until a destination is reached. And after receiving all the fragmented data, the receiver combines the fragmented data into a complete file, returns acknowledgement information of successful reception to the sender, and resends unacknowledged fragmented data packets if the acknowledgement is not received for a long time. The process not only improves the efficiency and the safety of file transmission, but also enhances the reliability and the stability of a system, supports large file transmission, simplifies error processing, optimizes resource utilization, and provides a solid foundation for realizing efficient, safe and easily-managed large-scale data transmission.
Optionally, the file transmission service includes a file transmission module, an encryption module, a message queue module and a transmission node management module, where the file transmission module is configured to perform fragmentation processing on a file and write the file into the message queue module, the encryption module performs encryption processing on content of the file transmission, the message queue module is configured to serve as a hub for content transmission of the fragmented file, and the transmission node management module is configured to store a mutually authenticated node list, and obtain a transmission path before file transmission.
By adopting the technical scheme, the file transmission module is responsible for carrying out slicing processing on the file and writing the file into the message queue module, so that the transmission efficiency is improved, the memory occupation is reduced, the file is transmitted while being read, the encryption component carries out encryption processing on the file content, the confidentiality and the integrity of the data in the transmission process are ensured, unauthorized access and data tampering are prevented, the message queue module is used as a hub for slicing file content transmission, effectively manages and schedules the data slicing to be transmitted, optimizes the system throughput, supports the breakpoint continuous transmission function, improves the reliability and the stability of the system, the transmission node management module stores mutually authenticated node lists, acquires the optimal path before file transmission, simplifies the complex network configuration required by cross-network-section file transmission, and reduces the operation and maintenance cost.
Optionally, the file transfer module supports registering Minio, FTP, SFTP types of file data sources.
Optionally, the specific steps for encrypting the file in step S4 are as follows:
s4.1, registering each other between transmission nodes, and storing a public key and a private key of the other party and an access mode of a message queue module;
s4.2, when the file transmission module needs to send the file, encrypting the file fragments by using the public key of the receiver;
S4.3, writing the encrypted file fragments into a message queue module and sending the file fragments to a next transmission node;
and S4.4, the receiving node reads the data packet from the message queue module and decrypts the data packet by using the corresponding private key according to the source node information.
By adopting the technical scheme, the security of the file content is enhanced, the fact that even if data is intercepted, an unauthorized third party cannot read the file content is ensured, meanwhile, a verification means of data integrity is provided, potential data damage or tampering can be detected, in addition, the mechanism realizes security authentication among nodes, reduces the risk of unauthorized access, simplifies key management, keeps the encryption state all the time on the whole transmission link until reaching a final destination, thereby ensuring the end-to-end security, the encryption method also supports a breakpoint continuous transmission function, and can continuously and safely transmit the file based on the existing key without renegotiating the key even after the network is interrupted.
Optionally, the access manner of the message queue module in step S4.1 includes IP, port, user name and password.
The size of the fragments is defined according to the transmitted file size, specifically, the file with the file size smaller than 10MB is set as a small file, the fragment size is set as 512KB-1MB, the file with the file size of 10MB-1GB is set as a medium file, the fragment size is set as 4MB-16MB, the file with the file size larger than 1MB is set as a large file, and the fragment size is set as 16MB-32MB.
By adopting the technical scheme, the method can optimize the network bandwidth utilization, improve the transmission efficiency and enhance the fault tolerance, small fragments can rapidly finish transmission for small files, and large fragments reduce the total fragments and speed up the whole transmission speed for large files, and in addition, when transmission errors or network interruption occur, only specific fragments which are not successfully transmitted are required to be retransmitted instead of the whole files, so that the recovery process is accelerated. The strategy also simplifies the file transfer management flow, facilitates configuration and maintenance, and improves user experience, particularly in applications where large amounts of data are handled or time sensitive. In summary, adjusting the size of the fragments based on the file size not only improves the efficiency and reliability of file transmission, but also enhances the flexibility and user satisfaction of the system.
Optionally, when file data is read from the file data source, the specified file is read by a client provided by the file data source, converted into a file stream, and converted into a byte array when read in slices.
By adopting the technical scheme, the efficiency and reliability of file transmission are obviously improved, the method supports transmission while reading, reduces memory occupation, enables a system to process files with any size without causing memory overflow, and simultaneously enhances the fault tolerance and recovery efficiency of the system, because only specific fragments which are not successfully transmitted but the whole file are required to be retransmitted even when network interruption or error occurs. In addition, the network bandwidth utilization is optimized by dynamically adjusting the sizes of the fragments, the problem of congestion or packet loss caused by large data packets is reduced, each fragment is treated as an independent data unit, an error detection and correction mechanism is simplified, the problem of single fragment is solved, finally, clients (such as Minio SDK, FTPCLIENT of Apache Commons Net and the like) provided by file data sources are used for accessing different types of storage services, support of various file protocols is ensured, and the flexibility and compatibility of the scheme are improved.
Optionally, the content transmitted to the message queue module by the file transmission module is a byte stream, and the byte stream includes a route length section, a route content section, a source node section, a fragment sequence number section, a file length section, a file content section and a file hash value section.
By adopting the technical scheme, when the file transmission assembly reads the data packet, the length of the route content is acquired through the route length section, the route content section is read according to the length, then the source node section is read, the source node number is acquired, then the fragment sequence number section is read, the index of the file fragment is acquired, then the file length section is read, the file content section is read according to the file length, finally the file hash value section is read, and whether the file is complete after the file is merged is judged.
Optionally, the file transmission module monitors the data in the message queue module, and once a new file fragment is detected, the file transmission module immediately consumes and parses the file and transmits the file fragment to the next node.
By adopting the technical scheme, the file transmission component monitors the data in the message queue module, and immediately consumes, analyzes and transmits the data to the next node when detecting the new file fragments, and the mechanism remarkably improves the efficiency and the reliability of file transmission. By processing newly arrived data in real time, the waiting time of the data in the message queue module can be reduced, and the speed of the whole file transmission process can be accelerated. Meanwhile, the instant processing mode is beneficial to quickly finding and responding to possible transmission errors or failures, and supports a breakpoint continuous transmission function, namely, after the network is interrupted, the transmission can be continued from the last successfully transmitted fragment without retransmitting the whole file, so that the success rate of transmitting the large file across network segments is improved. In addition, dividing the file into multiple small blocks for processing not only optimizes the use of network bandwidth and other computing resources, but also simplifies error handling logic so that most of the data is guaranteed to be properly transmitted even in the case of poor network conditions. Asynchronous operation is supported based on the design of the message queue module, the file transmission service and other services are allowed to work in parallel, and the concurrent processing capacity of the system is improved.
In summary, the present application includes at least one of the following beneficial technical effects:
1. The file transmission service is arranged in a local area network, the service can be communicated with all node networks in the local area network, meanwhile, the file transmission service network communication with other nodes is ensured, before file transmission authentication configuration is carried out, transmission node registration is carried out among all local area networks through the file transmission service, complex network configuration required by cross-network segment file transmission is simplified, operation and maintenance cost is reduced, safety and trust degree of a system are enhanced, when the file is required to be transmitted from one node to another node, an optimal transmission path is determined according to whether a target node is contained in the local area network, then the file is subjected to fragmentation processing and encryption, and then is sent to the next transmission node, and a receiving node decrypts a data packet according to source node information and continues transmission according to route information until a destination is reached. And after receiving all the fragmented data, the receiver combines the fragmented data into a complete file, returns acknowledgement information of successful reception to the sender, and resends unacknowledged fragmented data packets if the acknowledgement is not received for a long time. The process not only improves the efficiency and the safety of file transmission, but also enhances the reliability and the stability of a system, supports large file transmission, simplifies error processing, optimizes resource utilization, and provides a solid foundation for realizing efficient, safe and easily-managed large-scale data transmission.
2. The process enhances the security of the file content, and ensures that even if the data is intercepted, an unauthorized third party cannot read the file content; in addition, the mechanism realizes the security authentication among nodes, reduces the risk of unauthorized access, simplifies key management, keeps the encryption state all the time on the whole transmission link until reaching the final destination, thereby ensuring the end-to-end security, and the encryption method also supports the breakpoint continuous transmission function, can continuously and safely transmit the file based on the existing key without renegotiating the key even after the network is interrupted.
3. The method can optimize network bandwidth utilization, improve transmission efficiency and enhance fault tolerance, small fragments can rapidly finish transmission for small files, and large fragments reduce total fragments and speed up overall transmission speed for large files, and in addition, when transmission errors or network interruption occur, only specific fragments which are not successfully transmitted but not the whole files are retransmitted, so that recovery process is accelerated. The strategy also simplifies the file transfer management flow, facilitates configuration and maintenance, and improves user experience, particularly in applications where large amounts of data are handled or time sensitive. In summary, adjusting the size of the fragments based on the file size not only improves the efficiency and reliability of file transmission, but also enhances the flexibility and user satisfaction of the system.
4. The method supports transmission while reading, reduces memory occupation, enables a system to process files with any size without causing memory overflow, and enhances fault tolerance and recovery efficiency of the system because only specific fragments which are not successfully delivered but the whole file are required to be retransmitted even when network interruption or error occurs. In addition, the network bandwidth utilization is optimized by dynamically adjusting the sizes of the fragments, the problem of congestion or packet loss caused by large data packets is reduced, each fragment is treated as an independent data unit, an error detection and correction mechanism is simplified, the problem of single fragment is solved, finally, clients (such as Minio SDK, FTPCLIENT of Apache Commons Net and the like) provided by file data sources are used for accessing different types of storage services, support of various file protocols is ensured, and the flexibility and compatibility of the scheme are improved.
5. The file transmission component monitors the data in the message queue module, and immediately consumes, analyzes and transmits the data to the next node when detecting the new file fragments, and the mechanism remarkably improves the efficiency and the reliability of file transmission. By processing newly arrived data in real time, the waiting time of the data in the message queue module can be reduced, and the speed of the whole file transmission process can be accelerated. Meanwhile, the instant processing mode is beneficial to quickly finding and responding to possible transmission errors or failures, and supports a breakpoint continuous transmission function, namely, after the network is interrupted, the transmission can be continued from the last successfully transmitted fragment without retransmitting the whole file, so that the success rate of transmitting the large file across network segments is improved. In addition, the file is divided into a plurality of small blocks to process, so that the use of network bandwidth and other computing resources is optimized, error processing logic is simplified, most of data can be guaranteed to be correctly transmitted even under the condition of poor network conditions, asynchronous operation is supported based on the design of a message queue module, file transmission service and other services are allowed to work in parallel, and the concurrent processing capacity of a system is improved.
Drawings
FIG. 1 is a flow diagram of an embodiment of the present application.
Fig. 2 is a flow chart of a file transfer service in an embodiment of the application.
FIG. 3 is a schematic diagram of the format of content transmitted by the file transfer component module to the message queue module in accordance with an embodiment of the present application.
Detailed Description
In order that the application may be readily understood, a more particular description thereof will be rendered by reference to specific embodiments that are illustrated in the appended drawings. It will be understood that when an element is referred to as being "fixed" to another element, it can be directly on the other element or one or more intervening elements may be present therebetween. When an element is referred to as being "connected" to another element, it can be directly connected to the other element or one or more intervening elements may be present therebetween. The terms "vertical," "horizontal," "left," "right," and the like are used herein for illustrative purposes only.
Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used in the description of the application herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. The term "and/or" as used in this specification includes any and all combinations of one or more of the associated listed items.
The embodiment of the application discloses a transmission method of a cross-network-segment file based on distribution. Referring to fig. 1 and 2, the method comprises the steps of S1, arranging file transmission service in a local area network, ensuring that the service can communicate with all node networks in the local area network and needs to ensure that the service communicates with file transmission service networks of other nodes, S2, registering transmission nodes among the local area networks through the file transmission service before file transmission authentication configuration is carried out, S3, determining a transmission path according to whether a target node is contained in the local area network when the file needs to be transmitted from one node to the other node, S4, after the transmission path is acquired, encrypting the file fragments according to path information and sending the file fragments to the next transmission node, S5, decrypting a data packet according to source node information and continuing to transmit the data packet according to route information until reaching a destination, S6, combining the receiving party into a complete file and returning acknowledgement information of successful reception to a sender, and S7, if the sender does not receive acknowledgement for a long time, retransmitting the unacknowledged fragment data packet. The file transmission service is arranged in a local area network, the service can be communicated with all node networks in the local area network, meanwhile, the file transmission service network communication with other nodes is ensured, before file transmission authentication configuration is carried out, transmission node registration is carried out among all local area networks through the file transmission service, complex network configuration required by cross-network segment file transmission is simplified, operation and maintenance cost is reduced, safety and trust degree of a system are enhanced, when the file is required to be transmitted from one node to another node, an optimal transmission path is determined according to whether a target node is contained in the local area network, then the file is subjected to fragmentation processing and encryption, and then is sent to the next transmission node, and a receiving node decrypts a data packet according to source node information and continues transmission according to route information until a destination is reached. And after receiving all the fragmented data, the receiver combines the fragmented data into a complete file, returns acknowledgement information of successful reception to the sender, and resends unacknowledged fragmented data packets if the acknowledgement is not received for a long time. The process not only improves the efficiency and the safety of file transmission, but also enhances the reliability and the stability of a system, supports large file transmission, simplifies error processing, optimizes resource utilization, and provides a solid foundation for realizing efficient, safe and easily-managed large-scale data transmission.
Referring to fig. 1 and 2, the file transmission service includes a file transmission module, an encryption module, a message queue module, and a transmission node management module, where the file transmission module is configured to perform a slicing process on a registrable file data source and write the processed file data source into the message queue module, and the file transmission module supports registering Minio, FTP, SFTP types of file data sources. The encryption component encrypts the file transmission content, the message queue module is used as a hub for transmitting the fragmented file content, and the transmission node management module is used for storing a mutually authenticated node list and acquiring a transmission path before file transmission. The file transmission module is responsible for carrying out slicing processing on the file and writing the file into the message queue module, which not only improves the transmission efficiency, but also reduces the memory occupation, supports the transmission while reading, the encryption component carries out encryption processing on the file content, ensures the confidentiality and the integrity of the data in the transmission process, prevents unauthorized access and data tampering, the message queue module is used as a hub for slicing file content transmission, effectively manages and schedules the data slicing to be transmitted, optimizes the system throughput, supports the breakpoint continuous transmission function, improves the reliability and the stability of the system, and the transmission node management module stores mutually authenticated node lists, acquires the optimal path before file transmission, simplifies the complex network configuration required by cross-network file transmission, and reduces the operation and maintenance cost.
Referring to fig. 1 and 2, the file transfer module listens to the data in the message queue module, and once a new file fragment is detected, immediately consumes, parses, and transfers to the next node. The file transmission component monitors the data in the message queue module, and immediately consumes, analyzes and transmits the data to the next node when detecting the new file fragments, and the mechanism remarkably improves the efficiency and the reliability of file transmission. By processing newly arrived data in real time, the waiting time of the data in the message queue module can be reduced, and the speed of the whole file transmission process can be accelerated. Meanwhile, the instant processing mode is beneficial to quickly finding and responding to possible transmission errors or failures, and supports a breakpoint continuous transmission function, namely, after the network is interrupted, the transmission can be continued from the last successfully transmitted fragment without retransmitting the whole file, so that the success rate of transmitting the large file across network segments is improved. In addition, dividing the file into multiple small blocks for processing not only optimizes the use of network bandwidth and other computing resources, but also simplifies error handling logic so that most of the data is guaranteed to be properly transmitted even in the case of poor network conditions. Asynchronous operation is supported based on the design of the message queue module, the file transmission service and other services are allowed to work in parallel, and the concurrent processing capacity of the system is improved.
Referring to fig. 1 and 2, the specific steps of encrypting the file in step S4 include the steps of registering each other between the transmission nodes, storing the public key and the private key of the other party and the access mode of the message queue module, the step S4.2 of encrypting the file fragments by using the public key of the receiving party when the file transmission module needs to transmit the file, the step S4.3 of writing the encrypted file fragments into the message queue module and transmitting the encrypted file fragments to the next transmission node, and the step S4.4 of reading the data packet from the message queue module by the receiving node and decrypting the data packet by using the corresponding private key according to the source node information. Before the file transmission authentication configuration is carried out, transmission node registration is needed to be carried out among all local area networks through a file transmission module.
Referring to fig. 1 and 2, a file transfer component a needs to initiate an authentication request to a file transfer component B, the file transfer component a generates a public key a and a private key a, the file transfer component B generates a public key B and a private key B, the file transfer component a stores the private key a and the public key B, the file transfer component B stores the public key a and the private key B, and stores connection information of a transmitting node of an opposite party and an access mode (IP, port, user name, password) of a message queue module. The document delivery component B and the document delivery component C, the document delivery component B and the document delivery component D, and the document delivery component C and the document delivery component D also need to do this authentication. The public and private key pairs are generated through an RSA algorithm, encryption is carried out through the public key in the subsequent data transmission process, and the receiver decrypts through the private key. The process enhances the security of the file content, and ensures that even if the data is intercepted, an unauthorized third party cannot read the file content; in addition, the mechanism realizes the security authentication among nodes, reduces the risk of unauthorized access, simplifies key management, keeps the encryption state all the time on the whole transmission link until reaching the final destination, thereby ensuring the end-to-end security, and further supports the breakpoint continuous transmission function, and can continuously and safely transmit the file based on the existing key without renegotiating the key even after the network is interrupted.
Referring to fig. 1 and fig. 2, when a file on a node a needs to be transmitted to a node F, the file transmission service first confirms whether the local area network includes the node F, if so, the return transmission path is [ node a- > -file transmission component a- > -node F ], if the local area network does not include the node F, a request is sent to registered node devices (in this example, a file transmission component B is registered, an acquisition path request is sent to the file transmission component), and so on, the file transmission component B returns a path 1 [ file transmission component B- > -file transmission component C- > -node F ] and a path 2 [ file transmission component B- > -file transmission component C- > -node F ], and according to the length of the transmission path, the finally selected transmission path is [ node a- > -file transmission component C- > -node F ].
Referring to fig. 1,2 and 3, after a transmission path is acquired, the file transmission component a reads the file on the node a in slices, the sizes of the slices are defined according to the size of the transmitted file, the file with the file size smaller than 10MB is set as a small file, the size of the slices is set as 512KB-1MB, the file with the file size of 10MB-1GB is set as a medium file, the size of the slices is set as 4MB-16MB, the file with the file size larger than 1GB is set as a large file, and the size of the slices is set as 16MB-32MB. When file data is read from a file data source, a specified file is read, converted to a file stream, and converted to a byte array at the time of fragment reading, by a client (e.g., FTPCLIENT of Minio SDK, apache common net) provided by the file data source. The method can optimize network bandwidth utilization, improve transmission efficiency and enhance fault tolerance, small fragments can rapidly finish transmission for small files, and large fragments reduce total fragments and speed up overall transmission speed for large files, and in addition, when transmission errors or network interruption occur, only specific fragments which are not successfully transmitted but not the whole files are retransmitted, so that recovery process is accelerated. The strategy also simplifies the file transfer management flow, facilitates configuration and maintenance, and improves user experience, particularly in applications where large amounts of data are handled or time sensitive. The method supports transmission while reading, reduces memory occupation, enables a system to process files with any size without causing memory overflow, and enhances fault tolerance and recovery efficiency of the system because only specific fragments which are not successfully delivered but the whole file are required to be retransmitted even when network interruption or error occurs. In addition, the network bandwidth utilization is optimized by dynamically adjusting the sizes of the fragments, the problem of congestion or packet loss caused by large data packets is reduced, each fragment is treated as an independent data unit, an error detection and correction mechanism is simplified, the problem of single fragment is solved, finally, clients (such as Minio SDK, FTPCLIENT of Apache Commons Net and the like) provided by file data sources are used for accessing different types of storage services, support of various file protocols is ensured, and the flexibility and compatibility of the scheme are improved.
Referring to fig. 1,2 and 3, the contents transmitted to the message queue module by the file transmission module are byte streams, and the byte streams include a route length section, a route content section, a source node section, a fragment sequence number section, a file length section, a file content section and a file hash value section. When the file transmission component reads the data packet, the length of the route content is obtained through the route length section, the route content section is read according to the length, then the source node section is read, the source node number is obtained, then the fragment sequence number section is read, the index of the file fragment is obtained, then the file length section is read, the file content section is read according to the file length, finally the file hash value section is read, and whether the file is complete after the file is merged is judged. For example, when the file transmission component reads a data packet, the first 8 bits of the byte stream are acquired, the length of the routing content is acquired, the routing content is read according to the length, then the data of 32 bytes after reading is used for indicating the source node number, the next 8 bits are used for indicating the index of the file fragment, the next 8 bits are used for indicating the file length, the file content is acquired according to the file length, and the last 256 bits are used for indicating the hash value of the file, and the hash value is used for judging whether the file is complete after the file is merged.
The implementation principle of the distributed cross-network file transmission method is that file transmission service is arranged in a local area network, the service can be communicated with all node networks in the local area network, meanwhile, the file transmission service network communication with other nodes is guaranteed, before file transmission authentication configuration is carried out, transmission node registration is carried out among all local area networks through the file transmission service, complex network configuration required by cross-network file transmission is simplified, operation and maintenance cost is reduced, safety and trust degree of a system are enhanced, when a file is required to be transmitted from one node to another node, an optimal transmission path is determined according to whether a target node is contained in the local area network, then the file is subjected to fragmentation processing and encryption, and then the file is sent to the next transmission node, and a receiving node decrypts a data packet according to source node information and continues transmission according to route information until a destination is reached. And after receiving all the fragmented data, the receiver combines the fragmented data into a complete file, returns acknowledgement information of successful reception to the sender, and resends unacknowledged fragmented data packets if the acknowledgement is not received for a long time. The process not only improves the efficiency and the safety of file transmission, but also enhances the reliability and the stability of a system, supports large file transmission, simplifies error processing, optimizes resource utilization, and provides a solid foundation for realizing efficient, safe and easily-managed large-scale data transmission.
The above embodiments are not intended to limit the scope of the application, so that the equivalent changes of the structure, shape and principle of the application are covered by the scope of the application.
Claims (9)
1. A transmission method of a cross-network section file based on distribution is characterized by comprising the following steps:
The method comprises the steps that S1, file transmission service is arranged in a local area network, so that the service can be communicated with all node networks in the local area network, and the file transmission service network communication with other nodes is required to be ensured;
Step S2, before the file transmission authentication configuration is carried out, all local area networks register each other among transmission nodes through file transmission service;
step S3, when the file is required to be transmitted from one node to another node, determining a transmission path according to whether the local area network contains the target node, if the local area network does not contain the target node, sending a request to a registered node, and returning the transmission path by a file transmission component in the registered node;
S4, after the transmission path is acquired, encrypting the file fragments according to the path information, and sending the file fragments to a next transmission node;
S5, the receiving node decrypts the data packet according to the source node information and continues to transmit according to the route information until reaching the destination;
Step S6, after receiving all the fragment data, the receiving party combines the fragment data into a complete file and returns confirmation information of successful receiving to the sending party;
and S7, if the sender does not receive the receiving confirmation for a long time, retransmitting the unacknowledged fragmented data packet.
2. The method for transmitting the file based on the distributed cross-network segment according to claim 1, wherein the file transmission service comprises a file transmission module, an encryption module, a message queue module and a transmission node management module, wherein the file transmission module is used for carrying out fragmentation processing on the file and writing the file into the message queue module, the encryption module is used for carrying out encryption processing on the content of the file transmission, the message queue module is used as a hub for transmitting the content of the fragmented file, and the transmission node management module is used for storing a mutually authenticated node list and acquiring a transmission path before the file transmission.
3. The method for transmitting files across network segments based on distributed as set forth in claim 2, wherein the file transmission module supports registering Minio, FTP, SFTP types of file data sources.
4. The method for transmitting the file across the network segments based on the distribution type according to claim 3, wherein the specific steps of encrypting the file in the step S4 are as follows:
s4.1, registering each other between transmission nodes, and storing a public key and a private key of the other party and an access mode of a message queue module;
s4.2, when the file transmission module needs to send the file, encrypting the file fragments by using the public key of the receiver;
S4.3, writing the encrypted file fragments into a message queue module and sending the file fragments to a next transmission node;
and S4.4, the receiving node reads the data packet from the message queue module and decrypts the data packet by using the corresponding private key according to the source node information.
5. The method of claim 4 wherein the access mode of the message queue module in step S4.1 includes IP, port, user name and password.
6. The method for transmitting the cross-network-segment file based on the distribution type of claim 2, wherein the size of the fragments is customized according to the size of the transmitted file, specifically, the file with the file size smaller than 10MB is set as a small file, the size of the fragments is set as 512KB-1MB, the file with the file size of 10MB-1GB is set as a medium file, the size of the fragments is set as 4MB-16MB, the file with the file size larger than 1GB is set as a large file, and the size of the fragments is set as 16MB-32MB.
7. The method of claim 6, wherein the client provided by the file data source reads the specified file, converts the specified file into a file stream, and converts the file stream into a byte array when reading the file data from the file data source in a fragmented manner.
8. The method for transmitting files across network segments based on distribution type as set forth in claim 7, wherein the content transmitted to the message queue module by the file transmission module is a byte stream, and the byte stream includes a route length segment, a route content segment, a source node segment, a fragment sequence number segment, a file length segment, a file content segment and a file hash value segment.
9. The method for transmitting files across network segments based on distributed mode as set forth in claim 8, wherein the file transmission module listens to data in the message queue module, and once a new file fragment is detected, the file is consumed and parsed immediately and transmitted to the next node.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202510058883.XA CN119520512B (en) | 2025-01-15 | 2025-01-15 | A distributed cross-segment file transmission method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202510058883.XA CN119520512B (en) | 2025-01-15 | 2025-01-15 | A distributed cross-segment file transmission method |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN119520512A CN119520512A (en) | 2025-02-25 |
| CN119520512B true CN119520512B (en) | 2025-06-17 |
Family
ID=94657879
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202510058883.XA Active CN119520512B (en) | 2025-01-15 | 2025-01-15 | A distributed cross-segment file transmission method |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN119520512B (en) |
Families Citing this family (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN120281765A (en) * | 2025-06-12 | 2025-07-08 | 中国兵器装备集团兵器装备研究所 | Cross-domain large file transmission method based on distributed soft bus |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103747091A (en) * | 2014-01-16 | 2014-04-23 | 电信科学技术第一研究所 | Distributed data synchronization system and method for embedded equipment |
| CN112235193A (en) * | 2020-10-12 | 2021-01-15 | 南威软件股份有限公司 | Data transmission method, device, equipment and medium based on cross-network multi-level routing |
Family Cites Families (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| KR100358387B1 (en) * | 2000-06-27 | 2002-10-25 | 엘지전자 주식회사 | Apparatus for extended firewall protecting internal resources in network system |
| US8085741B2 (en) * | 2004-03-10 | 2011-12-27 | Core Wireless Licensing S.A.R.L. | System and method for pushing content to a terminal utilizing a network-initiated data service technique |
| EP1912404B1 (en) * | 2006-10-11 | 2011-06-01 | Murata Machinery, Ltd. | File transfer server |
| CN106230817B (en) * | 2016-07-29 | 2019-04-02 | 中国电子科技集团公司第二十八研究所 | Distributed mass data transmission method and system |
| CN111182047B (en) * | 2019-12-24 | 2023-04-07 | 中国银联股份有限公司 | Method and system for transferring files between large data platforms across a network |
| CN111314480B (en) * | 2020-02-26 | 2022-09-20 | 烟台海颐软件股份有限公司 | Load self-adaptive cross-platform file transfer protocol distributed service implementation method |
| CN116305194B (en) * | 2023-02-15 | 2023-11-17 | 中国科学院空天信息创新研究院 | A method and system for asymmetric encryption and decryption of sustainable information disclosure data |
| CN116471283B (en) * | 2023-06-20 | 2023-08-25 | 北京中宏立达科技发展有限公司 | File transmission method |
| CN118573672A (en) * | 2024-07-30 | 2024-08-30 | 杭州光芯科技有限公司 | File transmission method, device, medium and equipment |
| CN119299165A (en) * | 2024-10-10 | 2025-01-10 | 中控技术股份有限公司 | A method, device, storage medium and electronic device for forwarding data across network segments |
-
2025
- 2025-01-15 CN CN202510058883.XA patent/CN119520512B/en active Active
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103747091A (en) * | 2014-01-16 | 2014-04-23 | 电信科学技术第一研究所 | Distributed data synchronization system and method for embedded equipment |
| CN112235193A (en) * | 2020-10-12 | 2021-01-15 | 南威软件股份有限公司 | Data transmission method, device, equipment and medium based on cross-network multi-level routing |
Also Published As
| Publication number | Publication date |
|---|---|
| CN119520512A (en) | 2025-02-25 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US11522838B2 (en) | Secure end-to-end transport through in intermediary nodes | |
| US7948921B1 (en) | Automatic network optimization | |
| EP1494426B1 (en) | Secure network processing | |
| CN100552626C (en) | Method for synchronizing and uploading offloaded network stack connections with network stack | |
| CN119520512B (en) | A distributed cross-segment file transmission method | |
| US7120792B1 (en) | System and method for secure communication of routing messages | |
| EP1469653A2 (en) | Object aware transport-layer network processing engine | |
| CN1640093B (en) | Method and system for accelerating the conversion process between encryption schemes | |
| EP1494424B1 (en) | System and method for message-based scalable data transport | |
| US20100185586A1 (en) | Message-based scalable data transport protocol | |
| US20060010245A1 (en) | Internet protocol for the delivery of complex digital media content | |
| CN112152914A (en) | Instant messaging method and system based on Beidou short message | |
| CN114095499B (en) | Neutral verification method and device for block chain relay communication network | |
| CN108429682A (en) | Method and system for optimizing network transmission link | |
| EP1210808B1 (en) | Packet authentication | |
| CN115118526B (en) | VR device data migration method | |
| US9877357B2 (en) | Changing wireless carriers during a mobile gateway session | |
| KR20180028648A (en) | Apparatus for one-way data transmission, apparatus for one-way data reception, and one-way data transmission method for using the same | |
| JP3618508B2 (en) | Receiving protocol device and broadcast message transmitting device | |
| US12418584B1 (en) | Computer-implemented method and a system for synchronizing data messages between multiple network nodes of a communication network | |
| US20230179656A1 (en) | Method for synchronising data of a database, computer programme, device for processing data, and mobile terminal therefor | |
| Silva et al. | Delay and disruption tolerant network architecture | |
| Pinto et al. | HTTP-DTNSec: An HTTP-Based Security Extension for Delay/Disruption Tolerant Networking | |
| KR100921491B1 (en) | Lossless message transmission method in ring network | |
| Maatkamp et al. | Unidirectional secure information transfer via RabbitMQ |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |