CN119030967A - A method and system for user data transmission based on distributed network - Google Patents

Abstract

The invention discloses a method and a system for transmitting user data based on a distributed network, wherein a server receives data sent by a first client for the first time and converts the data into dictionary format data so as to store the common structural characteristics of the data through the dictionary format data and compress and simplify the data. When the second client sends a downloading request for data, dictionary format data is downloaded from the server and restored to the original data format through a formula, so that the second client decompresses the appointed data through the dictionary, and repeated downloading of the repeated part of data is not needed during data transmission; the data with the same structure has a common structure, a dictionary-data storage technology is established for realizing the conversion of data 'slimming' and data dictionary format, and meanwhile, the transmission mode is not required to be changed through the conversion of the data dictionary format instead of data encryption, so that the data transmission efficiency is improved, and the safety of data transmission is ensured.

Description

A method and system for user data transmission based on distributed network

Technical Field

The present invention relates to the technical field of data transmission, and in particular to a method and system for user data transmission based on a distributed network.

Background Art

In the current information technology field, with the explosive growth of data volume, data storage and access efficiency have become one of the core challenges of distributed system design. Distributed file systems, as key technologies for handling large-scale data storage and access, significantly improve data reliability and access speed by distributing data across multiple nodes. However, in an environment with high concurrent data requests, traditional distributed file systems still face many bottlenecks in response speed and data processing capabilities.

Data compression technology plays a vital role in distributed systems. By reducing the storage space occupied by data, compression technology can not only reduce storage costs, but also improve the efficiency of data transmission and access to a certain extent. In mainstream operating systems such as Windows, data compression technology has been widely used, but most of them focus on the compression of single files or data blocks, and lack an overall solution optimized for distributed environments.

Although existing technologies such as Windows data compression schemes perform well in single-node environments, they still have obvious shortcomings in distributed file systems:

1. Current technologies mainly focus on data compression itself, that is, reducing the physical size of data through algorithms, but often ignore the targeted optimization of common file types or data features. This "one-size-fits-all" compression method cannot fully utilize the characteristics of different data types, thus limiting the compression efficiency and effect.

2. Although existing distributed file systems support data compression, they usually only stay at the level of data compression storage and lack effective management and organization of compressed files. This means that when accessing data, the system needs extra time to decompress the data, which in turn affects the overall access speed.

3. As data security becomes increasingly important, data encryption has become an indispensable function. However, existing technologies often find it difficult to achieve effective data encryption while ensuring fast data access. This lack of security makes distributed file systems a potential risk in storing sensitive data.

4. In the high-concurrency data request mode, the distributed file system needs to quickly respond to a large number of concurrent requests and provide stable data access services. However, the existing data compression and access mechanisms often cannot meet these requirements at the same time, resulting in a decrease in system performance.

In view of the above background and technical challenges, it is particularly important to develop a data compression and management solution that can be optimized for the characteristics of distributed file systems.

Summary of the invention

The technical problem to be solved by the present invention is to provide a method and system for user data transmission based on a distributed network, which can improve the data transmission efficiency and ensure the security of data transmission.

In order to solve the above technical problems, the technical solution adopted by the present invention is:

A method for transmitting user data based on a distributed network, comprising the steps of:

The server converts the received data from the first client into dictionary format data, wherein the dictionary format data includes structural attribute characteristics of the data and data content characteristics compressed by a formula, and stores the dictionary format data;

The second client downloads the dictionary format data from the server in response to the download request for the data, and restores the dictionary format data to the original data format through a formula.

In order to solve the above technical problems, another technical solution adopted by the present invention is:

A system for transmitting user data based on a distributed network, comprising a first client, a server, and a second client:

The server converts the received data from the first client into dictionary format data, wherein the dictionary format data includes structural attribute characteristics of the data and data content characteristics compressed by a formula, and stores the dictionary format data;

The second client downloads the dictionary format data from the server in response to the download request for the data, and restores the dictionary format data to the original data format through a formula.

The beneficial effect of the present invention is that after receiving the data sent by the first client for the first time, the server converts it into dictionary format data, so as to store the common structural features of the data through the dictionary format data and perform data compression and simplification. When the second client issues a download request for data, the dictionary format data is downloaded from the server, and the dictionary format data is restored to the original data format through a formula. In this way, the second client decompresses the specified data through the dictionary, and there is no need to repeatedly download the repeated part of the data during data transmission; and the data of the same structure have a common structure, and a dictionary-data storage technology is established for it to realize data "slimming" and data dictionary format conversion. At the same time, through data dictionary format conversion instead of data encryption, there is no need to change the transmission mode, which can improve the data transmission efficiency while ensuring the security of data transmission.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG1 is a flow chart of a method for user data transmission based on a distributed network according to an embodiment of the present invention;

FIG2 is a schematic diagram of a system for transmitting user data based on a distributed network according to an embodiment of the present invention;

3 is a flowchart of specific steps of a method for user data transmission based on a distributed network according to an embodiment of the present invention;

FIG. 4 is a schematic diagram of dictionary format conversion in a method for transmitting user data based on a distributed network according to an embodiment of the present invention.

DETAILED DESCRIPTION

In order to explain the technical content, achieved objectives and effects of the present invention in detail, the following is an explanation in combination with the implementation modes and the accompanying drawings.

Referring to FIG. 1 , an embodiment of the present invention provides a method for user data transmission based on a distributed network, comprising the steps of:

The server converts the received data from the first client into dictionary format data, wherein the dictionary format data includes structural attribute characteristics of the data and data content characteristics compressed by a formula, and stores the dictionary format data;

The second client downloads the dictionary format data from the server in response to the download request for the data, and restores the dictionary format data to the original data format through a formula.

From the above description, it can be seen that the beneficial effect of the present invention is that after the server receives the data sent by the first client for the first time, it converts it into dictionary format data, so as to store the common structural features of the data through the dictionary format data and compress and simplify the data. When the second client issues a download request for the data, the dictionary format data is downloaded from the server, and the dictionary format data is restored to the original data format through a formula. In this way, the second client decompresses the specified data through the dictionary, and there is no need to repeatedly download the repeated part of the data during data transmission; and the data of the same structure have a common structure, and a dictionary-data storage technology is established for it to realize data "slimming" and data dictionary format conversion. At the same time, through data dictionary format conversion instead of data encryption, there is no need to change the transmission mode, which can improve the data transmission efficiency while ensuring the security of data transmission.

Further, the server converts the received data from the first client into dictionary format data. The server converts the received data from the first client into dictionary format data, and the dictionary format data includes structural attribute characteristics of the data and data content characteristics compressed by a formula, including:

The server obtains the data of the first client, taking the data format of the data as the structural attribute feature, and taking the data other than the structural attribute feature of the data as the data content feature;

If the data content feature can be compressed by a formula, formula compressed data is obtained; otherwise, a data storage segment is obtained, and the formula compressed data, the data storage segment and the structural attribute feature are used as dictionary format data.

From the above description, it can be seen that for data content features that cannot be processed by the dictionary, they are directly generated as data storage segments, and for data content features that can be processed by the dictionary, corresponding formula compressed data are generated. Therefore, the dictionary format data includes both structural attribute features and data storage segments and formula compressed data in the data content features, and can process multiple data types, further improving data transmission efficiency.

Furthermore, the data format of the data is used as the structural attribute feature, and the data other than the structural attribute feature of the data is used as the data content feature, and then the following is further included:

The structural attribute characteristics and data content characteristics of the data are stored in corresponding nodes in the form of a binary tree.

From the above description, it can be seen that the structural attribute features and the data content features are stored in a binary tree to facilitate the subsequent restoration of the dictionary format data.

Furthermore, storing the dictionary format data includes:

The formula compressed data is segmented and the segmented formula compressed data is stored in the binary tree nodes.

From the above description, it can be seen that through segmentation, a dictionary-data-segment storage technology can be established to further achieve data "slimming".

Furthermore, the dictionary format data is restored to the original data format through a formula, including:

Generating structural information of corresponding data according to structural attribute characteristics in the dictionary format data;

The formula compressed data in the dictionary format data is restored, and the restored formula compressed data and the data storage segment are filled into the structural information of the data.

It can be seen from the above description that generating structural information through structural attribute features in the dictionary format data and filling in the data by restoring the data in the dictionary format data can effectively improve data transmission efficiency.

Referring to FIG. 2 , another embodiment of the present invention provides a system for user data transmission based on a distributed network, including a first client, a server, and a second client:

The server converts the received data from the first client into dictionary format data, wherein the dictionary format data includes structural attribute characteristics of the data and data content characteristics compressed by a formula, and stores the dictionary format data;

The second client downloads the dictionary format data from the server in response to the download request for the data, and restores the dictionary format data to the original data format through a formula.

From the above description, it can be seen that the beneficial effect of the present invention is that after the server receives the data sent by the first client for the first time, it converts it into dictionary format data, so as to store the common structural features of the data through the dictionary format data and compress and simplify the data. When the second client issues a download request for the data, the dictionary format data is downloaded from the server, and the dictionary format data is restored to the original data format through a formula. In this way, the second client decompresses the specified data through the dictionary, and there is no need to repeatedly download the repeated part of the data during data transmission; and the data of the same structure have a common structure, and a dictionary-data storage technology is established for it to realize data "slimming" and data dictionary format conversion. At the same time, through data dictionary format conversion instead of data encryption, there is no need to change the transmission mode, which can improve the data transmission efficiency while ensuring the security of data transmission.

Further, the server converts the received data from the first client into dictionary format data. The server converts the received data from the first client into dictionary format data, and the dictionary format data includes structural attribute characteristics of the data and data content characteristics compressed by a formula, including:

The server obtains the data of the first client, taking the data format of the data as the structural attribute feature, and taking the data other than the structural attribute feature of the data as the data content feature;

If the data content feature can be compressed by a formula, formula compressed data is obtained; otherwise, a data storage segment is obtained, and the formula compressed data, the data storage segment and the structural attribute feature are used as dictionary format data.

From the above description, it can be seen that for data content features that cannot be processed by the dictionary, they are directly generated as data storage segments, and for data content features that can be processed by the dictionary, corresponding formula compressed data are generated. Therefore, the dictionary format data includes both structural attribute features and data storage segments and formula compressed data in the data content features, and can process multiple data types, further improving data transmission efficiency.

Furthermore, the data format of the data is used as the structural attribute feature, and the data other than the structural attribute feature of the data is used as the data content feature, and then the following is further included:

The structural attribute characteristics and data content characteristics of the data are stored in corresponding nodes in the form of a binary tree.

From the above description, it can be seen that the structural attribute features and the data content features are stored in a binary tree to facilitate the subsequent restoration of the dictionary format data.

Furthermore, storing the dictionary format data includes:

The formula compressed data is segmented and the segmented formula compressed data is stored in the binary tree nodes.

From the above description, it can be seen that through segmentation, a dictionary-data-segment storage technology can be established to further achieve data "slimming".

Furthermore, the dictionary format data is restored to the original data format through a formula, including:

Generating structural information of corresponding data according to structural attribute characteristics in the dictionary format data;

The formula compressed data in the dictionary format data is restored, and the restored formula compressed data and the data storage segment are filled into the structural information of the data.

It can be seen from the above description that generating structural information through structural attribute features in the dictionary format data and filling in the data by restoring the data in the dictionary format data can effectively improve data transmission efficiency.

The above-mentioned method and system for user data transmission based on a distributed network of the present invention are suitable for improving data transmission efficiency while ensuring the security of data transmission, and are described below through specific implementation methods:

Embodiment 1

Referring to FIG. 1 , FIG. 3 and FIG. 4 , a method for user data transmission based on a distributed network includes the following steps:

S1. The server converts the data received from the first client into dictionary format data, wherein the dictionary format data includes structural attribute characteristics of the data and data content characteristics compressed by a formula, and stores the dictionary format data.

Specifically, after the first client uploads data to the server for the first time, the server processes the data and converts it into dictionary format data for storage. The server performs the following steps:

S11. The server obtains the data of the first client, takes the data format of the data as the structural attribute feature, takes the data other than the structural attribute feature as the data content feature, and stores the structural attribute feature and the data content feature of the data in the form of a binary tree in corresponding nodes.

In this embodiment, after the server obtains the data, it strips the data according to the data structure, that is, obtains the structural attribute characteristics and data content characteristics of the data, and stores both in the form of a binary tree on the corresponding nodes.

The structural attribute feature is to use the data format as a common feature. For example, the data header of each MP3 file has a common data structure, and the common data structure is used as the structural attribute feature.

Data content features are the remaining features excluding structural attribute features, and the members in the dictionary are used as feature identifiers for storage.

S12. If the data content feature can be compressed by a formula, formula compressed data is obtained; otherwise, a data storage segment is obtained, and the formula compressed data, the data storage segment and the structural attribute feature are used as dictionary format data.

Specifically, the data content features are obtained, and the data formula is matched therefor, and the data content features are simplified according to the data formula. In this embodiment, the formula template has multiple formulas. When matching the data formula, the data can be first loaded into the formula template to obtain the data processed by the formula and compare it with the original data. If the comparison similarity is greater than the threshold, it means that the processed data is still similar to the original data, and the data can be further compressed, that is, the factors in the formula can be adjusted. If the comparison similarity indicates that the data processed by the formula is completely consistent with the original data, it means that the data processed by the formula can be directly simplified, and the corresponding position and formula can be directly generated on the binary tree.

It also includes:

The formula compressed data is segmented and the segmented formula compressed data is stored in the binary tree nodes.

S2. The second client downloads the dictionary format data from the server in response to the download request for the data, and restores the dictionary format data to the original data format through a formula.

Specifically, according to the structural attribute characteristics in the dictionary format data, the structural information of the corresponding data is generated;

The formula compressed data in the dictionary format data is restored, and the restored formula compressed data and the data storage segment are filled into the structural information of the data.

Embodiment 2

Please refer to Figure 2, a system for user data transmission based on a distributed network includes a first client, a server, and a second client. The first client implements each step of a method for user data transmission based on a distributed network in Example 1, which is performed by the first client as the execution subject. The server implements each step of a method for user data transmission based on a distributed network in Example 1, which is performed by the server as the execution subject. The second client implements each step of a method for user data transmission based on a distributed network in Example 1, which is performed by the second client as the execution subject.

In summary, the present invention provides a method and system for user data transmission based on a distributed network. After receiving the data sent by the first client for the first time, the server converts it into dictionary format data, so as to store the common structural features of the data through the dictionary format data and perform data compression and simplification. When the second client issues a download request for data, the dictionary format data is downloaded from the server, and the dictionary format data is restored to the original data format through a formula. In this way, the second client decompresses the specified data through the dictionary, and there is no need to repeatedly download the repeated part of the data during data transmission; and the data of the same structure have a common structure, and a dictionary-data storage technology is established for it to achieve data "slimming" and data dictionary format conversion. At the same time, through data dictionary format conversion instead of data encryption, there is no need to change the transmission mode, which can improve the data transmission efficiency while ensuring the security of data transmission.

The above descriptions are merely embodiments of the present invention and are not intended to limit the patent scope of the present invention. Any equivalent transformations made using the contents of the present invention's specification and drawings, or directly or indirectly applied in related technical fields, are also included in the patent protection scope of the present invention.

Claims (10)

1. A method for transmitting user data based on a distributed network, characterized in that it comprises the steps of: The server converts the received data from the first client into dictionary format data, wherein the dictionary format data includes structural attribute characteristics of the data and data content characteristics compressed by a formula, and stores the dictionary format data; The second client downloads the dictionary format data from the server in response to the download request for the data, and restores the dictionary format data to the original data format through a formula. 2. According to claim 1, a method for transmitting user data based on a distributed network is characterized in that the server converts the data received from the first client into dictionary format data, and the server converts the data received from the first client into dictionary format data, wherein the dictionary format data includes structural attribute characteristics of the data and data content characteristics compressed by a formula, including: The server obtains the data of the first client, taking the data format of the data as the structural attribute feature, and taking the data other than the structural attribute feature of the data as the data content feature; If the data content feature can be compressed by a formula, formula compressed data is obtained; otherwise, a data storage segment is obtained, and the formula compressed data, the data storage segment and the structural attribute feature are used as dictionary format data. 3. A method for transmitting user data based on a distributed network according to claim 2, characterized in that the data format of the data is used as a structural attribute feature, and the data other than the structural attribute feature of the data is used as a data content feature, and then further comprising: The structural attribute characteristics and data content characteristics of the data are stored in corresponding nodes in the form of a binary tree. 4. The method for transmitting user data based on a distributed network according to claim 3, wherein storing the dictionary format data comprises: The formula compressed data is segmented and the segmented formula compressed data is stored in the binary tree nodes. 5. The method for transmitting user data based on a distributed network according to claim 2, characterized in that restoring the dictionary format data to the original data format through a formula comprises: Generating structural information of corresponding data according to structural attribute characteristics in the dictionary format data; The formula compressed data in the dictionary format data is restored, and the restored formula compressed data and the data storage segment are filled into the structural information of the data. 6. A system for user data transmission based on a distributed network, characterized by comprising a first client, a server, and a second client: The server converts the received data from the first client into dictionary format data, wherein the dictionary format data includes structural attribute characteristics of the data and data content characteristics compressed by a formula, and stores the dictionary format data; The second client downloads the dictionary format data from the server in response to the download request for the data, and restores the dictionary format data to the original data format through a formula. 7. A distributed network-based user data transmission system according to claim 6, characterized in that the server converts the received data from the first client into dictionary format data, and the server converts the received data from the first client into dictionary format data, wherein the dictionary format data includes structural attribute characteristics of the data and data content characteristics compressed by a formula, including: The server obtains the data of the first client, taking the data format of the data as the structural attribute feature, and taking the data other than the structural attribute feature of the data as the data content feature; If the data content feature can be compressed by a formula, formula compressed data is obtained; otherwise, a data storage segment is obtained, and the formula compressed data, the data storage segment and the structural attribute feature are used as dictionary format data. 8. A distributed network-based user data transmission system according to claim 7, characterized in that the data format of the data is used as the structural attribute feature, and the data other than the structural attribute feature of the data is used as the data content feature, and then further comprising: The structural attribute characteristics and data content characteristics of the data are stored in corresponding nodes in the form of a binary tree. 9. The system for transmitting user data based on a distributed network according to claim 8, wherein storing the dictionary format data comprises: The formula compressed data is segmented and the segmented formula compressed data is stored in the binary tree nodes. 10. The user data transmission system based on a distributed network according to claim 7, characterized in that restoring the dictionary format data to the original data format through a formula comprises: Generating structural information of corresponding data according to structural attribute characteristics in the dictionary format data; The formula compressed data in the dictionary format data is restored, and the restored formula compressed data and the data storage segment are filled into the structural information of the data.