CN107071073A - A kind of document transmission method and device applied to object storage system - Google Patents

Abstract

The invention discloses a file transmission method and device applied to an object storage system. The method includes: traversing the files to be uploaded, and screening out the first target file with the target suffix name, wherein the target suffix name is jpg/jpeg ; Use the lepton compression tool to compress the first target file to obtain a compressed file; transmit the compressed file and other files to be uploaded to the object storage system. It can be seen that, in this method, the first target file is compressed by the lepton compression tool to obtain a compressed file, so that when uploading files to the object storage system, some files are compressed, and therefore, the network transmission data can be greatly reduced. The required bandwidth increases the transfer rate of JPEG image data. In addition, the effect of the file transfer device applied to the object storage system disclosed in the present invention is as above.

Description

A file transfer method and device applied to an object storage system

technical field

The present invention relates to the field of object storage systems, in particular to a file transmission method and device applied to object storage systems.

Background technique

The object storage system combines the advantages of NAS and SAN, and has the advantages of high-speed direct access of SAN and data sharing of NAS, and provides a storage architecture with high reliability, cross-platform and safe data sharing. Since the object storage system has a storage function, a client can upload a file to the object storage system for storage, or can download a required file from the object storage system.

In the era of cloud storage, users will generate a large number of files, especially image files. When these image files are uploaded to the object storage system, a large amount of network bandwidth is required, which puts huge pressure on the bandwidth of the transmission network, and the transmission time is too long. long, and even lead to network paralysis.

It can be seen that how to reduce the pressure on the transmission network and improve the transmission efficiency is an urgent problem to be solved by those skilled in the art.

Contents of the invention

The purpose of the present invention is to provide a file transmission method and device applied to an object storage system, which is used to reduce the pressure on the transmission network and improve transmission efficiency.

In order to solve the above technical problems, the present invention provides a file transfer method applied to an object storage system, including:

Traversing the files to be uploaded, filtering out the first target file with the target suffix name, wherein the target suffix name is jpg/jpeg;

Using the lepton compression tool to compress the first target file to obtain a compressed file;

The compressed file and other files to be uploaded are transmitted to the object storage system.

Preferably, it also includes:

After the transmission of the compressed file is completed, a compression mark is added to the compressed file.

Preferably, before filtering out the first target file with the target suffix name, it also includes:

Judging whether the relationship between the storage space occupied by all the files to be uploaded and the current network bandwidth satisfies the preset relationship;

If yes, then enter the step of filtering out the first target file with the target suffix name.

Preferably, the transferring the compressed file and other files to be uploaded to the object storage system specifically includes:

Sorting the storage space occupied by each of the compressed files and the storage space occupied by the remaining files to be uploaded;

The compressed file and the remaining files to be uploaded are transmitted to the object storage system in ascending order.

Preferably, it is characterized in that it also includes:

After obtaining the user's download instruction, transmit the file to be downloaded in the object storage system to the client according to the download instruction;

Traversing the files to be downloaded, filtering out the second target file with the target suffix name;

judging whether the second target file is compressed;

If so, using the lepton compression tool to decompress the second target file to obtain a decompressed file.

In order to solve the above technical problems, the present invention also provides a file transfer device applied to an object storage system, including:

The first traversal module is used to traverse the files to be uploaded and filter out the first target file with the target suffix name, wherein the target suffix name is jpg/jpeg;

A compression module, configured to compress the first target file using a lepton compression tool to obtain a compressed file;

An upload module, configured to transmit the compressed file and other files to be uploaded to the object storage system.

Preferably, it also includes:

The adding module is used for adding a compression mark to the compressed file after the transmission of the compressed file is completed.

Preferably, it also includes:

The first judging module is used to judge whether the relationship between the storage space occupied by all the files to be uploaded and the current network bandwidth satisfies the preset relationship;

If so, trigger the first traversal module to perform the step of filtering out the first target file with the target suffix name.

Preferably, the upload module specifically includes:

A sorting unit, configured to sort the storage space occupied by each of the compressed files and the storage space occupied by each of the remaining files to be uploaded;

A transmission unit, configured to transmit the compressed file and the remaining files to be uploaded to the object storage system in ascending order.

Preferably, it also includes:

A download module, configured to transmit the file to be downloaded in the object storage system to the client according to the download instruction after obtaining the user's download instruction;

The second traversal module is used to traverse the files to be downloaded and filter out the second target files with the target suffix name;

A second judging module, configured to judge whether the second target file has been compressed;

The decompression module is configured to use the lepton compression tool to decompress the second target file to obtain a decompressed file when the judgment result of the second judgment module is yes.

In the file transfer method applied to the object storage system provided by the present invention, the lepton compression tool is used to compress the first target file to obtain a compressed file, so that when the file is uploaded to the object storage system, a part of the file is compressed. Therefore, It can greatly reduce the bandwidth required for network transmission data and increase the transmission rate of JPEG image data. In addition, the file transmission device applied to the object storage system provided by the present invention also has the above effects.

Description of drawings

In order to illustrate the embodiments of the present invention more clearly, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. As far as people are concerned, other drawings can also be obtained based on these drawings on the premise of not paying creative work.

FIG. 1 is a flow chart of a file transfer method applied to an object storage system provided by an embodiment of the present invention;

FIG. 2 is a flowchart of another file transfer method applied to an object storage system provided by an embodiment of the present invention;

FIG. 3 is a structural diagram of a file transfer device applied to an object storage system according to an embodiment of the present invention.

detailed description

The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only some of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by persons of ordinary skill in the art without making creative efforts belong to the protection scope of the present invention.

The core of the present invention is to provide a file transmission method and device applied to an object storage system, which is used to reduce the pressure on the transmission network and improve transmission efficiency.

In order to enable those skilled in the art to better understand the solution of the present invention, the present invention will be further described in detail below in conjunction with the accompanying drawings and specific embodiments.

FIG. 1 is a flow chart of a file transfer method applied to an object storage system provided by an embodiment of the present invention. As shown in Figure 1, the file transfer methods applied to the object storage system include:

S10: traverse the files to be uploaded, and filter out the first target file with the target suffix, wherein the target suffix is jpg/jpeg.

It can be understood that there are many types of files to be uploaded, such as data-type files, image-type files, and the like. Especially for image-type files, a large amount of network bandwidth will be occupied during the transmission process, resulting in the problem of low transmission time and low transmission efficiency. In this embodiment, the file with the target suffix name jpg/jpeg is selected from the files to be uploaded as the first target file.

S11: Using the lepton compression tool to compress the first target file to obtain a compressed file.

In order to reduce the amount of data to be transmitted, this embodiment adopts the lepton compression tool to realize file compression. The lepton compression tool is an open source compression tool provided by Dropbox, which can compress images with the suffix jpg/jpeg. The average compression rate of the compression tool for JPEG images is 22%, thus greatly reducing the network transmission data. The required bandwidth increases the transfer rate of JPEG image data.

S12: Transfer the compressed file and other files to be uploaded to the object storage system.

It is understandable that the remaining files to be uploaded do not need to be compressed, so they will be directly transmitted to the upload thread pool for uploading. When the first target file is compressed to form a compressed file, at this time, the compressed file can be directly transmitted to the upload thread pool for processing. upload.

As a preferred implementation manner, S12 specifically includes:

Sort the storage space occupied by each compressed file and the storage space occupied by the remaining files to be uploaded;

Transfer the compressed file and other files to be uploaded to the object storage system in ascending order.

It is understandable that the smaller the occupied storage space is, the shorter the required transmission time will be. Arranging and uploading according to the occupied storage space helps to transfer more files within a certain time range. For example, if the network suddenly crashes during the transfer process, the smaller the storage space occupied by the files transferred earlier, the more files will be obtained in the object storage system.

The file transfer method applied to the object storage system provided in this embodiment uses the lepton compression tool to compress the first target file to obtain a compressed file, so that when uploading files to the object storage system, some files are compressed. Therefore, It can greatly reduce the bandwidth required for network transmission data and increase the transmission rate of JPEG image data.

FIG. 2 is a flowchart of another file transfer method applied to an object storage system provided by an embodiment of the present invention. As shown in Figure 2, as a preferred embodiment, it also includes:

S20: Add a compression mark to the compressed file after the transmission of the compressed file is completed.

Because there are a large number of files stored in the object storage system, and these files are not only of different types, but also some of the files of the same type may be compressed, and some may not be compressed. In order to facilitate the distinction, in this embodiment, a compression mark is added to the compressed file. On this basis, it can be determined whether the file has been compressed by judging whether the file has a compression flag.

As a preferred implementation, before filtering out the first target file with the target suffix name, it also includes:

Determine whether the relationship between the storage space occupied by all files to be uploaded and the current network bandwidth satisfies the preset relationship;

If yes, then enter the step of filtering out the first target file with the target suffix name.

In a specific implementation, if the current network bandwidth is sufficient and the storage space occupied by the files to be uploaded is small, then in this case, there is no need to compress the first target file, because the compression process also takes time , and if the network bandwidth is sufficient, the transmission of the first target file may be completed within this time.

Therefore, in this embodiment, at first judge the storage space occupied by all the files to be uploaded to determine the required network bandwidth, through the relationship between the required network bandwidth and the current network bandwidth, it can be determined whether it is necessary to upload the first target file compression. It can be understood that the preset relationship needs to be determined in advance, and the specific content of the preset relationship will not be repeated in this embodiment.

As shown in Figure 2, as a preferred embodiment, it also includes:

S21: After obtaining the user's download instruction, transmit the file to be downloaded in the object storage system to the client according to the download instruction.

S22: traverse the files to be downloaded, and filter out the second target file with the target suffix name.

S23: Determine whether the second target file has been compressed, and if so, proceed to S24.

S24: Using the lepton compression tool to decompress the second target file to obtain a decompressed file.

The transmission of files can be uploaded or downloaded. Steps S10-S12 above are the uploading process, while steps S21-S22 realize the downloading process. It can be understood that the file to be downloaded and the file to be uploaded can be completely the same, completely different, or partly the same, therefore, the second target file can also be completely the same as the first target file, or can be completely different, or partly the same. The corresponding files can be transferred from the object storage system to the client by downloading and decompressing.

In the foregoing, the corresponding embodiments of the file transfer method applied to the object storage system are described in detail, and the present invention also provides a file transfer device applied to the object storage system. Since the embodiment of the device part corresponds to the embodiment of the method part, please refer to the description of the embodiment of the method part for the embodiment of the device part, and details will not be repeated here.

FIG. 3 is a structural diagram of a file transfer device applied to an object storage system according to an embodiment of the present invention. As shown in Figure 3, including:

The first traversal module 10 is used to traverse the files to be uploaded, and filter out the first target file with the target suffix name, wherein the target suffix name is jpg/jpeg;

Compression module 11 is used to utilize lepton compression tool to compress the first target file to obtain a compressed file;

The upload module 12 is configured to transmit the compressed file and other files to be uploaded to the object storage system.

The file transfer device applied to the object storage system provided in this embodiment uses the lepton compression tool to compress the first target file to obtain a compressed file, so that when uploading files to the object storage system, some files are compressed. Therefore, It can greatly reduce the bandwidth required for network transmission data and increase the transmission rate of JPEG image data.

As a preferred embodiment, it also includes:

Add a module for adding a compression flag to the compressed file after the compressed file transfer is completed.

As a preferred embodiment, it also includes:

The first judging module is used to judge whether the relationship between the storage space occupied by all the files to be uploaded and the current network bandwidth satisfies the preset relationship;

If yes, the first traversal module is triggered to perform the step of filtering out the first target file with the target suffix name.

As a preferred implementation, the upload module specifically includes:

A sorting unit, configured to sort the storage space occupied by each compressed file and the storage space occupied by the remaining files to be uploaded;

The transfer unit is configured to transfer the compressed file and other files to be uploaded to the object storage system in ascending order.

As a preferred embodiment, it also includes:

The download module is used to transmit the file to be downloaded in the object storage system to the client according to the download instruction after obtaining the download instruction from the user;

The second traversal module is used to traverse the files to be downloaded and filter out the second target files with the target suffix name;

The second judging module is used to judge whether the second target file is compressed;

The decompression module is configured to use the lepton compression tool to decompress the second target file to obtain the decompressed file when the judgment result of the second judgment module is yes.

The file transfer method and device applied to the object storage system provided by the present invention have been introduced in detail above. Each embodiment in the description is described in a progressive manner, each embodiment focuses on the difference from other embodiments, and the same and similar parts of each embodiment can be referred to each other. As for the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and for the related part, please refer to the description of the method part. It should be pointed out that for those skilled in the art, without departing from the principle of the present invention, some improvements and modifications can be made to the present invention, and these improvements and modifications also fall within the protection scope of the claims of the present invention.

It should also be noted that in this specification, relative terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply that these entities or operations There is no such actual relationship or order between the operations. Furthermore, the term "comprises", "comprises" or any other variation thereof is intended to cover a non-exclusive inclusion such that a process, method, article, or apparatus comprising a set of elements includes not only those elements, but also includes elements not expressly listed. other elements of or also include elements inherent in such a process, method, article, or device. Without further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of additional identical elements in the process, method, article or apparatus comprising said element.

Claims (10)

1. a kind of document transmission method applied to object storage system, it is characterised in that including：

File to be uploaded is traveled through, the first object file with target suffix name is filtered out, wherein, the target suffix is entitled jpg/jpeg；

The first object file is compressed using lepton tools of compression and obtains compressed file；

The compressed file and remaining file to be uploaded are transmitted to object storage system.

2. the document transmission method according to claim 1 applied to object storage system, it is characterised in that also include：

After the compressed file is transmitted, to compressed file addition compression mark.

3. the document transmission method according to claim 1 applied to object storage system, it is characterised in that filtering out Before first object file with target suffix name, in addition to：

Judge whether the relation of the memory space shared by whole files to be uploaded and current network bandwidth meets preset relation；

If it is, into the step of filtering out the first object file with target suffix name.

4. the document transmission method according to claim 1 applied to object storage system, it is characterised in that described by institute State compressed file and remaining file to be uploaded is transmitted to object storage system and specifically included：

Memory space shared by memory space shared by each compressed file and remaining each described file to be uploaded is arranged Sequence；

The compressed file and remaining described file to be uploaded are transmitted into the object storage system according to ascending order.

5. the document transmission method applied to object storage system according to claim 1-4 any one, its feature exists In, in addition to：

After the download instruction of user is got, according to the download instruction by the file to be downloaded in the object storage system Transmit to client；

The file to be downloaded is traveled through, the second file destination with the target suffix name is filtered out；

Judge second file destination whether through overcompression；

If it is, second file destination decompress using the lepton tools of compression to obtain decompressing files.

6. a kind of file transfer means applied to object storage system, it is characterised in that including：

First spider module, for traveling through file to be uploaded, filters out the first object file with target suffix name, wherein, The entitled jpg/jpeg of target suffix；

Compression module, compressed file is obtained for being compressed the first object file using lepton tools of compression；

Uploading module, for the compressed file and remaining file to be uploaded to be transmitted to object storage system.

7. the file transfer means according to claim 6 applied to object storage system, it is characterised in that also include：

Add module, for after the compressed file is transmitted, to compressed file addition compression mark.

8. the file transfer means according to claim 6 applied to object storage system, it is characterised in that also include：

First judge module, the relation for judging memory space and current network bandwidth shared by all files to be uploaded Whether preset relation is met；

If it is, triggering first spider module performs the step for filtering out the first object file with target suffix name Suddenly.

9. the file transfer means according to claim 6 applied to object storage system, it is characterised in that the upload Module is specifically included：

Sequencing unit, for depositing shared by the memory space shared by each compressed file and remaining each described file to be uploaded Storage space is ranked up；

Transmission unit, it is described for transmitting the compressed file and remaining described file to be uploaded according to ascending order Object storage system.

10. the file transfer means applied to object storage system according to claim 6-9 any one, its feature exists In, in addition to：

Download module, for after the download instruction of user is got, according to the download instruction by the object storage system In file to be downloaded transmit to client；

Second spider module, for traveling through the file to be downloaded, filters out the second target text with the target suffix name Part；

Second judge module, for judging second file destination whether through overcompression；

Decompression module, in the case of being in the judged result of second judge module, is compressed using the lepton Second file destination decompress obtaining decompressing files by instrument.