CN104408047A - Method for uploading text file to HDFS (hadoop distributed file system) in multi-machine parallel mode based on NFS (network file system) file server - Google Patents

Abstract

The present invention provides a method for uploading text files to HDFS in parallel by multiple machines based on NFS file servers. It selects N hosts in the HDFS cluster, then selects any node as the master node, and other N-1 nodes as slave nodes. On the node, the NFS file server to be uploaded needs to upload the files in the directory. For each file, the parallel upload method is adopted, that is, all machines in the cluster participate in the upload, and each host in the cluster is responsible for uploading each file 1/N in size Continuous data blocks, to achieve the purpose of parallel upload, thereby increasing the upload speed.

Description

A method of parallel uploading text files to HDFS by multiple machines based on NFS file server

technical field

The present invention relates to the technical field of big data storage, and specifically relates to a method for uploading text files to HDFS in parallel by multiple machines based on an NFS file server.

Background technique

With the development of computer network, the era of massive data has come. The Internet Data Center predicts that the global data usage will increase by 44 times by 2020, reaching 35.2ZB.

For the storage, analysis, management and mining of such a large data set, traditional technologies (including traditional relational databases) are incapable. How to analyze and understand these data in the fastest and best way is an urgent task in front of everyone. Among the existing technologies and tools, the most mature and successful set of big data solutions is the Hadoop file storage computing framework and related components built on it. For a large number of text files generated every day, if they are quickly uploaded to HDFS for subsequent processing, it is currently a problem. In order to solve the problem of fast uploading of text files, this paper proposes a method based on NFS file server to upload text files to HDFS in parallel.

HDFS adopts the three-copy mechanism by default. For the HDFS client, when a user is using a client to write data to HDFS, if there is a DataNode node on the client, the NameNode will give priority to the data being written. One copy of the client is saved on the DataNode node of the client, and the other two copies are saved on other DataNode nodes in the cluster, so that in the entire cluster, if there is only one client write operation, only 3 DataNode nodes in the cluster work. Other DataNodes are idle and cannot perform the performance of the entire cluster.

Contents of the invention

The purpose of the present invention is to provide a kind of text file multi-computer parallel upload method to HDFS based on NFS file server.

The purpose of the present invention is achieved in the following manner, select N hosts in the HDFS cluster, then select any node as the master node, and other N-1 nodes as slave nodes, on the master node, obtain the NFS file server to be uploaded To upload files in the directory, for each file, adopt the parallel upload method, that is, all machines in the cluster participate in the upload, and each host in the cluster is responsible for uploading continuous data blocks of 1/N size for each file, achieving parallel uploading purpose, so as to improve the upload speed, the specific steps are as follows:

1) The MainPut program on the master node calculates the start and end byte streams of the data blocks to be uploaded on each of the N nodes, and starts the BlockPut program on the N nodes to upload in parallel; if it is run for the first time, an executable will be installed on each node The program BlockPut is used to upload the data block that the node is responsible for uploading, and then initiates a command to each slave node to start the BlockPut program;

2) The BlockPut program on each node is responsible for uploading the data blocks to be uploaded to HDFS. BlockPut opens an input stream InputStream of the file to be uploaded. The InputStream locates the start byte stream, and then creates an independent file on HDFS, and the start and end bytes Streams are written to HDFS standalone files.

Mount the directory to be uploaded to the default unified directory of N nodes.

N is not greater than the number of clients that can reach the maximum bandwidth when the NFS file server reads in parallel,

The beneficial effects of the purpose of the present invention are: the invention clearly selects N nodes in the cluster as clients, divides a file into N data blocks and uploads them at the same time, each client is responsible for one block, and each block is saved as an independent block on HDFS. files, which can maximize the performance of the entire cluster. Upload a text file in blocks in parallel to maximize the performance of the cluster and improve upload efficiency.

Description of drawings

Figure 1 is a framework diagram based on multi-machine parallel upload processing.

Detailed ways

The method of the present invention is described in detail below with reference to the accompanying drawings.

Select N hosts in the HDFS cluster, then select any node as the master node, and the other N-1 nodes as slave nodes. On the master node, obtain the files in the directory to be uploaded by the NFS file server. For each file, The parallel upload method is adopted, that is, all machines in the cluster participate in the upload, and each host in the cluster is responsible for uploading continuous data blocks of 1/N size for each file, so as to achieve the purpose of parallel upload, thereby increasing the upload speed, an aspect of the present invention A method of uploading text files to HDFS in parallel on multiple machines based on NFS file server. The whole process is as follows:

1) The MainPut program on the master node calculates the start and end byte streams of the data blocks to be uploaded on each of the N nodes, and starts the BlockPut program on the N nodes to upload in parallel; if it is run for the first time, it will install a Execute the program BlockPut, which is used to upload the data blocks that the node is responsible for uploading, and then initiate a command to each slave node to start the BlockPut program;

2) The BlockPut program on each node is responsible for uploading the data blocks to be uploaded to HDFS. BlockPut opens an input stream InputStream of the file to be uploaded. The InputStream locates the start byte stream, and then creates an independent file on HDFS, and the start and end bytes Streams are written to HDFS standalone files.

Except for the technical features described in the instructions, all are known technologies by those skilled in the art.

Claims (3)

1. one kind uploads to HDFS method based on the text multi-host parallel of NFS file server, it is characterized in that, choose N number of main frame in HDFS cluster, then select any node as host node, other N-1 node is as from node, on the primary node, the NFS file server that acquisition will be uploaded will upload file under catalogue, for each file, adopt parallel method for uploading, namely in cluster, all machines all participate in uploading, in cluster, the continuous print data block uploading each file 1/N size is responsible for by each main frame, reach the parallel object uploaded, thus raising uploading speed, concrete steps flow process is:

The each node of the N number of node of MainPut program computation data block start-stop to be uploaded byte stream on host node, and start BlockPut program parallelization on N number of node and upload; If first time is run, on each node an executable program BlockPut being installed, being responsible for for uploading this node institute the data block uploaded, then initiate order startup BlockPut program to each from node;

2) on each node, BlockPut program is responsible for data block to be uploaded to upload to HDFS, BlockPut opens a file input stream InputStream to be uploaded, InputStream navigates to banner word throttling, after on HDFS create a unique file, start-stop byte stream is written in HDFS unique file.

2. method according to claim 1, is characterized in that acquiescence unified directory catalogue to be uploaded being mounted to N number of node.

3. method according to claim 1, is characterized in that, N is not more than the number clients that NFS file server walks abreast when can to reach maximum bandwidth when reading.