CN103092927B - File rapid read-write method under a kind of distributed environment - Google Patents

Abstract

The invention discloses a method for quickly reading files in a distributed environment. The connected data node keeps the connection. If not, the metadata node inquires whether the file exists in its data area according to the information in its index area. If not, the metadata node inquires whether the file exists in its primary index information. Data node, the client node establishes a connection with the data node, the data node searches for the data block where the file is located according to the secondary index information, obtains the file according to the secondary index information, and sends the file to the client node, the client node receives the data and keeps A connection to this data node. The invention can solve the problems of large memory occupied by metadata nodes and low writing efficiency of a large number of files existing in the existing method.

Description

A Fast File Reading and Writing Method in Distributed Environment

technical field

The invention belongs to the field of network communication, and more specifically relates to a method for fast reading and writing of files in a distributed environment.

Background technique

With the rapid development of science and technology and the Internet, storage systems need to store massive amounts of data, respond to high-concurrent user access, and provide highly reliable and highly available services. Traditional stand-alone systems can no longer meet these requirements, and distributed file systems can quickly Good to meet these needs. In practical applications (personal applications, web applications, scientific computing, etc.), massive file information will be generated. How to efficiently store and access massive files in a distributed environment is still a problem and challenge.

The current mainstream distributed file systems include googleGFS, HDFS, Lustre, Ceph, etc. The architecture and basic principles of these distributed file systems are roughly the same, mainly composed of metadata nodes, data nodes and client nodes. Among them, the metadata node stores the metadata of the distributed file system (the namespace of the file system, the mapping of file name -> data block, and the mapping of data block -> data node); the data node stores the actual file data (generally in the form of data block storage in the form of storage); the client node connects to the metadata node for file information query, connects to the data node for actual file transfer, and communicates with the metadata node before accessing data.

The distributed file system has relatively low read and write performance for files. The reasons for its poor file read and write performance are as follows: the metadata of the distributed file system is stored in the memory of the metadata node, and a large number of files will occupy a lot of memory of the metadata node (one file will occupy one index item); a large number of files frequently Access will increase the burden on metadata nodes (client nodes interact with metadata nodes continuously), causing frequent disk seeks of data nodes and reducing system performance; client nodes interact with metadata nodes when accessing files The time may be greater than the data transfer time with the data node.

Contents of the invention

Aiming at the defects of the prior art, the purpose of the present invention is to provide a method for fast file writing in a distributed environment, aiming at solving the problems in the existing method that the metadata nodes occupy a large amount of memory and the writing efficiency of a large number of files is low.

In order to achieve the above object, the present invention provides a method for fast writing of files in a distributed environment, comprising the following steps:

Step S301: Initialize the data area and index information of the metadata node in the distributed environment, where the index information includes the index area and primary index area of the metadata node, and the secondary index area of the data node;

Step S302: the client node sends a file write request to the metadata node;

Step S303: The metadata node determines whether the remaining space of the data area of the metadata node is greater than or equal to the size of the file according to the request for writing the file, if yes, then go to step S304, otherwise go to step S308;

Step S304: the metadata node receives the file of the client node, and stores the file in the remaining space of the data area of the metadata node;

Step S305: The metadata node updates the information of its index area:

Step S306: the metadata node judges whether the data stored in the data area of the metadata node is greater than a threshold, if yes, then go to step S307, otherwise the process ends;

Step S307: The metadata node stores the data in its data area as an ordinary file in the distributed file system, and clears the data in its data area and index area, and the process ends;

Step S308: the metadata node stores the data in its data area as an ordinary file in the distributed file system, and clears the data in its data area and index area;

Step S309: the metadata node receives the file data of the client node and stores it in the remaining space of its data area;

Step S310: The metadata node updates the information of its index area.

The size of the file is between 0~1MB, and the size of an ordinary file is larger than the threshold.

In step 305 and step S310, the metadata node adds a new entry in its index area, including the file ID, the offset of the file in the data area, and the size of the file.

Step S301 includes the following sub-steps:

Step S401: Determine whether the data area and index information of the metadata node in the distributed environment have been initialized, if yes, the process ends, otherwise, go to step S402;

Step S402: the metadata node opens an area of size M in its memory to store temporary files, where M is a positive integer greater than the above threshold;

Step S403: the metadata node sets an index area for storing the index information of each file in its data area;

Step S404: the metadata node sets a first-level index area for storing the mapping relationship between files and data nodes;

Step S405: The data node sets a secondary index area, which is located in the data node and used to store the secondary index information of the file.

The secondary index information includes: a mapping from a file to a data block, the offset of the file in the data block, and the size of the file.

Both steps S307 and S308 include the following sub-steps:

Step S501: the metadata node saves the data in its data area as an ordinary file in the distributed file system;

Step S502: the metadata node sends the index information of the ordinary file to the secondary index area of the corresponding data node, and the data node adds the index information to its secondary index area;

Step S503: the metadata node updates its primary index information according to the file ID and the data node ID;

Step S504: the metadata node clears the data in its data area;

Step S505: The metadata node clears the data in its index area.

Step S503 is specifically that the metadata node adds the mapping relationship between the file ID and the data node ID in its primary index area, so as to read and query the file.

Through the above technical solutions conceived by the present invention, compared with the prior art, this method has the following beneficial effects:

(1) Save the memory of the metadata node and increase the number of files that the distributed file system can store: due to the adoption of steps S301, S307 and S308, by storing the primary index information of the file in the metadata node, in the data node Store the secondary index information of the file, thus reducing the memory usage of the metadata node, increasing the number of files that the distributed file system can store, and improving the memory utilization rate of the data node.

(2) Improve the performance of writing files: due to the adoption of steps S301, S307 and S308, many files are merged in the data area of the metadata node and stored in the distributed file system, thus reducing the number of client nodes and data nodes The number of interactions is also reduced, and the time spent writing large files is also reduced.

Another object of the present invention is to provide a method for quickly reading files in a distributed environment, which aims to solve the problems of excessive load on metadata nodes and low efficiency of reading a large number of files existing in existing methods.

In order to achieve the above object, the present invention provides a method for quickly reading files in a distributed environment, comprising the following steps:

Step S601: the client node sends a file read request to the metadata node;

Step S602: The client node judges whether it maintains a connection with the data node connected to the client node for reading the file last time in the distributed file system, if so, proceeds to step S603, otherwise proceeds to step S606;

Step S603: the client node sends a file read request to the data node;

Step S604: The data node searches according to the secondary index information stored in its secondary index area to determine whether it has stored the file corresponding to the file read request, if so, proceed to step S609, otherwise proceed to step S605;

Step S605: the client node disconnects from the data node;

Step S606: The metadata node inquires whether the file exists in its data area according to the information in its index area, and if so, proceeds to step S611, otherwise proceeds to step S607;

Step S607: the metadata node queries the data node storing the file according to its primary index information;

Step S608: the client node establishes a connection with the data node;

Step S609: the data node searches for the data block where the file is located according to the secondary index information, obtains the file according to the secondary index information, and sends the file to the client node;

Step S610: the client node receives the data and maintains a connection with the data node, and then the process ends;

Step S611: the metadata node obtains the file from its data area according to the index information in its index area, and sends the file to the client node.

The connection between the client node and the data node can be a TCP connection or a UDP connection.

Through the above technical solutions conceived by the present invention, compared with the prior art, this method has the following beneficial effects:

(1) Reduce the load on the metadata node: due to the adoption of steps S602 and S610, the client node will maintain a connection with the data node where the file read last time is located, so that if the file to be read next is also on the data node (The reading of files is usually localized, and the files in the same data block may be read continuously), the client node does not need to connect to the metadata node, thus reducing the load on the metadata node and improving the system performance. response speed.

(2) Improve the performance of reading files: due to the adoption of step S611, if the file to be read is located in the data area of the metadata node, the client node can directly read the data from the data area of the metadata node (compared with the data from the disk). Read fast), and do not need to connect with data nodes and read files, so the efficiency of file reading can be significantly improved. Due to the adoption of steps S602 and S610, the client node directly connects to the data node to read the file, so the performance of reading the file can be improved.

Description of drawings

FIG. 1 is an architecture diagram of a distributed file system to which the fast file reading and writing method in the distributed environment of the present invention is applied.

Fig. 2 is a frame diagram of metadata nodes in the present invention.

FIG. 3 is a flow chart of the method for fast writing files in a distributed environment according to the present invention.

FIG. 4 is a detailed flow chart of step S301 in the method for fast file writing in a distributed environment of the present invention.

FIG. 5 is a detailed flow chart of steps S307/S308 in the method for fast writing files in a distributed environment according to the present invention.

FIG. 6 is a flow chart of the method for quickly reading files in a distributed environment according to the present invention.

detailed description

In order to make the object, technical solution and advantages of the present invention clearer, the present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described here are only used to explain the present invention, not to limit the present invention.

At first the technical terms in the present invention are explained and defined:

Metadata node: save the metadata of the distributed file system (the namespace of the file system, the mapping of file name -> data block, and the mapping of data block -> data node).

Data node: stores the actual file data (generally stored in the form of data blocks). It accepts block operation commands from metadata nodes through heartbeat.

Client node: connect to metadata nodes for file information query, connect metadata nodes and data nodes for actual file transfer.

The present invention will be described in detail below in conjunction with the accompanying drawings.

As shown in Figure 1, the distributed file system framework to which the file fast reading and writing method in the distributed environment of the present invention is applied includes the following parts:

Metadata node: For files, the metadata node opens up a data area, index area and first-level index area in the memory. The data area is used to save temporary files and merge files. The index area is used to store the data in each file. The index information of the area, the first-level index area stores the mapping from files to data nodes. The frame diagram of the metadata node is shown in Figure 2;

Data node: Open up a secondary index area in the memory to store the secondary index information of the file, including the following information: the mapping from the file to the data block; the offset of the file in the data block and the size of the file; and

Client node: connect to metadata nodes for file information query, connect metadata nodes and data nodes for actual file transfer.

As shown in Figure 2, the metadata node framework in the present invention includes the following contents:

Data area: used to save temporary files and merge files;

Index area: It is used to store the index information of each file in the data area. The index item in the index area is a fixed-length index. Each file corresponds to an index item. The index item includes fileID, offset and length, where fileID represents the file name, offset represents the offset of the file in the data area, and length represents the size of the file. For each additional file data in the data area, a corresponding index item must be added to the index area;

First-level index area: it is a global index. The first-level index item includes the mapping between fileID and data node ID. The data node ID identifies a specific data node. For files that are merged into files and stored in the data node, the The storage information is added to the first-level index, so that when the client node reads the file, it can locate the specific data node that stores the file.

As shown in Figure 3, the file fast writing method under the distributed environment of the present invention comprises the following steps:

Step S301: Initialize the data area and index information of the metadata node in the distributed environment, where the index information includes the index area and primary index area of the metadata node, and the secondary index area of the data node;

Step S302: the client node sends a file write request to the metadata node; in the present invention, the file size is between 0 and 1MB;

Step S303: The metadata node determines whether the remaining space of the data area of the metadata node is greater than or equal to the size of the file according to the request for writing the file, if yes, then go to step S304, otherwise go to step S308;

Step S304: the metadata node receives the file of the client node, and stores the file in the remaining space of the data area of the metadata node;

Step S305: The metadata node updates the information in its index area: specifically, the metadata node adds a new entry in its index area, including the file ID, the offset of the file in the data area, and the size of the file ;

Step S306: the metadata node judges whether the data stored in the data area of the metadata node is greater than a threshold, if yes, then proceed to step S307, otherwise the process ends; specifically, the value range of the threshold is 60 to 63Mb;

Step S307: The metadata node stores the data in its data area as an ordinary file in the distributed file system, and clears the data in its data area and index area, and the process ends; specifically, an ordinary file refers to a file whose size is larger than Documentation of the above thresholds;

Step S308: the metadata node stores the data in its data area as an ordinary file in the distributed file system, and clears the data in its data area and index area;

Step S309: the metadata node receives the file data of the client node and stores it in the remaining space of its data area;

Step S310: The metadata node updates the information in its index area: specifically, the metadata node adds a new entry in its index area, including the file ID, the offset of the file in the data area, and the size of the file ;

As shown in Figure 4, step S301 in the method of the present invention includes the following sub-steps:

Step S401: Determine whether the data area and index information of the metadata node in the distributed environment have been initialized, if yes, the process ends, otherwise, go to step S402;

Step S402: The metadata node opens an area of size M in its memory to store temporary files, where M is a positive integer greater than the above threshold, and its value range is 64-128Mb;

Step S403: the metadata node sets an index area for storing the index information of each file in its data area;

Step S404: the metadata node sets a first-level index area for storing the mapping relationship between files and data nodes;

Step S405: The data node sets the secondary index area, which is located in the data node and is used to store the secondary index information of the file; specifically, the secondary index information includes: the mapping from the file to the data block, the Offset, and file size.

As shown in Figure 5, steps S307 and S308 in the method of the present invention all include the following sub-steps:

Step S501: the metadata node saves the data in its data area as an ordinary file in the distributed file system;

Step S502: the metadata node sends the index information of the ordinary file to the secondary index area of the corresponding data node, and the data node adds the index information to its secondary index area;

Step S503: The metadata node updates its first-level index information according to the file ID and data node ID; specifically, the metadata node adds the mapping relationship between the file ID and the data node ID in its first-level index area, so that the file read query;

Step S504: the metadata node clears the data in its data area;

Step S505: The metadata node clears the data in its index area.

As shown in Figure 6, the file fast reading method under the distributed environment of the present invention comprises the following steps:

Step S601: the client node sends a file read request to the metadata node;

Step S602: The client node judges whether it maintains a connection with the data node connected to the client node last read file in the distributed file system, and if so, proceeds to step S603, otherwise proceeds to step S606; specifically, The connection between the client node and the data node can be a TCP connection or a UDP connection;

Step S603: the client node sends a file read request to the data node;

Step S604: The data node searches according to the secondary index information stored in its secondary index area to determine whether it has stored the file corresponding to the file read request, and if so, proceed to step S609; otherwise, proceed to step S605; Specifically, the secondary index information includes: the mapping of files to data blocks, the offset of files in data blocks, and the size of files;

Step S605: the client node disconnects from the data node;

Step S606: The metadata node inquires whether the file exists in its data area according to the information in its index area, and if so, proceeds to step S611, otherwise proceeds to step S607;

Step S607: The metadata node queries the data node storing the file according to its primary index information (ie, the mapping relationship between the file ID and the data node);

Step S608: the client node establishes a connection with the data node;

Step S609: the data node searches for the data block where the file is located according to the secondary index information, obtains the file according to the secondary index information, and sends the file to the client node;

Step S610: the client node receives the data and maintains a connection with the data node, and then the process ends;

Step S611: the metadata node obtains the file from its data area according to the index information in its index area, and sends the file to the client node.

It is easy for those skilled in the art to understand that the above descriptions are only preferred embodiments of the present invention, and are not intended to limit the present invention. Any modifications, equivalent replacements and improvements made within the spirit and principles of the present invention, All should be included within the protection scope of the present invention.

Claims (7)

1. the quick write method of the file under distributed environment, is characterized in that, comprise the following steps:

Step S301: carry out initialization to the data field of metadata node under distributed environment and index information, wherein index information comprises index area and the one-level index area of metadata node, and the secondary index district of back end; This step comprises following sub-step:

Step S401: judge whether to carry out initialization to the data field of metadata node under distributed environment and index information, if so, then process terminates, otherwise proceeds to step S402;

Step S402: metadata node opens up a size within it in depositing be the region of M, in order to preserve interim file, wherein M is the positive integer being greater than a threshold value;

Step S403: metadata node arranges index area, for storing the index information of each file in its data field;

Step S404: metadata node arranges one-level index area, for preserving the mapping relations of file to back end;

Step S405: back end arranges secondary index district, and it is arranged in back end, for the secondary index information of storage file;

Step S302: client node sends written document request to metadata node;

Step S303: metadata node judges according to written document request whether the remaining space of the data field of metadata node is more than or equal to this file size, if so, then proceeds to step S304, otherwise proceeds to step S308;

Step S304: metadata node receives the file of client node, and is stored into by this file in the remaining space of the data field of metadata node;

Step S305: metadata node upgrades the information of its index area:

Step S306: metadata node judges whether the data stored in the data field of metadata node are greater than above-mentioned threshold value, and if so, then proceed to step S307, else process terminates;

Step S307: the data of its data field are stored in distributed file system as an ordinary file by metadata node, and empty the data in its data field and index area, and process terminates;

Step S308: the data of its data field are stored in distributed file system as an ordinary file by metadata node, and empty the data in its data field and index area; Step S307 and step S308 includes following sub-step:

Step S501: the data of its data field are stored in distributed file system as an ordinary file by metadata node;

Step S502: the index information of this ordinary file is sent in the secondary index district of corresponding back end by metadata node, back end adds this index information to its secondary index district;

Step S503: metadata node upgrades its one-level index information according to the ID of file and back end ID;

Step S504: metadata node empties the data in its data field;

Step S505: metadata node empties the data in its index area;

Step S309: metadata node receives the file data of client node, and is stored in the remaining space of its data field;

Step S310: metadata node upgrades the information of its index area.

2. the quick write method of file according to claim 1, is characterized in that, the size of file is between 0 ~ 1MB, and the size of ordinary file is greater than described threshold value.

3. the quick write method of file according to claim 1, it is characterized in that, step 305 and step S310 are specially, and metadata node adds a new list item in its index area, includes the skew within a data area of file ID, file and the size of file.

4. the quick write method of file according to claim 1, is characterized in that, secondary index information comprises: file is to the mapping of data block, the skew of file in data block and the size of file.

5. the quick write method of file according to claim 1, it is characterized in that, step S503 is specially, and metadata node adds the mapping relations of file ID and back end ID in its one-level index area, to carry out the reading inquiry of file.

6. the fast fast reading method of the file under distributed environment, is characterized in that, comprise the following steps:

Step S601: client node sends to metadata node and reads file request;

Step S602: client node judges that the back end whether himself is connected with the last file reading of this client node in distributed file system remains connection, if so, then proceeds to step S603, otherwise proceeds to step S606;

Step S603: client node sends to this back end and reads file request;

Step S604: back end is inquired about according to the secondary index information stored in its secondary index district, to judge that whether himself stores the file read corresponding to file request, if then proceed to step S609, otherwise proceeds to step S605;

Step S605: client node disconnects the connection with this back end;

Step S606: whether metadata node is present in its data field according to this file of the information inquiry in its index area, if then proceed to step S611, otherwise proceeds to step S607;

Step S607: metadata node has the back end of this file according to the inquiry of its one-level index information;

Step S608: client node and this back end connect;

Step S609: this file, according to the data block at this file place of secondary index information searching, according to secondary index acquisition of information file, and is sent to client node by back end;

Step S610: client node receives data and keeps the connection with this back end, and then process terminates;

Step S611: metadata node obtains file according to the index information in its index area from its data field, and this file is sent to client node.

7. the fast fast reading method of file according to claim 6, is characterized in that, the connection between client node and back end can be that TCP connects or UDP connects.