CN114328431A - Distributed file system - Google Patents

Abstract

The invention discloses a distributed small file system, which comprises a client node, a NameNode node, a backNode node and a DataNode node, wherein the NameNode node is used for managing metadata of a file system, the backupNode node is used as a backup node of the NameNode node, the DataNode node is used for storing real files, the DataNode node is respectively in communication connection with the client node and the NameNode node, the client node is in communication connection with the NameNode node, and the NameNode node is also in communication connection with the backupNode node. In the implementation process of the invention, the nodes can be mutually matched to form a system, the responsibilities among the nodes are clear, the codes are popular and easy to understand, the high available architectural design and high expansibility are realized, and the elastic expansion of the cluster nodes is supported. The NameNode node supports two deployment modes of a single machine and a cluster, and the single machine mode can meet the storage requirements of a large number of small files of small and medium-sized companies. The cluster mode can support the storage requirements of large and medium-sized companies for large quantities of small files.

Description

a distributed file system

technical field

The invention belongs to the technical field of file storage systems, in particular to a distributed small file system.

Background technique

Most of the existing distributed storage systems are developed and implemented in C language and Go language. It is not very compatible in the application field of the Java language technology stack with a huge ecosystem. It is difficult for Java developers to get started, and it is difficult for small and medium-sized companies to maintain. High system latency and frequent system crashes. The system output of high concurrency, high stability and high availability cannot be provided, which increases the maintenance cost of enterprise use. There are certain bottlenecks in file storage and transmission, and horizontal expansion is impossible. At the level of system use, problems such as obscure user manuals and lack of documentation make it difficult to use, and usually require professional guidance to use.

SUMMARY OF THE INVENTION

In view of the above situation, in order to overcome the defects of the prior art, the present invention provides a distributed storage system, which effectively solves the problems in the background art.

In order to achieve the above object, the present invention provides the following technical solutions: a distributed small file system, including a client node, a NameNode node, a BackNode node, a DataNode node and a storage node,

The NameNode node is used to manage the metadata of the file system,

The BackupNode node is responsible for serving as the backup node of the NameNode node,

The DataNode node is responsible for storing real files,

The DataNode node is connected to the client node and the NameNode node respectively, the client node is connected to the NameNode node, and the NameNode node is also connected to the BackupNode node.

Preferably, it also includes a Netty framework, and the NameNode, BackNode, DataNode and storage nodes all communicate based on the Netty framework.

Preferably, ProtoBuf serialization technology is also included.

Preferably, the client node, NameNode node, BackNode node, DataNode node and storage node are all developed based on Java language.

Compared with the prior art, the beneficial effects of the present invention are:

The invention adopts NameNode, DataNode and BackupNode to form a system by collocating three types of nodes, the responsibilities between nodes are clear, the code is easy to understand, the architecture design of high availability and high scalability are realized, and the elastic expansion and contraction of cluster nodes is supported. The NameNode node supports both single-machine and cluster deployment modes. The single-machine mode can meet the storage requirements of small and medium-sized companies for a large number of small files. The cluster mode can support the storage requirements of large and medium-sized companies with a large number of small files.

The invention is developed in Java language, and can be perfectly compatible with the relatively large domestic Java technology ecosystem. For Java developers, it is extremely difficult to get started. When encountering problems, because it is developed in Java language, it is possible to quickly read the source code positioning problem. . It supports basic functions such as uploading, storing, and downloading massive small files. It can store hundreds of millions of small files, and support horizontal expansion. Theoretically, there is no upper limit on the system capacity. Using the relatively mature Netty technology in the industry as the network communication framework, it can provide high concurrency, high performance and high stability of the system performance. At the same time, an advanced active-standby switching mechanism is designed, which has the characteristics of high availability.

Description of drawings

The accompanying drawings are used to provide a further understanding of the present invention, and constitute a part of the specification, and are used to explain the present invention together with the embodiments of the present invention, and do not constitute a limitation to the present invention. In the attached image:

Fig. 1 is the structural representation of the prior art of the present invention;

FIG. 2 is a schematic diagram of the system structure of the present invention.

Detailed ways

The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments; based on the The embodiments of the present invention, and all other embodiments obtained by those of ordinary skill in the art without creative work, fall within the protection scope of the present invention.

As shown in Figure 1-2, the present invention discloses a distributed small file system, which is characterized in that it includes a client node, a NameNode node, a BackNode node, a DataNode node and a storage node,

The NameNode node is responsible for managing the metadata of the file system,

The BackupNode node is responsible for serving as the backup node of the NameNode node,

The DataNode node is responsible for storing real files,

The DataNode node is connected to the client node and the NameNode node respectively, the client node is connected to the NameNode node, and the NameNode node is also connected to the BackupNode node.

It also includes a Netty framework, and the NameNode nodes, BackNode nodes, DataNode nodes and storage nodes all communicate based on the Netty framework.

This system adopts Netty, Google Protobuf technology. The entire system contains nodes with several roles, among which the NameNode node is responsible for managing the metadata of the file system. The BackupNode node is responsible for serving as the backup node of the NameNode node to provide the active-standby switching function. The DataNode node is responsible for storing the real files on the machine disk. Using the Netty framework to achieve high concurrency, high performance file transfer and management functions. Using ProtoBuf serialization technology, it has extremely high performance and compression ratio.

The client node, NameNod node, BackNode node, DataNode node and storage node are all developed based on Java language.

And when the present invention is implemented, a supporting tool set can also be used, including WebUI, operation and maintenance scripts, etc., to provide a visual operation interface and data migration scripts.

Working principle: The client-side integrated SDK sends a request to the NameNode node through the network, and the NameNode node allocates a list of DataNode nodes that store files, and then the client-side SDK establishes a link with the DataNode, and then transfers the file content. Through this architecture design, metadata management and file storage responsibilities are separated, and a scalable system architecture is realized.

It should be noted that, in this document, relational terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any relationship between these entities or operations. any such actual relationship or sequence exists. Moreover, the terms "comprising", "comprising" or any other variation thereof are intended to encompass non-exclusive inclusion such that a process, method, article or device comprising a list of elements includes not only those elements, but also includes not explicitly listed or other elements inherent to such a process, method, article or apparatus.

Although embodiments of the present invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, and substitutions can be made in these embodiments without departing from the principle and spirit of the invention and modifications, the scope of the present invention is defined by the appended claims and their equivalents.

Claims (4)

1. a distributed small file system, is characterized in that: comprise client node, NameNode node, BackNode node, DataNode node and storage node, The NameNod node is used to manage the metadata of the file system, The BackupNode node is responsible for serving as the backup node of the NameNode node, The DataNode node is responsible for storing real files, The DataNode node is connected to the client node and the NameNode node respectively, the client node is connected to the NameNode node, and the NameNode node is also connected to the BackupNode node. 2 . The distributed small file system according to claim 1 , further comprising a Netty framework, and the NameNode nodes, BackNode nodes, DataNode nodes and storage nodes all communicate based on the Netty framework. 3 . 3. A kind of distributed small file system according to claim 1, is characterized in that: also comprises ProtoBuf technology. 4 . The distributed small file system according to claim 1 , wherein the client node, the NameNode node, the BackNode node and the DataNode node are all developed based on the Java language. 5 .

Images