CN110990458B - Distributed database system, interface communication middleware - Google Patents

Abstract

The invention discloses a distributed database system and an interface communication middleware, which relate to the technical field of databases, wherein the distributed database system is designed based on an OceanBase database system and comprises the OceanBase database system, a distributed physical cluster, a management container, a data container, a Docker Swarm cluster, an overlay cross-domain network and a micro-service, the system has high safety and pluggability, when the database system is added with extension, the database system can be constructed on different hosts through micro-services, the same various service modules can be quickly activated, the service is placed in the same network segment, and when the service needs to be quickly extended, the scripted one-key deployment of the cluster based on a micro-service design script can be realized, so that the distributed database system has high expansibility; the interface communication middleware is used for being installed on a client and corresponds to the distributed database system, convenience of program development can be improved, installation is simple, communication efficiency is high, the communication process is encrypted, and the communication process can be well protected from being cracked easily.

Description

Distributed database system, interface communication middleware

technical field

The present invention relates to the technical field of databases, in particular to an OceanBase distributed database system based on a microservice architecture and interface communication middleware corresponding to the distributed database system.

Background technique

Nowadays, most database architectures are designed based on open source databases, many API interfaces are not applicable, and there are many situations such as database performance and data redundancy. Most of today's self-controllable databases have poor stability: customers dare not use them to undertake key businesses (or even ordinary businesses); the ecological environment is poor: there are few forming applications and few cooperative developers. These problems have led to the gradual decline of the database industry, but the OceanBase system is different. Its performance is good, and it can ensure complete operation while ensuring that the system is in a long-term operation and maintenance state. In addition, OceanBase has excellent performance tests and is suitable for large-scale deployment. Strong advantages, the use of micro-service architecture facilitates large-scale distributed cluster deployment, and can greatly reduce maintenance time. But correspondingly, OceanBase has high requirements for the system, and it is extremely difficult to build. No one has proposed a reasonable solution for its microservice architecture and supporting design scheme, and its weak pluggability limits its application scope and application field. , At the same time, the secondary development API interface does not have an excellent design, and it is very difficult to develop based on the database.

Contents of the invention

The present invention is to provide an OceanBase distributed database system and interface communication middleware based on a microservice architecture, which can alleviate the above-mentioned problems.

In order to alleviate the above-mentioned problems, the technical scheme that the present invention takes is as follows:

In the first aspect, the present invention provides a distributed database system, including the OceanBase database system, the OceanBase database system includes the OceanBase management program and the OceanBase data storage program, and the distributed database system also includes:

A distributed physical cluster, which includes three database slaves and a management host, the management host and the database slaves are all created with a system image;

The four containers are respectively a management container and three data containers, and the OceanBase data storage program of the OceanBase database system is created in each data container correspondingly, and the OceanBase management program of the OceanBase database system is created in the In the management container, the three data containers are respectively created on the three database slaves, the management container is created on the management host, and the management container is used to analyze and distribute the input command byte stream Execute for each data container, and the result byte stream obtained after each data container executes the command is sent back to the management container for encryption;

A Docker Swarm cluster, which is used to manage the container, includes a swarm management node created on the management host, and a swarm work node created on each data host;

overlay cross-domain network, which is created on the management host, and generates a virtual network card net, and the four containers are connected through the virtual network card net communication, and ssh inline intercommunication can be realized between the management container and each data container;

Microservices, which are created on the management host, are used to implement resource scheduling and management of the OceanBase management program and the OceanBase data storage program.

The technical effect of this technical solution is: based on the rapid deployment of the container, the container individual leaves the cluster without affecting the overall operation of the cluster, and the creation and separation of individual modules will not affect the performance and operation of the entire distributed architecture, so it has pluggable Pullability; when adding extensions to the database system, it can be built through microservices on different hosts, which can quickly activate the same business modules and place the services on the same network segment. When the services need to be rapidly expanded, it can be realized The cluster scripted one-click deployment based on the microservice design script has high scalability; while ensuring high availability, it also guarantees the security of the original system host, and the attacks and harassment against the system database are transferred to the container , the local environment is guaranteed.

Optionally, the software and hardware requirements of the management host are:

Operating system Deepin 15.11;

Memory: 16G;

Disk: 400G;

File system: EXT4;

Network card: wireless network card;

Docker version: 18.03.1-ce;

Docker image: CentOS Linux release 7.2-1511;

Linux kernel: 3.10.

The technical effect of the technical solution is to ensure the normal operation of the framework and the program.

Optionally, the system image is a CentOS image, which is constructed based on Dockerfile.

The technical effect of the technical solution is: CentOS is easy to maintain, widely used, stable in version, best in performance among similar operating systems, and easy to use.

In a second aspect, the present invention provides an interface communication middleware, the interface communication middleware corresponds to the above-mentioned distributed database system, and includes:

SQL instruction definition module, used to define SQL keywords;

The SQL command analysis module is used to identify the keywords of the SQL commands input by the client and convert them into command byte streams;

The communication process encryption module is used to encrypt the command byte stream;

The communication protocol analysis module is used to analyze the communication protocol adopted by the command byte stream, and inform the management container;

The communication connection module is used to establish communication with the management host, and send the encrypted command byte stream to the management container;

The result processing module is used for decrypting and splicing the result byte stream returned by the management container, and returning the splicing processing result to the client computer.

The technical effect of this technical solution is: through the interface communication middleware, the application scenarios and application fields of the distributed database system can be expanded to a certain extent, making the use of the original database more convenient and increasing the convenience of program development; The interface communication middleware is relatively lightweight, easy to install, convenient for users to use immediately, high communication efficiency, and the communication process is encrypted, which can well protect the communication process from being cracked.

Further, the interface communication middleware is used to be installed on the client computer, so that the client computer and the distributed database system can communicate through the interface communication middleware.

The technical effect of the technical solution is that the underlying difference of the operating system is shielded, and rapid cross-platform development can be realized.

Further, the encryption algorithm adopted by the communication process encryption module is the sha256 algorithm.

The technical effect of this technical solution is: the sha256 algorithm is more common than other encryption algorithms, and its HASH chain is longer, which can more effectively resist cracking and attacking.

Further, the communication mode of the communication connection module is socket communication mode.

The technical effect of this technical solution is: the method of socket communication can effectively ensure that data is not lost, and in the use scenario of a large number of long connections such as database access, it can provide communication more effectively and reliably, and the transmission data is at the byte level, Short transfer times and high performance are suitable for managing interactive communications between host and client, and data can be encrypted.

In order to make the above-mentioned objects, features and advantages of the present invention more comprehensible, the embodiments of the present invention will be described in detail below together with the accompanying drawings.

Description of drawings

In order to illustrate the technical solutions of the embodiments of the present invention more clearly, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present invention, and thus It should be regarded as a limitation on the scope, and those skilled in the art can also obtain other related drawings based on these drawings without creative work.

Fig. 1 is the architectural diagram of the distributed database system described in the embodiment;

Fig. 2 is the architectural diagram of the interface communication middleware described in the embodiment;

Fig. 3 is a working flow chart of the interface communication middleware in the embodiment.

Detailed ways

In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments It is a part of embodiments of the present invention, but not all embodiments. The components of the embodiments of the invention generally described and illustrated in the figures herein may be arranged and designed in a variety of different configurations.

Accordingly, the following detailed description of the embodiments of the invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely represents selected embodiments of the invention. Based on the embodiments of the present invention, all other embodiments obtained by persons of ordinary skill in the art without creative efforts fall within the protection scope of the present invention.

Example 1

Please refer to Fig. 1, the present embodiment provides a kind of distributed database system, including: OceanBase database system, it includes OceanBase management program and OceanBase data storage program; Distributed physical cluster, it includes three database slaves and a management The host, management host, and database slave are all created with system images; four containers are a management container and three data containers, and the OceanBase data storage program of the OceanBase database system is created in each data container correspondingly. OceanBase The OceanBase management program of the database system is created in the management container. The three data containers are respectively created on the three database slaves. The management container is created on the management host. The management container is used to analyze the input command byte stream and distribute it to each The data container is executed, and the result byte stream obtained after each data container executes the command is sent back to the management container for encryption; the Docker Swarm cluster is used to manage the container, which includes the swarm management node created on the management host, and the swarm management node created on each data container. The swarm working node of the host; the overlay cross-domain network, which is created on the management host and generates a virtual network card net, and the four containers are connected through the virtual network card net communication, and the management container and each data container can realize ssh inline intercommunication; The service, which is created on the management host, is used to realize the resource scheduling and management of the OceanBase management program and the OceanBase data storage program.

In this embodiment, the software and hardware requirements of the management host are: operating system Deepin 15.11; memory: 16G; disk: 400G; file system: EXT4; network card: wireless network card; Docker version: 18.03.1-ce; Docker image: CentOSLinux release 7.2-1511; Linux kernel: 3.10.

In this embodiment, the system images of the management host and the database slave are CentOS images, and the CentOS images are constructed based on Dockerfile. The entire Dockerfile has the following 8 parts: pull the basic image, install the basic software dependencies, install the software dependencies required by OceanBase, configure the SSH config file, add the admin user and grant permissions, download the OceanBase installation package, modify the system configuration information, and complete Firewall configuration.

In this embodiment, the process of creating a DockerSwarm cluster, microservice, cross-domain network, and container is as follows:

1) Build a Docker Swarm cluster on the management host: build a swarm management node on a management host with relatively better performance, initialize it, and then add a swarm working node to the three database slaves, check it through dockernodels, and complete the cluster deployment.

2) Create an overlay cross-domain network on the DockerSwarm cluster of the management host, and complete the creation of the cross-domain network net on the management host through the command "docker network create--driver=overlay-attachablenet", generate a net virtual network card, and then make three The database slave machine is added to the cross-domain network net.

3) Create a deployment script docker-compose.yml on the management host. docker-compose.yml includes virtual network card net, system image, and Docker Swarm cluster. Run "docker compose" on the management host to create microservices and create 3+1 containers (three data containers + one management container). The management container is responsible for scheduling database container resources and responding to external database requests.

In this embodiment, the OceanBase database system is the program body of the distributed database system and the main operating module in the container. The deployment process of the OceanBase database system is as follows:

1) Perform performance tuning on the management host, database slave and each container, configure the maximum number of threads, and modify the swap memory;

2) After the 3+1 container is built, the ssh inline communication between the management container and the three data containers is realized to realize ssh passwordless login;

3) Create the /home/admin/ob-deploy directory on the management container, copy the deploy.py file in the installation package to this directory and execute the command to generate the target directory bin directory.

4) Unzip the OceanBase installation package on the management container, copy observer and obproxy to the bin directory

5) Modify the ssh default port of the management container and data container, and then generate and modify the observer configuration file to modify the IP configuration address.

6) Run ./deploy.py ob1.reboot to deploy the management container to the data container service, run ./deploy.py ob1.sql, and connect to the Docker Swarm cluster through this step.

7) The test is correct, the management container is packaged, a new Docker image is generated, and a new docker-compose.yml is generated during the service deployment process.

Example 2

Please refer to Fig. 2, this implementation provides a kind of interface communication middleware, this interface communication middleware corresponds to the distributed database system in embodiment 1, and includes:

SQL instruction definition module, used to define SQL keywords;

The SQL command analysis module is used to identify the keywords of the SQL commands input by the client and convert them into command byte streams;

The communication process encryption module is used to encrypt the command byte stream;

The communication protocol analysis module is used to analyze the communication protocol adopted by the command byte stream, and inform the management container;

The communication connection module is used to establish communication with the management host, and send the encrypted command byte stream to the management container;

The result processing module is used for decrypting and splicing the result byte stream returned by the management container, and returning the splicing processing result to the client computer.

In this embodiment, the interface communication middleware is used to be installed on the client computer, so that the client computer and the distributed database system can communicate through the interface communication middleware; the encryption algorithm adopted by the communication process encryption module is the sha256 algorithm; the communication connection The communication mode of the module is socket communication mode.

Please refer to Fig. 2 and Fig. 3, the use method of the interface communication middleware described in this embodiment is as follows:

1. The client completes the sql command design, simulates the request, and converts the command block into a command byte stream by calling the SQL command definition module of the interface;

2. Send a connection request to the management container through the communication connection module. Before sending the request, the communication connection module needs to analyze the communication protocol adopted by the container, analyze the sql command, and encrypt the command byte stream through the communication process encryption module;

3. If the connection fails, the whole process will be ended directly. If the connection is successful, a socket connection will be created, the encrypted command bytes will be transmitted to the management container, and step 4 will be continued;

4. The management container receives the command byte stream, parses the command byte stream and distributes it to each data container to execute the command, and each data container searches the database to obtain the result byte stream, and the management container performs the same processing on the result byte stream. Encryption measures, and then sent back to the result processing module;

5. The result processing module decrypts and splices the result byte stream returned by the management container, and returns the splicing processing result to the client;

6. The client computer closes the database connection to complete the whole process.

The above are only preferred embodiments of the present invention, and are not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and changes. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present invention shall be included within the protection scope of the present invention.

Claims (6)

1. A distributed database system, comprising OceanBase database system, which comprises OceanBase management program and OceanBase data storage program, is characterized in that, comprising: A distributed physical cluster, which includes three database slaves and a management host, the management host and the database slaves are all created with a system image; The four containers are respectively a management container and three data containers, and the OceanBase data storage program of the OceanBase database system is created in each data container correspondingly, and the OceanBase management program of the OceanBase database system is created in the In the management container, the three data containers are respectively created on the three database slaves, the management container is created on the management host, and the management container is used to analyze and distribute the input command byte stream Execute for each data container, and the result byte stream obtained after each data container executes the command is sent back to the management container for encryption; A Docker Swarm cluster, which is used to manage the container, includes a swarm management node created on the management host, and a swarm work node created on each data host; overlay cross-domain network, which is created on the management host, and generates a virtual network card net, and the four containers are connected through the virtual network card net communication, and ssh inline intercommunication can be realized between the management container and each data container; Micro-service, which is created on the management host, for realizing resource scheduling and management of OceanBase management program and OceanBase data storage program; The system image is a CentOS image, which is built based on Dockerfile. 2. The distributed database system according to claim 1, wherein the software and hardware requirements of the management host are: Operating system Deepin 15.11; Memory: 16G; Disk: 400G; File system: EXT4; Network card: wireless network card; Docker version: 18.03.1-ce; Docker image: CentOS Linux release 7.2-1511; Linux kernel: 3.10. 3. A kind of interface communication middleware, it is characterized in that, this interface communication middleware corresponds to the distributed database system described in claim 1, and comprises: SQL instruction definition module, used to define SQL keywords; The SQL command analysis module is used to identify the keywords of the SQL commands input by the client and convert them into command byte streams; The communication process encryption module is used to encrypt the command byte stream; The communication protocol analysis module is used to analyze the communication protocol adopted by the command byte stream, and inform the management container; The communication connection module is used to establish communication with the management host, and send the encrypted command byte stream to the management container; The result processing module is used for decrypting and splicing the result byte stream returned by the management container, and returning the splicing processing result to the client computer. 4. The interface communication middleware according to claim 3, characterized in that, the interface communication middleware is used to be installed on the client computer, so that the client computer and the distributed database system can pass through the interface communication middleware to communicate. 5. The interface communication middleware according to claim 3, characterized in that, the encryption algorithm adopted by the communication process encryption module is sha256 algorithm. 6. The interface communication middleware according to claim 3, characterized in that, the communication mode of the communication connection module is a socket communication mode.

Images