CN118689630A - A GPU service tidal deployment method and device based on K8s - Google Patents

Abstract

The invention discloses a K8 s-based GPU service tide deployment method and device, wherein the K8 s-based GPU service tide deployment method comprises the following steps: deploying a main control node tidal service tide-master, a working node tidal service tide-worker and a gateway tidal service tide-gateway based on a K8s platform; defining operation information of the GPU service instance by defining CRD through the custom resource; the master control node tidal service tide-master monitors user-defined resource definition CRD and GPU resource information, and all tide-worker nodes of the working node tidal service tide-worker are initialized and started according to the GPU resource information; the master node tidal service tide-master dynamically adjusts the states and flow allocation policies of the working node tidal service tide-worker and the gateway tidal service tide-gateway according to actual GPU resource conditions, GPU use conditions of models in each tide-worker node and flow conditions of the gateway tidal service tide-gateway. According to the technical scheme, the system is timely adjusted according to the model requirements and flow changes, the system is guaranteed to be in an optimal state, and the execution speed and the execution effect of deep learning and artificial intelligence application are improved.

Description

A GPU service tidal deployment method and device based on K8s

Technical Field

The present invention relates to the technical field of distributed systems and resource management, and specifically provides a GPU service tidal deployment method and device based on K8s.

Background Art

In today's artificial intelligence and deep learning applications, using GPUs for computing is one of the key factors in improving model training and inference performance. However, the management and scheduling of GPU resources is a challenge for distributed GPU services. Currently, Kubernetes has become the de facto standard for containerized application management, but it still faces some challenges in GPU resource management, such as static binding, low resource utilization, and insufficient scaling.

In traditional GPU resource management, inference tasks and training tasks are usually handled separately. Inference tasks generally have lower computational complexity and memory requirements, while training tasks require more computing and memory resources. In order to improve GPU resource utilization, existing technologies statically deploy and scale inference tasks, while training tasks are managed using a separate resource pool.

In the prior art, static allocation and binding of GPU services is a common method, but it has the following disadvantages:

Static binding: Static allocation and binding cannot meet dynamic service requirements, resulting in low resource utilization. Due to changes in models and traffic, some GPU resources may remain idle, while others may be overloaded, resulting in performance degradation. Therefore, static binding leads to low resource utilization and performance degradation.

Scaling: Existing technologies are insufficient in the dynamic scaling of GPU services. In static binding, scaling is often done manually or in a scheduled manner, or through observation methods such as GPU load and logs to make decisions. Moreover, scaling is often done at the pod level, which comes with many additional operations, such as mounting and unmounting nfs or downloading and cleaning models, which are all performed simultaneously during the pod startup and recycling process, affecting the pod startup and recycling speed. Frequent mounting and unmounting of the file system caused by frequent scaling may cause operating system abnormalities.

Low resource utilization: Because inference and training tasks are processed separately, GPU resources may be idle or overloaded at some point, and resources cannot be fully utilized.

It can be seen that the traditional static deployment method often cannot effectively utilize GPU resources, resulting in resource waste or insufficient resources. Therefore, the present invention aims to solve how to automatically and intelligently manage and schedule GPU resources in a dynamic GPU scenario to meet the ever-changing model requirements and traffic conditions.

Summary of the invention

In view of the above technical problems, the present invention proposes a GPU service tidal deployment method and device based on K8s to solve the problems of resource management and deployment optimization in dynamic GPU scenarios and improve the efficiency of the system and user experience.

Specifically, the following technical solutions are adopted:

In a first aspect, the present invention provides a GPU service tidal deployment method based on K8s, comprising:

Deploy the master node tide service tide-master, the worker node tide service tide-worker, and the gateway tide service tide-gateway based on the K8s platform;

Define the running information of the GPU service instance through the custom resource definition CRD;

The master control node tide service tide-master monitors the custom resource definition CRD and GPU resource information, and initializes all tide-worker nodes of the starting working node tide service tide-worker according to the GPU resource information;

The master control node tide service tide-master dynamically adjusts the status and traffic allocation strategy of the working node tide service tide-worker and the gateway tide service tide-gateway according to the actual GPU resource situation, the GPU usage of the model in each tide-worker node and the traffic situation of the gateway tide service tide-gateway.

As an optional implementation of the present invention, in a GPU service tide deployment method based on K8s of the present invention, the master control node tide service tide-master listens to the custom resource definition CRD and GPU resource information, and initializes all tide-worker nodes of the starting working node tide service tide-worker according to the GPU resource information, including:

Start the master node tide service tide-master, the worker node tide service tide-worker and the gateway tide service tide-gateway;

The master control node tide service tide-master listens to the custom resource definition CRD and GPU resource information, and starts all tide-worker nodes of the working node tide service tide-worker according to the GPU resource information. After the tide-worker node is started, it will asynchronously load all models in the CRD;

The working node tide service tide-worker polls the master control node tide service tide-master to obtain the model information or training tasks that need to be run;

The gateway tide service tide-gateway monitors the master control node tide service tide-master to obtain the PodIP corresponding to the model and the corresponding relationship information of the model.

As an optional embodiment of the present invention, in a GPU service tide deployment method based on K8s of the present invention, the master control node tide service tide-master dynamically adjusts the status and traffic allocation strategy of the working node tide service tide-worker and the gateway tide service tide-gateway according to the actual resource situation, the GPU usage of the model in each tide-worker node and the traffic situation of the gateway tide service tide-gateway, including:

The master control node tide service tide-master regularly pulls the traffic information of the gateway tide service tide-gateway and the error rate and GPU usage rate of the model under the working node tide service tide-worker;

The master control node tide service tide-master determines whether it is necessary to adjust the association relationship between the tide-worker node and the model in the working node tide service tide-worker, or the number of Pods of the gateway tide service tide-gateway according to the threshold.

As an optional implementation of the present invention, in a GPU service tide deployment method based on K8s of the present invention, the master control node tide service tide-master determines whether it is necessary to adjust the association relationship between the tide-worker node and the model in the working node tide service tide-worker, or the number of Pods of the gateway tide service tide-gateway according to the threshold value, including:

If the traffic that the master node tide service tide-master regularly pulls from the gateway tide service tide-gateway is greater than the preset traffic threshold;

Alternatively, the error rate of the model under the working node tide service tide-worker periodically pulled by the master control node tide service tide-master is greater than the preset error rate;

Alternatively, the GPU usage rate of the model under the working node tide service tide-worker that is periodically pulled by the master control node tide service tide-master is greater than the preset GPU usage rate;

The master node tide service tide-master determines that it is necessary to adjust the association between the tide-worker node and the model in the working node tide service tide-worker, or the number of Pods in the gateway tide service tide-gateway.

As an optional implementation of the present invention, a GPU service tidal deployment method based on K8s of the present invention includes:

The working node tide service tide-worker polls the master control node tide service tide-master to obtain model information or training tasks;

When it is perceived that the association between the tide-worker node and the model in the working node tide service tide-worker is adjusted, the working node tide service tide-worker closes the originally running model and switches to the adjusted model.

As an optional implementation of the present invention, a GPU service tidal deployment method based on K8s of the present invention includes:

The gateway tide service tide-gateway monitors the association between the tide-worker node and the model in the master node tide service tide-master, and is responsible for forwarding the traffic request of the corresponding model to the Pod of the corresponding working node tide service tide-worker;

After the gateway tide service tide-gateway detects that the association relationship information between the tide-worker node and the model has changed, it stops distributing the traffic of the model to the removed tide-worker node.

As an optional implementation of the present invention, in a K8s-based GPU service tidal deployment method of the present invention, the operation information of the GPU service instance defined by the custom resource definition CRD includes: the model operation required information and routing information of the GPU service instance defined by the custom resource definition CRD.

In a second aspect, the present invention provides a GPU service tidal deployment device based on K8s, comprising:

The master node tide module deploys the master node tide service tide-master based on the K8s platform;

The worker node tide module deploys the worker node tide service tide-worker based on the K8s platform;

The gateway tide module deploys the gateway tide service tide-gateway based on the K8s platform;

Custom resource definition module CRD, which defines the running information and routing information of the GPU service instance;

The master node tide module monitors the custom resource definition module CRD and GPU resource information, and starts all tide-worker nodes of the working node tide service tide-worker according to the GPU resource information during initialization;

The master node tide module dynamically adjusts the status and traffic allocation strategy of the working node tide service tide-worker and the gateway tide service tide-gateway according to the actual GPU resource situation, the GPU usage of the model in each tide-worker node and the traffic situation of the gateway tide service tide-gateway.

In a third aspect, the present invention provides an electronic device comprising a processor and a memory, wherein the memory is used to store a computer executable program, and when the computer program is executed by the processor, the processor executes the K8s-based GPU service tidal deployment method.

In a fourth aspect, the present invention provides a computer-readable recording medium storing a computer executable program, which, when executed, implements the K8s-based GPU service tidal deployment method.

Compared with the prior art, the present invention has the following beneficial effects:

Automation and intelligence: The GPU service tide deployment method based on K8s of the present invention realizes the automatic dynamic deployment of GPU services through custom CRD and dynamic adjustment mechanism. The master node tide service tide-master monitors CRD and GPU resource information, and intelligently adjusts the status and traffic distribution strategy of the working node tide service tide-worker and the gateway tide service tide-gateway according to the actual situation, realizing automatic and intelligent service management.

Real-time and responsiveness: Through real-time monitoring, prediction and dynamic adjustment, the GPU service tidal deployment method based on K8s of the present invention can be adjusted in time according to model requirements and traffic changes. It has real-time and responsiveness, can quickly adapt to changing needs, and improve the flexibility and efficiency of services.

Resource optimization and utilization improvement: By dynamically adjusting and optimizing GPU resource allocation, a GPU service tidal deployment method based on K8s of the present invention can achieve optimal resource utilization. It dynamically allocates and adjusts resources according to model requirements and traffic conditions, avoids resource waste and load imbalance, and improves resource utilization and the overall efficiency of the system. Moreover, by dynamically scaling training and reasoning tasks, GPU resources are flexibly allocated according to actual load conditions, thereby improving resource utilization. In the case of resource constraints, training tasks can be shrunk and reasoning tasks can be expanded, thereby making more effective use of limited resources.

Scalability and extensibility: The GPU service tidal deployment method based on K8s of the present invention is based on the Kubernetes platform and has good scalability and extensibility. It can quickly expand and reduce the number of running models according to actual needs to adapt to the growing number of models and changing traffic conditions.

High availability and stability: Through dynamic deployment and adjustment mechanisms, the GPU service tidal deployment method based on K8s of the present invention can improve the availability and stability of GPU services. It can automatically adjust and optimize service configuration according to the results of real-time monitoring and prediction, avoid service failure or overload, and improve the stability and reliability of the system.

Real-time monitoring and control capabilities: All traffic is managed and monitored through the Tide Gateway service tide-gateway, which has the ability to monitor and control in real time. Traffic can be limited and downgraded in real time to ensure the stability of the system and make timely adjustments based on actual needs.

Therefore, compared with normal static deployment, GPU resources are usually statically allocated to different models, resulting in some GPU resources being idle while others being overloaded. The present invention's tidal deployment method for GPU services based on K8s intelligently allocates GPU resources according to actual model requirements and traffic conditions through dynamic adjustment, thereby improving the utilization of GPU resources. Due to the dynamic allocation and optimization of GPU resources, as well as real-time service load balancing, the technical solution can reasonably allocate resources and optimize system performance and efficiency. The technical solution makes timely adjustments based on model requirements and traffic changes to ensure that the system is in the best state and improve the execution speed and effect of deep learning and artificial intelligence applications.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG1 is an overall structural diagram of a GPU service tidal deployment device based on K8s according to an embodiment of the present invention;

FIG2 is a processing flow chart of a GPU service tidal deployment method based on K8s according to an embodiment of the present invention;

FIG3 is a working sequence diagram of a GPU service tidal deployment method based on K8s according to an embodiment of the present invention;

FIG4 is a schematic structural diagram of an electronic device according to an embodiment of the present invention;

FIG. 5 is a schematic diagram of a computer-readable recording medium according to an embodiment of the present invention.

DETAILED DESCRIPTION

To make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be described clearly and completely in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of them.

Therefore, the following detailed description of the embodiments of the present invention is not intended to limit the scope of the invention claimed for protection, but merely represents some embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

It should be noted that, in the absence of conflict, the embodiments of the present invention and the features and technical solutions in the embodiments may be combined with each other.

It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, it does not require further definition and explanation in the subsequent drawings.

In the description of the present invention, it should be noted that the terms "upper", "lower", etc. indicate the orientation or positional relationship based on the orientation or positional relationship shown in the drawings, or the orientation or positional relationship in which the invention product is usually placed when in use, or the orientation or positional relationship commonly understood by those skilled in the art. Such terms are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as limiting the present invention. In addition, the terms "first", "second", etc. are only used to distinguish the description, and cannot be understood as indicating or implying relative importance.

Embodiment 1

A GPU service tidal deployment method based on K8s in this embodiment includes:

Deploy the master node tide service tide-master, the worker node tide service tide-worker, and the gateway tide service tide-gateway based on the K8s platform;

Define the running information of the GPU service instance through the custom resource definition CRD;

The master control node tide service tide-master monitors the custom resource definition CRD and GPU resource information, and initializes all tide-worker nodes of the starting working node tide service tide-worker according to the GPU resource information;

The master control node tide service tide-master dynamically adjusts the status and traffic allocation strategy of the working node tide service tide-worker and the gateway tide service tide-gateway according to the actual GPU resource situation, the GPU usage of the model in each tide-worker node and the traffic situation of the gateway tide service tide-gateway.

A GPU service tide deployment method based on K8s in this embodiment is based on the K8s (i.e. Kubernetes) platform, and implements a solution for dynamic deployment of GPU services. The solution includes three services: master node tide service tide-master, worker node tide service tide-worker and gateway tide service tide-gateway, and supports custom CRD (custom resource definition). The model operation information and routing information of the GPU service instance are defined by CRD. The master node tide service tide-master will monitor CRD and GPU resource information, and dynamically adjust the status and traffic allocation strategy of the working node tide service tide-worker and the gateway tide service tide-gateway according to the actual GPU resource situation, the GPU usage of the model in each tide-worker node and the traffic situation of the gateway tide service tide-gateway, so as to realize the dynamic expansion and contraction model operation, and manage the model information that needs to be run for the Pod allocation of the working node tide service tide-worker.

In a K8s-based GPU service tidal deployment method of this embodiment, the operation information of the GPU service instance defined by the custom resource definition CRD includes: the model operation information and routing information required for defining the GPU service instance by the custom resource definition CRD.

The GPU service tidal deployment method based on K8s in this embodiment has the following technical effects:

Automation and intelligence: A GPU service tide deployment method based on K8s in this embodiment realizes automatic dynamic deployment of GPU services through custom CRD and dynamic adjustment mechanism. The master node tide service tide-master monitors CRD and GPU resource information, and intelligently adjusts the status and traffic distribution strategy of the working node tide service tide-worker and the gateway tide service tide-gateway according to the actual situation, realizing automatic and intelligent service management.

Real-time and responsiveness: Through real-time monitoring, prediction and dynamic adjustment, the GPU service tidal deployment method based on K8s in this embodiment can be adjusted in time according to model requirements and traffic changes. It has real-time and responsiveness, can quickly adapt to changing needs, and improve the flexibility and efficiency of services.

Resource optimization and utilization improvement: By dynamically adjusting and optimizing GPU resource allocation, a GPU service tidal deployment method based on K8s in this embodiment can achieve optimal resource utilization. It dynamically allocates and adjusts resources according to model requirements and traffic conditions, avoids resource waste and load imbalance, and improves resource utilization and overall system efficiency. Moreover, by dynamically scaling training and reasoning tasks, GPU resources are flexibly allocated according to actual load conditions, thereby improving resource utilization. In the case of tight resources, training tasks can be shrunk and reasoning tasks can be expanded, thereby making more effective use of limited resources.

Scalability and extensibility: The GPU service tidal deployment method based on K8s in this embodiment is based on the Kubernetes platform and has good scalability and extensibility. It can quickly expand and reduce the number of running models according to actual needs to adapt to the growing number of models and changing traffic conditions.

High availability and stability: Through dynamic deployment and adjustment mechanisms, the GPU service tidal deployment method based on K8s in this embodiment can improve the availability and stability of GPU services. It can automatically adjust and optimize service configuration based on real-time monitoring and prediction results, avoid service failures or overload states, and improve system stability and reliability.

Real-time monitoring and control capabilities: All traffic is managed and monitored through the Tide Gateway service tide-gateway, which has the ability to monitor and control in real time. Traffic can be limited and downgraded in real time to ensure the stability of the system and make timely adjustments based on actual needs.

Therefore, compared with normal static deployment, GPU resources are usually statically allocated to different models, resulting in some GPU resources being idle while others being overloaded. A tidal deployment method for GPU services based on K8s in this embodiment intelligently allocates GPU resources according to actual model requirements and traffic conditions through dynamic adjustment, thereby improving the utilization of GPU resources. Due to the dynamic allocation and optimization of GPU resources, as well as real-time service load balancing, this technical solution can reasonably allocate resources and optimize system performance and efficiency. This technical solution makes timely adjustments based on model requirements and traffic changes to ensure that the system is in the best state and improve the execution speed and effect of deep learning and artificial intelligence applications.

Furthermore, compared with traditional static deployment and scaling mechanisms, a K8s-based GPU service tidal deployment method in this embodiment starts all Pods when GPU resources are ready, and asynchronously processes operations such as NFS mounting and model downloading during CRD definition, thereby reducing the time it takes to pull up and recycle the Pod and improving the performance of the overall system.

Specifically, in a GPU service tide deployment method based on K8s in this embodiment, the master control node tide service tide-master listens to the custom resource definition CRD and GPU resource information, and initializes all tide-worker nodes of the starting working node tide service tide-worker according to the GPU resource information, including:

Start the master node tide service tide-master, the worker node tide service tide-worker and the gateway tide service tide-gateway;

The master control node tide service tide-master listens to the custom resource definition CRD and GPU resource information, and starts all tide-worker nodes of the working node tide service tide-worker according to the GPU resource information. After the tide-worker node is started, it will asynchronously load all models in the CRD;

The working node tide service tide-worker polls the master control node tide service tide-master to obtain the model information or training tasks that need to be run;

The gateway tide service tide-gateway monitors the master control node tide service tide-master to obtain the PodIP corresponding to the model and the corresponding relationship information of the model.

It can be seen that a GPU service tide deployment method based on K8s in this embodiment has the above-mentioned initialization process. CRD is used in the initialization process to define model operation information and routing information, which enables the master node tide service tide-master to obtain relevant information and routing information of the model in real time according to CRD, and dynamically adjust the status and traffic distribution strategy of the working node tide service tide-worker and the gateway tide service tide-gateway accordingly.

Furthermore, a GPU service tide deployment method based on K8s in this embodiment also has a periodic dynamic adjustment process. Specifically, the master control node tide service tide-master dynamically adjusts the status and traffic allocation strategy of the working node tide service tide-worker and the gateway tide service tide-gateway according to the actual resource situation, the GPU usage of the model in each tide-worker node and the traffic situation of the gateway tide service tide-gateway, including:

The master control node tide service tide-master regularly pulls the traffic information of the gateway tide service tide-gateway and the error rate and GPU usage rate of the model under the working node tide service tide-worker;

The master control node tide service tide-master determines whether it is necessary to adjust the association relationship between the tide-worker node and the model in the working node tide service tide-worker, or the number of Pods of the gateway tide service tide-gateway according to the threshold.

Among them, GPU utilization rate refers to the actual utilization degree of GPU device in a certain period of time, generally expressed as a percentage. By monitoring GPU utilization rate, you can understand the load of GPU device and make reasonable resource adjustments.

In this embodiment, a periodic dynamic adjustment process in a K8s-based GPU service tidal deployment method is implemented. Traffic information, error rate, and GPU usage rate are periodically pulled, and dynamically adjusted according to thresholds. The mechanism of the periodic dynamic adjustment process ensures the real-time and stability of the service, and can automatically adjust GPU resource allocation according to the needs of different models.

Further, in a GPU service tide deployment method based on K8s in this embodiment, the master control node tide service tide-master determines whether it is necessary to adjust the association relationship between the tide-worker node and the model in the working node tide service tide-worker, or the number of Pods of the gateway tide service tide-gateway according to the threshold value, including:

If the traffic that the master node tide service tide-master regularly pulls from the gateway tide service tide-gateway is greater than the preset traffic threshold;

Alternatively, the error rate of the model under the working node tide service tide-worker periodically pulled by the master control node tide service tide-master is greater than the preset error rate;

Alternatively, the GPU usage rate of the model under the working node tide service tide-worker that is periodically pulled by the master control node tide service tide-master is greater than the preset GPU usage rate;

The master node tide service tide-master determines that it is necessary to adjust the association between the tide-worker node and the model in the working node tide service tide-worker, or the number of Pods in the gateway tide service tide-gateway.

Furthermore, a GPU service tidal deployment method based on K8s in this embodiment includes:

The working node tide service tide-worker polls the master control node tide service tide-master to obtain model information or training tasks;

When it is perceived that the association between the tide-worker node and the model in the working node tide service tide-worker is adjusted, the working node tide service tide-worker closes the originally running model and switches to the adjusted model.

A GPU service tidal deployment method based on K8s in this embodiment also includes:

The gateway tide service tide-gateway monitors the association between the tide-worker node and the model in the master node tide service tide-master, and is responsible for forwarding the traffic request of the corresponding model to the Pod of the corresponding working node tide service tide-worker;

After the gateway tide service tide-gateway detects that the association relationship information between the tide-worker node and the model has changed, it stops distributing the traffic of the model to the removed tide-worker node.

As shown in FIG2 , a flowchart of a periodic dynamic adjustment process of a tidal deployment method for a GPU service based on K8s in this embodiment includes:

Step 1: The master node tide-master regularly pulls the traffic information, error rate, and GPU usage of the models under the gateway tide-gateway and the worker node tide-worker.

Step 2: The master node tide-master determines whether it is necessary to adjust the association between the worker and the model or the number of Pods of the gateway tide-gateway based on the threshold.

Step 3: If adjustment is required, the master node tide service tide-master adjusts the model information of the working node tide service tide-worker.

Step 4: After the gateway tide service tide-gateway detects that the information of the working node tide service tide-worker and the model has changed, it stops distributing the traffic of the model to the removed tide-worker node.

Step 5: After the working node tide service tide-worker listens to the new model information, it closes the previously running model and switches to the new model.

Referring to FIG. 3 , a timing diagram of a GPU service tidal deployment method based on K8s in this embodiment is shown. Through the above flow chart and timing diagram, it can be seen that:

The master node tide service tide-master monitors CRD and GPU resource information, starts all tide-workers according to the GPU resource information during initialization, and dynamically adjusts the association between tide-workers and models according to actual conditions.

tide-gateway monitors the association between tide-worker and the model in tide-master, and is responsible for forwarding the traffic requests of the corresponding model to the corresponding tide-workerPod.

tide-worker is responsible for polling tide-master to obtain the required model information or training tasks, and runs the corresponding model or training tasks based on the information.

Through such a process, this technical solution realizes the functions of dynamic expansion and contraction and deployment, reduces the time required for model instance adjustment, speeds up the startup of the model, improves the resource utilization and performance of GPU services, and realizes real-time service load balancing and resource optimization.

Scaling: refers to automatically scaling the system's resource size based on load conditions. When the system load is high, resources are automatically expanded to meet demand; when the system load is low, resources are automatically reduced to avoid resource waste. For static deployment, scaling refers to pod scaling. For this solution, because tide-worker nodes are created at the time of initialization based on the number of resources, the scaling of this solution refers to the scaling of the running model, that is, the association between tide-worker nodes and models. Scaling means adding more tide-worker nodes to run the model, and scaling means removing the association between some tide-worker nodes and the model, rather than scaling the pods of tide-worker nodes.

As shown in FIG1 , this embodiment also provides a GPU service tidal deployment device based on K8s, including:

The master node tide module deploys the master node tide service tide-master based on the K8s platform;

The worker node tide module deploys the worker node tide service tide-worker based on the K8s platform;

The gateway tide module deploys the gateway tide service tide-gateway based on the K8s platform;

Custom resource definition module CRD, which defines the running information and routing information of the GPU service instance;

The master node tide module monitors the custom resource definition module CRD and GPU resource information, and starts all tide-worker nodes of the working node tide service tide-worker according to the GPU resource information during initialization;

The master node tide module dynamically adjusts the status and traffic allocation strategy of the working node tide service tide-worker and the gateway tide service tide-gateway according to the actual GPU resource situation, the GPU usage of the model in each tide-worker node and the traffic situation of the gateway tide service tide-gateway.

A GPU service tide deployment device based on K8s in this embodiment is based on the K8s (i.e. Kubernetes) platform, and implements a solution for dynamic deployment of GPU services. The solution includes three services: master node tide service tide-master, worker node tide service tide-worker and gateway tide service tide-gateway, and supports custom CRD (custom resource definition). The model operation information and routing information of the GPU service instance are defined through CRD. The master node tide service tide-master will monitor CRD and GPU resource information, and dynamically adjust the status and traffic allocation strategy of the working node tide service tide-worker and the gateway tide service tide-gateway according to the actual GPU resource situation, the GPU usage of the model in each tide-worker node and the traffic situation of the gateway tide service tide-gateway, so as to realize the dynamic expansion and contraction model operation, and manage the model information that needs to be run for the Pod allocation of the working node tide service tide-worker.

Therefore, the GPU service tidal deployment device based on K8s in this embodiment has the following technical effects:

Automation and intelligence: A GPU service tide deployment device based on K8s in this embodiment realizes automatic dynamic deployment of GPU services through custom CRD and dynamic adjustment mechanism. The master node tide service tide-master monitors CRD and GPU resource information, and intelligently adjusts the status and traffic distribution strategy of the working node tide service tide-worker and the gateway tide service tide-gateway according to the actual situation, realizing automatic and intelligent service management.

Real-time and responsiveness: Through real-time monitoring, prediction and dynamic adjustment, the GPU service tidal deployment device based on K8s in this embodiment can be adjusted in time according to model requirements and traffic changes. It has real-time and responsiveness, can quickly adapt to changing needs, and improve the flexibility and efficiency of services.

Resource optimization and utilization improvement: By dynamically adjusting and optimizing GPU resource allocation, a GPU service tidal deployment device based on K8s in this embodiment can achieve optimal resource utilization. It dynamically allocates and adjusts resources according to model requirements and traffic conditions, avoids resource waste and load imbalance, and improves resource utilization and overall system efficiency. Moreover, by dynamically scaling training and reasoning tasks, GPU resources are flexibly allocated according to actual load conditions, thereby improving resource utilization. In the case of resource constraints, training tasks can be shrunk and reasoning tasks can be expanded, thereby making more effective use of limited resources.

Scalability and extensibility: The GPU service tidal deployment device based on K8s in this embodiment is based on the Kubernetes platform and has good scalability and extensibility. It can quickly expand and reduce the number of running models according to actual needs to adapt to the growing number of models and changing traffic conditions.

High availability and stability: Through dynamic deployment and adjustment mechanisms, a GPU service tidal deployment device based on K8s in this embodiment can improve the availability and stability of GPU services. It can automatically adjust and optimize service configuration based on real-time monitoring and prediction results, avoid service failures or overload states, and improve system stability and reliability.

Real-time monitoring and control capabilities: All traffic is managed and monitored through the Tide Gateway service tide-gateway, which has the ability to monitor and control in real time. Traffic can be limited and downgraded in real time to ensure the stability of the system and make timely adjustments based on actual needs.

Therefore, compared with normal static deployment, GPU resources are usually statically allocated to different models, resulting in some GPU resources being idle while others being overloaded. A GPU service tidal deployment device based on K8s in this embodiment intelligently allocates GPU resources according to actual model requirements and traffic conditions through dynamic adjustment, thereby improving the utilization of GPU resources. Due to the dynamic allocation and optimization of GPU resources and real-time service load balancing, this technical solution can reasonably allocate resources and optimize system performance and efficiency. This technical solution makes timely adjustments based on model requirements and traffic changes to ensure that the system is in the best state and improve the execution speed and effect of deep learning and artificial intelligence applications.

Embodiment 2

The electronic device embodiment of the present invention is described below, and the electronic device can be regarded as a specific physical implementation of the method and device embodiments of the present invention described above. The details described in the electronic device embodiment of the present invention should be regarded as a supplement to the above method or device embodiments; details not disclosed in the electronic device embodiment of the present invention can be implemented with reference to the above method or device embodiments.

Figure 4 is a structural diagram of an electronic device of an embodiment of the present invention, which includes a processor and a memory, wherein the memory is used to store a computer executable program. When the computer program is executed by the processor, the processor executes a GPU service tidal deployment method based on K8s in Example 1.

As shown in FIG. 4 , the electronic device is presented in the form of a general-purpose computing device. The processor may be one or more and work in coordination. The present invention does not exclude distributed processing, that is, the processor may be dispersed in different physical devices. The electronic device of the present invention is not limited to a single entity, but may also be the sum of multiple physical devices.

The memory stores a computer executable program, which is usually a machine-readable code. The computer-readable program can be executed by the processor to enable the electronic device to perform the method of the present invention, or at least part of the steps in the method.

The memory includes a volatile memory, such as a random access memory unit (RAM) and/or a cache memory unit, and may also be a non-volatile memory, such as a read-only memory unit (ROM).

Optionally, in this embodiment, the electronic device further includes an I/O interface, which is used for the electronic device to exchange data with an external device. The I/O interface can represent one or more of several types of bus structures, including a storage unit bus or a storage unit controller, a peripheral bus, a graphics acceleration port, a processing unit, or a local bus using any of a variety of bus structures.

It should be understood that the electronic device shown in FIG. 4 is only an example of the present invention, and the electronic device of the present invention may also include elements or components not shown in the above examples. For example, some electronic devices also include display units such as display screens, and some electronic devices also include human-computer interaction elements such as buttons, keyboards, etc. As long as the electronic device can execute the computer-readable program in the memory to implement the method of the present invention or at least part of the steps of the method, it can be considered as an electronic device covered by the present invention.

FIG5 is a schematic diagram of a computer-readable recording medium of an embodiment of the present invention. As shown in FIG5 , a computer-readable recording medium stores a computer executable program, and when the computer executable program is executed, a GPU service tidal deployment method based on K8s of the first embodiment of the present invention is implemented. The computer-readable recording medium may include a data signal propagated in a baseband or as part of a carrier, which carries a readable program code. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. The readable recording medium may also be any readable medium other than a readable recording medium, which may send, propagate, or transmit a program for use by or in combination with an instruction execution system, device, or device. The program code contained on the readable recording medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination of the above.

Program code for performing the operations of the present invention may be written in any combination of one or more programming languages, including object-oriented programming languages such as Java, C++, etc., and conventional procedural programming languages such as "C" or similar programming languages. The program code may be executed entirely on the enterprise employee computing device, partially on the enterprise employee device, as a separate software package, partially on the enterprise employee computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device may be connected to the enterprise employee computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., via the Internet using an Internet service provider).

Through the above description of the implementation mode, it is easy for technical enterprise employees in this field to understand that the present invention can be implemented by hardware capable of executing a specific computer program, such as the system of the present invention, and the electronic processing unit, server, client, mobile phone, control unit, processor, etc. contained in the system. The present invention can also be implemented by computer software that executes the method of the present invention, such as control software executed by a microprocessor, an electronic control unit, a client, a server, etc. However, it should be noted that the computer software that executes the method of the present invention is not limited to being executed by one or a specific hardware entity, and it can also be implemented in a distributed manner by unspecified specific hardware. For computer software, the software product can be stored in a computer-readable recording medium (which can be a CD-ROM, a U disk, a mobile disk, etc.), or it can be distributed and stored on the network, as long as it enables the electronic device to execute the method according to the present invention.

The above embodiments are only used to illustrate the present invention and are not intended to limit the technical solutions described in the present invention. Although the present invention has been described in detail with reference to the above embodiments, the present invention is not limited to the above specific implementation methods. Therefore, any modification or equivalent replacement of the present invention; and all technical solutions and improvements thereof that do not depart from the spirit and scope of the invention are included in the scope of the claims of the present invention.

Claims (10)

1. A GPU service tidal deployment method based on K8s, characterized by comprising: Deploy the master node tide service tide-master, the worker node tide service tide-worker, and the gateway tide service tide-gateway based on the K8s platform; Define the running information of the GPU service instance through the custom resource definition CRD; The master control node tide service tide-master monitors the custom resource definition CRD and GPU resource information, and initializes all tide-worker nodes of the starting working node tide service tide-worker according to the GPU resource information; The master control node tide service tide-master dynamically adjusts the status and traffic allocation strategy of the working node tide service tide-worker and the gateway tide service tide-gateway according to the actual GPU resource situation, the GPU usage of the model in each tide-worker node and the traffic situation of the gateway tide service tide-gateway. 2. According to a K8s-based GPU service tide deployment method according to claim 1, it is characterized in that the master control node tide service tide-master monitors the custom resource definition CRD and GPU resource information, and initializes all tide-worker nodes of the starting working node tide service tide-worker according to the GPU resource information, including: Start the master node tide service tide-master, the worker node tide service tide-worker and the gateway tide service tide-gateway; The master control node tide service tide-master listens to the custom resource definition CRD and GPU resource information, and starts all tide-worker nodes of the working node tide service tide-worker according to the GPU resource information. After the tide-worker node is started, it will asynchronously load all models in the CRD; The working node tide service tide-worker polls the master control node tide service tide-master to obtain the model information or training tasks that need to be run; The gateway tide service tide-gateway monitors the master control node tide service tide-master to obtain the Pod IP corresponding to the model and the corresponding relationship information of the model. 3. According to a K8s-based GPU service tide deployment method described in claim 1 or 2, it is characterized in that the master control node tide service tide-master dynamically adjusts the status and traffic allocation strategy of the working node tide service tide-worker and the gateway tide service tide-gateway according to the actual resource situation, the GPU usage of the model in each tide-worker node and the traffic situation of the gateway tide service tide-gateway, including: The master control node tide service tide-master regularly pulls the traffic information of the gateway tide service tide-gateway and the error rate and GPU usage rate of the model under the working node tide service tide-worker; The master control node tide service tide-master determines whether it is necessary to adjust the association relationship between the tide-worker node and the model in the working node tide service tide-worker, or the number of Pods of the gateway tide service tide-gateway according to the threshold. 4. According to a K8s-based GPU service tide deployment method according to claim 3, it is characterized in that the master control node tide service tide-master determines whether it is necessary to adjust the association relationship between the tide-worker node and the model in the working node tide service tide-worker, or the number of Pods of the gateway tide service tide-gateway according to the threshold value, including: If the traffic that the master node tide service tide-master regularly pulls from the gateway tide service tide-gateway is greater than the preset traffic threshold; Alternatively, the error rate of the model under the working node tide service tide-worker periodically pulled by the master control node tide service tide-master is greater than the preset error rate; Alternatively, the GPU usage rate of the model under the working node tide service tide-worker that is periodically pulled by the master control node tide service tide-master is greater than the preset GPU usage rate; The master node tide service tide-master determines that it is necessary to adjust the association between the tide-worker node and the model in the working node tide service tide-worker, or the number of Pods in the gateway tide service tide-gateway. 5. According to a K8s-based GPU service tidal deployment method according to claim 4, it is characterized by comprising: The working node tide service tide-worker polls the master control node tide service tide-master to obtain model information or training tasks; When it is perceived that the association between the tide-worker node and the model in the working node tide service tide-worker is adjusted, the working node tide service tide-worker closes the originally running model and switches to the adjusted model. 6. According to a K8s-based GPU service tidal deployment method according to claim 4, it is characterized by comprising: The gateway tide service tide-gateway monitors the association between the tide-worker node and the model in the master node tide service tide-master, and is responsible for forwarding the traffic request of the corresponding model to the Pod of the corresponding working node tide service tide-worker; After the gateway tide service tide-gateway detects that the association relationship information between the tide-worker node and the model has changed, it stops distributing the traffic of the model to the removed tide-worker node. 7. According to a K8s-based GPU service tidal deployment method according to claim 1, it is characterized in that the operation information of the GPU service instance defined by the custom resource definition CRD includes: the model operation required information and routing information of the GPU service instance defined by the custom resource definition CRD. 8. A GPU service tidal deployment device based on K8s, characterized by comprising: The master node tide module deploys the master node tide service tide-master based on the K8s platform; The worker node tide module deploys the worker node tide service tide-worker based on the K8s platform; The gateway tide module deploys the gateway tide service tide-gateway based on the K8s platform; Custom resource definition module CRD, which defines the running information and routing information of the GPU service instance; The master node tide module monitors the custom resource definition module CRD and GPU resource information, and starts all tide-worker nodes of the working node tide service tide-worker according to the GPU resource information during initialization; The master node tide module dynamically adjusts the status and traffic allocation strategy of the working node tide service tide-worker and the gateway tide service tide-gateway according to the actual GPU resource situation, the GPU usage of the model in each tide-worker node and the traffic situation of the gateway tide service tide-gateway. 9. An electronic device, comprising a processor and a memory, wherein the memory is used to store a computer executable program, and wherein when the computer program is executed by the processor, the processor executes a K8s-based GPU service tidal deployment method as described in any one of claims 1 to 7. 10. A computer-readable recording medium storing a computer executable program, wherein when the computer executable program is executed, a GPU service tidal deployment method based on K8s is implemented as described in any one of claims 1 to 7.