CN118642844A - Dynamic and smooth expansion of POD resources based on K8S and rescheduling method under heavy load - Google Patents

Abstract

The present invention provides a K8S-based POD resource dynamic smooth expansion and heavy load rescheduling method, the method comprising: pre-deploying a business microservice POD and an application management program on a K8S cluster; pre-configuring an original memory threshold value, a memory expansion threshold value, a memory maximum value threshold value, an original CPU threshold value, a CPU expansion threshold value and a CPU core maximum value threshold value; using an application management program to monitor the real-time memory occupancy and the real-time CPU core number of the business microservice POD; judging whether to update the original memory threshold value based on the memory expansion threshold value, the real-time memory occupancy, the original memory threshold value and the memory maximum value threshold value corresponding to the business microservice POD; judging whether to update the original CPU threshold value based on the CPU expansion threshold value, the real-time CPU core number, the original CPU threshold value and the CPU core maximum value threshold value corresponding to the business microservice POD; therefore, the present invention can quickly and accurately judge the timing of dynamic and smooth expansion of POD resources, and automatically perform dynamic and smooth expansion.

Description

Dynamic and smooth expansion of POD resources based on K8S and rescheduling method under heavy load

Technical Field

The present invention relates to the technical field of container cloud platform, and in particular to a K8S-based POD resource dynamic smooth expansion and heavy load rescheduling method.

Background Art

In the K8S (short for Kubernetes) cluster environment, POD is the smallest deployment unit, which can contain one or more containers. In the deployment and management of the railway integrated video surveillance system, the K8S cluster can be used to automatically deploy and expand the components of the monitoring system to improve the reliability and scalability of the system. At the same time, the data storage and persistence functions of the K8S cluster can also provide a stable data storage and backup solution for the railway integrated video surveillance system. However, some inherent internal mechanisms of the K8S cluster make the existing K8S cluster unable to fully meet the business needs of the railway integrated video surveillance system. The main problems are:

(1) The CPU and memory resources of the POD in the K8S cluster can only be statically defined in its yaml configuration file based on experience. Once set, regardless of whether the resources are sufficient, they cannot be changed unless they are manually modified. Therefore, the existing K8S cluster cannot be dynamically adjusted according to the actual resource usage and changes of the POD, and it is easy for the business POD to freeze or even fail to operate normally due to inaccurate and unreasonable resource definitions;

However, the railway integrated video surveillance system needs to continuously analyze and process a large number of high-definition video images. Under normal circumstances, the resource requirements are already very high. If there are situations where multiple parties need to centrally monitor and adjust the system, such as emergency command, post-event analysis, hot spot backtracking, construction and maintenance, the required resources will surge instantly and fluctuate greatly. This requires automatic monitoring and judgment of the business POD resources in K8S to achieve dynamic expansion on demand. Otherwise, the video service will respond slowly or even be interrupted.

(2) In previous versions of K8S, even if static capacity expansion was performed, the newly added resources could not be used immediately and had to be restarted before the business Pod could take effect. This would inevitably cause business interruptions (even multiple frequent interruptions during continuous capacity expansion);

However, the resource expansion of the railway integrated video surveillance system is often sudden, irregular, and irregular. There are no professional K8S and business on-site personnel, and it is impossible to perform correct, effective, and timely expansion and restart business PODs and other manual operations. Therefore, it is required that POD resource expansion must be automatic without human intervention and there is no need to restart the relevant PODs. Otherwise, resource expansion will be meaningless.

(3) The Node to which a POD is scheduled to run is calculated by the K8S scheduler component based on certain conditions and internal algorithms. This process is not controlled by humans. After the POD is scheduled, it cannot be rescheduled based on the real-time resource usage of the node without human participation in rescheduling or node failure. This will cause the resource utilization of each Node in the cluster to become very unbalanced, which will not only cause resource waste, but may also cause the resource utilization of some Nodes to soar, causing all services hosted on them to freeze or even become completely unresponsive;

To avoid the problems caused by this "first time, lifetime" model, the K8S platform that carries the railway integrated video surveillance system must not only realize the dynamic expansion of resources at the POD level, but also realize the automatic rescheduling function between higher-level Node physical nodes, so that POD can migrate from heavily loaded Node nodes to lightly loaded Node nodes, thereby balancing and optimizing the resource load of the entire cluster.

In order to solve the above problems, people have been seeking an ideal technical solution.

Summary of the invention

Based on this, it is necessary to provide a K8S-based POD resource dynamic and smooth expansion and heavy load rescheduling method to address the above technical problems, so that POD resources can be dynamically and smoothly expanded, and at the same time make the resource utilization of each node in the cluster more balanced, so as to achieve the purpose of improving business POD stability and cluster reliability.

In order to achieve the above-mentioned object, the first aspect of the present invention provides a method for dynamic and smooth expansion of POD resources based on K8S, which comprises: pre-deploying a business microservice POD and an application management program; wherein the business microservice POD is used to install video surveillance management software;

Pre-configure the original memory threshold value, memory expansion threshold value, memory maximum value threshold value, original CPU threshold value, CPU expansion threshold value and CPU core maximum value threshold value corresponding to the business microservice POD; wherein the original memory threshold value and the original CPU threshold value are both POD resource dynamic parameters;

During the operation of the business microservice Pod, the pre-deployed application management program is used to monitor the real-time memory usage and real-time CPU core number of the business microservice Pod;

Based on the memory expansion threshold, real-time memory usage, original memory threshold and maximum memory threshold corresponding to the business microservice POD, determine whether to update the original memory threshold;

If yes, the application management program obtains a new memory threshold value based on the preset step size I and the original memory threshold value, and updates the original memory threshold value based on the new memory threshold value; wherein the new memory threshold value>the original memory threshold value;

Based on the CPU expansion threshold, the real-time number of CPU cores, the original CPU threshold and the maximum value threshold of the number of CPU cores corresponding to the business microservice POD, determine whether to update the original CPU threshold;

If so, the application management program obtains a new CPU threshold value based on the preset step size II and the original CPU threshold value, and updates the original CPU threshold value based on the new CPU threshold value; wherein the new CPU threshold value>the original CPU threshold value.

In order to achieve the above-mentioned object, the second aspect of the present invention provides a rescheduling method for a POD under heavy load based on K8S, which comprises: setting a target physical node for a Node node where a business microservice POD is deployed, and pre-configuring a rescheduling threshold of the target physical node;

In the process of executing the K8S-based POD resource dynamic smooth expansion method as claimed in the claim, the target physical node is monitored by the application management program, and whether the target physical node is in a heavy load state is determined based on the rescheduling threshold;

If the target physical node is in a heavy load state, the business microservice POD on the Node node will be rescheduled to other available nodes in the K8S cluster.

In order to achieve the above-mentioned object, the third aspect of the present invention provides a K8S cluster, which includes N master nodes and M Node nodes, and a load proxy component Nginx is compiled and installed on the master node; wherein the configuration file of the load proxy component Nginx includes proxy information of the N Master nodes;

Pre-deploy a business microservice POD on the K8S cluster; wherein the business microservice POD is used to install the video surveillance management software;

An application management program is pre-installed on the K8S cluster; wherein the application management program is used to monitor the real-time memory usage and the real-time number of CPU cores of the business microservice POD;

Pre-configure the original memory threshold value, memory expansion threshold value, memory maximum value threshold value, original CPU threshold value, CPU expansion threshold value and CPU core maximum value threshold value corresponding to the business microservice POD; wherein the original memory threshold value and the original CPU threshold value are both POD resource dynamic parameters;

During the operation of the business microservice Pod, the real-time memory usage and the real-time number of CPU cores of the business microservice Pod are monitored using the pre-installed application management program;

Based on the memory expansion threshold, real-time memory usage, original memory threshold and maximum memory threshold corresponding to the business microservice POD, determine whether to update the original memory threshold;

If yes, the application management program obtains a new memory threshold value based on the preset step size I and the original memory threshold value, and updates the original memory threshold value based on the new memory threshold value; wherein the new memory threshold value>the original memory threshold value;

Based on the CPU expansion threshold, the real-time number of CPU cores, the original CPU threshold and the maximum value threshold of the number of CPU cores corresponding to the business microservice POD, determine whether to update the original CPU threshold;

If so, the application management program obtains a new CPU threshold value based on the preset step size II and the original CPU threshold value, and updates the original CPU threshold value based on the new CPU threshold value; wherein the new CPU threshold value>the original CPU threshold value.

The beneficial effects of the present invention are:

1) The present invention pre-installs an application management program for monitoring business microservice POD on the K8S cluster, sets the original memory threshold value and the original CPU threshold value as POD resource dynamic parameters, and judges the timing of dynamic and smooth expansion of POD resources based on the original memory threshold value, memory expansion threshold value, memory maximum value threshold, original CPU threshold value, CPU expansion threshold value, CPU core maximum value threshold, real-time memory usage and real-time CPU core number. The application management program dynamically and smoothly expands POD resources using a preset step size;

2) In the dynamic process of POD resource expansion, the business microservice POD deployed on the K8S cluster does not need to be restarted to ensure uninterrupted business;

3) The present invention pre-configures the rescheduling threshold of the target physical node, uses the application management program to monitor whether the target physical node is in a heavy load state, and rescheduling the business microservice POD to other available nodes in the K8S cluster when the target physical node is in a heavy load state, thereby achieving the purpose of rescheduling when the POD is heavily loaded;

At the same time, the resource utilization of each node in the cluster is more balanced, thereby improving the stability of the business POD and the reliability of the cluster.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG1 is a schematic diagram of a flow chart of a method for dynamically and smoothly expanding POD resources based on K8S of the present invention;

2 is a schematic diagram of a flow chart of a method for dynamic and smooth expansion of POD resources based on K8S in an embodiment of the present invention;

3 is a schematic diagram of a flow chart of a method for dynamic and smooth expansion of POD resources based on K8S in another embodiment of the present invention;

FIG4 is a flow chart of a rescheduling method for a POD under heavy load based on K8S of the present invention;

FIG5 is a schematic diagram of the structure of the K8S cluster of the present invention.

DETAILED DESCRIPTION

The technical solution of the present invention is further described in detail below through specific implementation methods.

For ease of understanding, the interactive parties and/or terms and/or custom words involved in the present invention are first described in conjunction with the technical solution of the present invention:

K8S cluster: short for Kubernetes, is an open source container orchestration system used to automatically deploy, scale and manage containerized applications. It includes Master nodes and Node nodes. The Master node is responsible for the management and scheduling of the entire cluster, while the Node node is responsible for running containerized applications.

For example, in a production environment, four servers are used to build a K8S cluster consisting of two Master nodes and four Node nodes. The business microservice POD is deployed on the Node node of the K8S cluster; the K8S version must be 1.27 or above, and the four Node nodes use Containerd as the container runtime. Three nodes in the cluster are reused to build an Etcd cluster, as shown in Figure 5.

Application microservice POD: POD is the smallest deployment unit in Kubernetes. Different application microservice PODs implement different business functions. For example, in the comprehensive video surveillance business system, the application microservice POD is used to install the railway comprehensive video surveillance management software to implement video surveillance business.

Application management program: refers to a self-developed management program that periodically calls the K8S API interface to obtain the CPU and memory performance data of the business microservice POD and Node nodes, and performs resource calculation, threshold comparison, POD expansion and other operations. It monitors the real-time memory usage and real-time CPU core number of the business microservice POD, the real-time memory usage and real-time CPU core number of the physical node, etc., to dynamically and smoothly expand the POD resources or reschedule the POD under heavy load.

Original memory threshold and original CPU threshold: Both are dynamic parameters of POD resources. The application management program determines whether the original memory threshold needs to be adjusted dynamically based on the real-time memory usage, original memory threshold, and memory expansion threshold. It also determines whether the original CPU threshold needs to be adjusted dynamically based on the real-time number of CPU cores, original CPU threshold, and CPU expansion threshold.

It can be understood that when deploying a business microservice POD, the initial resource values are pre-defined in its yaml configuration file, including the memory request (initial minimum memory usage) and memory limit (original memory threshold value, i.e. initial maximum memory usage) that the business microservice POD can use at runtime, and the CPU request (initial minimum number of CPU cores) and CPU limit (original CPU threshold value, i.e. initial maximum number of CPU cores) that the business microservice POD can use at runtime.

Memory expansion threshold and CPU expansion threshold: As the expansion threshold that triggers the dynamic expansion of POD resources, it can be adjusted dynamically; the memory expansion threshold and CPU expansion threshold can be a percentage or a specific value.

Preset step size I refers to the pre-configured memory resource expansion step size, which is preset and can be adjusted according to actual conditions.

Preset step size II: refers to the pre-configured CPU resource expansion step size, which is also preset and can be adjusted according to actual conditions.

Maximum memory limit and maximum CPU core number limit: In order not to affect the normal operation of other microservices and physical nodes on the node, a business microservice Pod is not allowed to overuse resources. A maximum resource limit will be set for all business microservice Pods; therefore, Pod resources cannot be expanded indefinitely;

For example, in the production instance, the maximum memory threshold corresponding to the business microservice POD is set to 9000 millicore, and the maximum CPU core number threshold is 9000M; the CPU dynamic adjustment step is 1000 millicore, which means that the CPU threshold value will increase by 1000 millicore after each dynamic adjustment, and the memory dynamic adjustment step is 1000M, which means that the memory threshold value will increase by 1000M after each dynamic adjustment; therefore, the original memory threshold value of the business microservice POD can be adjusted to 10000 millicore at most, and the original CPU threshold value can be adjusted to 10000M at most;

As shown in Figure 4, after the application management program detects that the original memory threshold value corresponding to the business microservice POD is greater than the maximum memory threshold value, or the original CPU threshold value is greater than the maximum CPU core number threshold, the application management program no longer performs the corresponding capacity expansion operation and waits for human intervention.

Rescheduling threshold of physical node: pre-configured threshold value that triggers POD rescheduling, including memory rescheduling threshold and CPU rescheduling threshold of target physical node. Memory rescheduling threshold and CPU rescheduling threshold can be pre-set as percentage or in other ways (such as specific values).

After the business microservice POD is deployed, the application management program will obtain the real-time memory usage and real-time CPU core number of the physical node where the business microservice POD is deployed through the API interface of the K8S cluster at a certain period (usually a few seconds) to detect whether the physical node is in a heavy load state; if it is in a heavy load state, it means that the current load of the physical node is high, and the application management program will evict the POD on the node. The evicted POD will be rescheduled to other nodes after calculation by the K8Skube-scheduler component; if it is not in a heavy load state, it means that the current physical node load is relatively idle and no rescheduling is required;

Specific scheduling strategies include but are not limited to resource priority, data locality, node affinity, anti-affinity, taint, tolerance, etc.

Example 1

As shown in FIG. 1 , this embodiment provides a specific implementation of a method for dynamically and smoothly expanding POD resources based on K8S, and the method includes:

Pre-deploy the business microservice POD on the Node node of the K8S cluster; wherein the business microservice POD is used to install the video surveillance management software;

Pre-deploy business microservice POD and application management program; wherein the business microservice POD is used to install video surveillance management software;

Pre-configure the original memory threshold value, memory expansion threshold value, memory maximum value threshold value, original CPU threshold value, CPU expansion threshold value and CPU core maximum value threshold value corresponding to the business microservice POD; wherein the original memory threshold value and the original CPU threshold value are both POD resource dynamic parameters;

During the operation of the business microservice Pod, the pre-deployed application management program is used to monitor the real-time memory usage and real-time CPU core number of the business microservice Pod;

Based on the memory expansion threshold, real-time memory usage, original memory threshold and maximum memory threshold corresponding to the business microservice POD, determine whether to update the original memory threshold;

If so, the application management program obtains a new memory threshold value based on the preset step length I and the original memory threshold value, and updates the original memory threshold value based on the new memory threshold value; wherein the new memory threshold value>the original memory threshold value, such as the new memory threshold value=the original memory threshold value+preset step length I;

Based on the CPU expansion threshold, the real-time number of CPU cores, the original CPU threshold and the maximum value threshold of the number of CPU cores corresponding to the business microservice POD, determine whether to update the original CPU threshold;

If so, the application management program obtains a new CPU threshold value based on the preset step size II and the original CPU threshold value, and updates the original CPU threshold value based on the new CPU threshold value; wherein, the new CPU threshold value > the original CPU threshold value, such as the new CPU threshold value = the original CPU threshold value + the preset step size II.

It should be noted that the specific value of the memory threshold value in the configuration file is the original memory threshold value, and the original memory threshold value and the new memory threshold value are relative. When the memory threshold value is updated, the specific value of the memory threshold value in the configuration file is changed from the original memory threshold value to the new memory threshold value; when the next dynamic adjustment judgment is made, the memory threshold value in the configuration file is used as the basis for judgment;

The specific value of the CPU threshold value in the configuration file is the original CPU threshold value. The original CPU threshold value and the new CPU threshold value are also relative. When the CPU threshold value is updated, the specific value of the CPU threshold value in the configuration file is changed from the original CPU threshold value to the new CPU threshold value. When the next dynamic adjustment is made, the CPU threshold value in the configuration file is used as the basis for judgment.

It should also be noted that this embodiment pre-installs an application management program for monitoring the business microservice POD on the K8S cluster, sets the original memory threshold value and the original CPU threshold value as POD resource dynamic parameters, judges the timing of dynamic and smooth expansion of POD resources based on the memory expansion threshold and the CPU expansion threshold, and automatically dynamically and smoothly expands POD resources using a preset step size through the application management program.

It should also be noted that this embodiment automatically determines whether to update the original memory threshold value based on the comparison result between the original memory threshold value and the maximum memory value threshold, and the relationship between the real-time memory usage and the memory expansion threshold; and automatically determines whether to update the original CPU threshold value based on the comparison result between the original CPU threshold value and the maximum CPU core number threshold, and the relationship between the real-time CPU core number and the CPU expansion threshold.

It should also be noted that in this embodiment, POD is expanded according to two resource types, memory and CPU, respectively, and the POD resource expansion is dynamic rather than static; dynamic expansion of POD resources refers to real-time calculation, judgment and automatic expansion based on the actual occupancy of POD memory and CPU monitored by the application management program in real time, while traditional POD resources can only be defined in static yaml files. After setting, no matter whether the resources are sufficient or not, they cannot be changed unless they are modified manually.

It should also be noted that this embodiment breaks through the limitations of static allocation of POD resources and scheduling only once in traditional K8S, making resource allocation and utilization more flexible and reasonable.

Example 2

Based on Example 1, this example provides two specific implementation methods.

In a specific implementation, as shown in FIG2 , the memory expansion threshold is a percentage of the original memory threshold value, and the CPU expansion threshold is a percentage of the original CPU threshold value;

When determining whether to update the original memory threshold value based on the memory expansion threshold, real-time memory usage, original memory threshold value, and maximum memory threshold value corresponding to the business microservice POD, execute:

Calculate the real-time memory percentage, where the real-time memory percentage = the real-time memory usage of the business microservice Pod/the original memory threshold value;

When the real-time memory percentage is greater than the memory expansion threshold and the original memory threshold value is less than or equal to the preset maximum memory threshold value, the application management program determines to update the original memory threshold value; otherwise, the application management program determines not to update the original memory threshold value;

When determining whether to update the original CPU threshold value based on the CPU expansion threshold value, the real-time number of CPU cores, the original CPU threshold value, and the maximum threshold value of the number of CPU cores corresponding to the business microservice POD, the following is executed:

Calculate the real-time CPU percentage, where the real-time CPU percentage = the number of real-time CPU cores of the business microservice POD/the original CPU threshold value;

When the real-time CPU percentage is greater than the CPU expansion threshold and the original CPU threshold value is less than or equal to the preset CPU core maximum value threshold, the application management program determines to update the original CPU threshold value; otherwise, the application management program determines not to update the original CPU threshold value.

For example, the memory expansion threshold is set to 60%. If the real-time memory percentage (real-time memory usage of the business microservice POD/original memory threshold) is greater than 60%, and the original memory threshold is less than or equal to the preset maximum memory threshold, the application management program will dynamically increase the original memory threshold of the POD, and dynamically adjust the original memory threshold to a new memory threshold with a preset step length I. The new memory threshold = the original memory threshold + the preset step length I;

For example, if the CPU expansion threshold is set to 60%, if the real-time CPU percentage (real-time number of CPU cores of the business microservice POD/original CPU threshold value) is greater than 60%, and the original CPU threshold value is ≤ the preset maximum value threshold of the number of CPU cores, the application management program will dynamically increase the original CPU threshold value of the POD, and dynamically adjust the original CPU threshold value to the new CPU threshold value with a preset step size II, and the new CPU threshold value = original CPU threshold value + preset step size II.

In another specific implementation, as shown in FIG3 , the memory expansion threshold is in the same unit as the real-time memory usage, and the CPU expansion threshold is in the same unit as the real-time CPU core number;

When determining whether to update the original memory threshold value based on the memory expansion threshold, real-time memory usage, original memory threshold value, and maximum memory threshold value corresponding to the business microservice POD, execute:

When the real-time memory usage of the business microservice POD is greater than the memory expansion threshold, and the original memory threshold value is less than or equal to the preset maximum memory threshold value, the application management program determines to update the original memory threshold value; otherwise, the application management program determines not to update the original memory threshold value;

When determining whether to update the original CPU threshold value based on the CPU expansion threshold value, the real-time number of CPU cores, the original CPU threshold value, and the maximum threshold value of the number of CPU cores corresponding to the business microservice POD, the following is executed:

When the real-time number of CPU cores of the business microservice POD is greater than the CPU expansion threshold, and the original CPU threshold value is less than or equal to the preset maximum value threshold of the number of CPU cores, the application management program determines to update the original CPU threshold value; otherwise, the application management program determines not to update the original CPU threshold value.

For example, the memory expansion threshold is set to 60% of the original memory threshold. If the real-time memory usage of the business microservice POD is greater than the memory expansion threshold, and the original memory threshold is less than or equal to the preset maximum memory threshold, the application management program will dynamically increase the original memory threshold of the POD, and dynamically adjust the original memory threshold to a new memory threshold with a preset step size I. The new memory threshold = the original memory threshold + the preset step size I;

For example, if the CPU expansion threshold is set to 60% of the original CPU threshold value, if the real-time number of CPU cores of the business microservice POD is greater than the CPU expansion threshold, and the original CPU threshold value is ≤ the preset maximum value threshold of the number of CPU cores, the application management program will dynamically increase the original CPU threshold value of the POD, and dynamically adjust the original CPU threshold value to the new CPU threshold value with a preset step size II, and the new CPU threshold value = the original CPU threshold value + the preset step size II.

It should be noted that in the specific implementation shown in FIG. 3, the memory expansion threshold is a dynamic parameter, which changes dynamically following the original memory threshold value; the CPU expansion threshold is also a dynamic parameter, which changes dynamically following the original CPU threshold value. After the original memory threshold value is updated, the memory expansion threshold changes automatically, such as the new memory expansion threshold is 60% of the new memory threshold value; after the original CPU threshold value is updated, the CPU expansion threshold also changes automatically.

It should also be noted that the above two specific implementation methods are parallel solutions, and in practical applications, only one can be used.

In some embodiments, the K8S-based POD resource dynamic and smooth expansion method further includes:

In the configuration files of the API (Application Programming Interface) server, the key component kubelet, and the default scheduler in the K8S cluster, in-place vertical scaling parameters are added respectively, and the in-place vertical scaling parameters are pre-configured to true to enable smooth expansion operations.

It should be noted that this embodiment enables smooth capacity expansion by adding the additional parameter InPlacePodVerticalScaling=true in the configuration files of the API server (kube-apiserver), the key component kubelet and the default scheduler (kube-scheduler) in the K8S cluster in advance. After the resource expansion is triggered, it is automatically carried out without human intervention. Therefore, the POD resource expansion in this embodiment is carried out smoothly, the POD does not need to be restarted during the expansion process, and the business is not interrupted.

Example 3

Based on the above embodiment, this embodiment provides a specific implementation method of a rescheduling method when a POD is heavily loaded based on K8S.

In some embodiments, the rescheduling method based on K8S when the POD is heavily loaded includes:

Set the target physical node for the Node node where the business microservice POD is deployed, and pre-configure the rescheduling threshold of the target physical node;

In the process of executing the K8S-based POD resource dynamic smooth expansion method in Example 1 or 2, the target physical node is monitored by the application management program, and whether the target physical node is in a heavy load state is determined based on the rescheduling threshold;

If the target physical node is in a heavy load state, the business microservice POD on the Node node will be rescheduled to other available nodes in the K8S cluster.

It should be noted that if the actual resource usage of the target physical node is abnormal, the application management program will evict the business microservice POD on the Node node to release the load of the target physical node to ensure the availability of the cluster;

It should also be noted that POD rescheduling is dynamic rather than static, and is performed automatically without human intervention.

In some embodiments, before rescheduling the business microservice POD on the Node node to other available nodes in the K8S cluster, it also includes:

Determine whether the node where the target business microservice POD is located is the same physical node as the node in the heavy load state. If so, first reschedule the target business microservice POD to other available nodes in the K8S cluster, as shown in Figure 4;

The target business microservice POD refers to the business microservice POD for which the original memory threshold value and/or the original CPU threshold value need to be updated;

If the node where the target business microservice POD is located is not the same physical node as the node in the heavy load state, the POD resource dynamic smooth expansion operation and the POD rescheduling operation are processed in parallel, as shown in Figure 4.

It should be noted that, as shown in FIG4, the relevant steps in Example 1 are executed when only the original memory threshold value and/or the original CPU threshold value need to be updated; and the POD operation on the physical node is executed when only the target physical node is in a heavy load state.

Example 4

Based on the above embodiments, this embodiment provides a specific implementation method of a K8S cluster;

In some embodiments, the K8S cluster includes N master nodes and M Node nodes, and the K nodes in the cluster are reused to deploy the Etcd distributed database. The load proxy component Nginx is compiled and installed on the master node; wherein the configuration file of the load proxy component Nginx includes the proxy information of the N Master nodes;

Pre-deploy a business microservice POD on the K8S cluster; wherein the business microservice POD is used to install the video surveillance management software;

An application management program is pre-installed on the K8S cluster; wherein the application management program is used to monitor the real-time memory usage and the real-time number of CPU cores of the business microservice POD;

Pre-configure the original memory threshold value, memory expansion threshold value, memory maximum value threshold value, original CPU threshold value, CPU expansion threshold value and CPU core maximum value threshold value corresponding to the business microservice POD; wherein the original memory threshold value and the original CPU threshold value are both POD resource dynamic parameters;

During the operation of the business microservice Pod, the real-time memory usage and the real-time number of CPU cores of the business microservice Pod are monitored using the pre-installed application management program;

Based on the memory expansion threshold, real-time memory usage, original memory threshold and maximum memory threshold corresponding to the business microservice POD, determine whether to update the original memory threshold;

If yes, the application management program obtains a new memory threshold value based on the preset step size I and the original memory threshold value, and updates the original memory threshold value based on the new memory threshold value; wherein the new memory threshold value>the original memory threshold value;

Based on the CPU expansion threshold, the real-time number of CPU cores, the original CPU threshold and the maximum value threshold of the number of CPU cores corresponding to the business microservice POD, determine whether to update the original CPU threshold;

If so, the application management program obtains a new CPU threshold value based on the preset step size II and the original CPU threshold value, and updates the original CPU threshold value based on the new CPU threshold value; wherein the new CPU threshold value>the original CPU threshold value.

It should be noted that the CPU and memory resources of the existing POD can only be statically defined in its yaml file based on experience, and cannot be dynamically adjusted according to the actual resource occupancy and changes of the POD. It is easy for the business to fail to operate normally due to inaccurate and unreasonable resource definitions; and the railway integrated video monitoring system needs to continuously analyze and process a large number of high-definition video images. Under normal circumstances, the resource requirements are already very high. For example, in the case of emergency command, post-analysis, hot spot backtracking, construction and maintenance, etc., the required resources will surge instantly and fluctuate greatly. This requires that the business POD resources in K8S can be automatically monitored and judged to achieve dynamic expansion on demand, otherwise it will cause the video business to respond slowly or even interrupted. To solve this problem, this embodiment provides a K8S cluster that can dynamically and smoothly expand POD resources to meet the above application requirements.

In some embodiments, in the configuration files of the API server, key components and default scheduler in the K8S cluster, in-place vertical scaling parameters are pre-added respectively, and the in-place vertical scaling parameters are pre-configured to true to enable smooth expansion operations.

It should be noted that in previous versions of K8S, even if static expansion is performed, the newly added resources cannot be used immediately, and the business POD must be restarted before it can take effect, which will inevitably cause business interruptions (even multiple frequent interruptions during continuous expansion); and the resource expansion of the comprehensive video surveillance system is often sudden, irregular, and irregular, and there are no professional K8S and business on-duty personnel on site, and it is impossible to perform correct, effective, and timely expansion and restart of business POD and other manual operations. Therefore, it is required that POD resource expansion must be performed automatically without human intervention, and there is no need to restart the relevant POD, otherwise the resource expansion will have no practical significance. To solve this problem, this embodiment enables smooth expansion operation. During the dynamic process of POD resource expansion, when modifying the original memory threshold value and the original CPU threshold value, the business microservice POD is not affected, the original application connection will not be interrupted, and the modified memory threshold value and CPU threshold value are effective in real time. Therefore, during the expansion process, the POD does not need to be restarted and the business is not interrupted.

Example 5

Based on the above embodiment, this embodiment provides another specific implementation method of K8S cluster;

In some embodiments, the K8S cluster also sets the Node node where the business microservice POD is deployed in the K8S cluster as the target physical node, and pre-configures the rescheduling threshold of the target physical node;

Monitoring the target physical node using the application management program, and determining whether the target physical node is in a heavy load state based on the rescheduling threshold;

If the target physical node is in a heavy load state, the business microservice POD on the Node node will be rescheduled to other available nodes in the K8S cluster.

It should be noted that other available nodes refer to other Node nodes except the target physical node, and the Node node is not currently in a heavy load state.

In some embodiments, before rescheduling the business microservice POD on the Node node to other available nodes in the K8S cluster, it also includes:

Determine whether the node where the target business microservice POD is located is the same physical node as the node in a heavy load state. If so, reschedule the target business microservice POD to other available nodes in the K8S cluster;

The target business microservice POD refers to a business microservice POD for which the original memory threshold value and/or the original CPU threshold value need to be updated.

In a specific implementation, when implementing dynamic and smooth expansion of POD resources and rescheduling under heavy load in a K8S cluster, the following steps are performed:

Step 1: Based on the four-node K8S cluster consisting of two Master/Node multiplexing nodes and two Nodes, deploy the Etcd distributed database on three nodes in the multiplexing cluster, and the working mode is cluster;

Configure IPv4 addresses on the four servers, change system parameters, disable server firewalls and SELINUX, close SWAP space, and change node host names;

Install and deploy the application management program on the Node node of the K8S cluster;

Step 2: The communication between K8S and Etcd uses the https protocol, which requires the issuance of a certificate. Custom certificates are used in the production environment. Use open source tools to generate a CA certificate, which is valid for 10 years. Use the generated CA certificate to issue Etcd certificates, K8S component kube-apiserver/kube-scheduler/kube-proxy/kubelet/kube-controller-manager and other related certificates.

Step 3. After the certificate configuration is completed, create an installation directory, transfer the generated certificate to other nodes, deploy the K8S cluster using the binary installation package, and edit the configuration file required by the K8S component to add the additional parameter InPlacePodVerticalScaling=true to enable the new feature of smooth expansion, such as the kubelet.conf file:

KUBELET_OPTS="--v=4 \

--container-runtime-endpoint=unix:///run/containerd/containerd.sock \

--feature-gates=InPlacePodVerticalScaling=true " // New parameters

Other component configuration files kube-proxy.conf/kube-scheduler.conf/

The configuration of kube-apiserver.conf is similar. After adding the new feature parameters, you also need to add related configuration contents such as node IP/port/certificate storage directory/log storage directory. After the configuration is completed, start the related services and check the cluster status.

Step 4: Deploy the network component Calico in the cluster to ensure cross-node communication between PODs.

Step 5: After the network components are deployed, deploy the application microservice POD. In the POD's YAML, you need to define the CPU and memory resources as follows:

limits:

memory: "100Mi"

cpu: "100m"

requests:

memory: "100Mi"

cpu: "100m"

Step 6. After the application is deployed, the cluster API interface is periodically called through the application management program to obtain the CPU and memory information of the pod and the CPU and memory information of the cluster nodes. For example, a similar API interface is available at https://X.X.X.X:9009/k8s/clusters/c-m-bd7ssk4c/apis/metrics.k8s.io/v1beta1/namespaces/default/pods/cms-bd89844c9-x492r. Calling this interface will return the information of the corresponding POD in JSON format. Filter out the corresponding fields to obtain the relevant information of CPU and memory.

Step 7: The obtained POD performance data is calculated with the threshold stored in the cluster. The threshold is stored in the cluster through ConfigMap. The threshold content of ConfigMap is defined as follows:

[App]

CpuDrainLimit=60

MemDrainLimit=60

Create the cluster using the kubectl create command;

When the result of the real-time memory usage of the business microservice POD/the original memory threshold is greater than 60%, and/or the result of the real-time number of CPU cores of the business microservice POD/the original CPU threshold is greater than 60%, it means that the current load of the POD is too high, and the capacity expansion operation will be performed according to the pre-set step length program. This operation is automatically completed by the application management program background. However, each POD is set with a maximum value. When the capacity expansion operation reaches the maximum value, the automatic capacity expansion operation stops and no more capacity expansion operations are performed;

Step 8: The application management program obtains the cluster node performance data through the cluster API interface and calculates the threshold stored in the cluster, such as the interface for obtaining Node information:

https://X.X.X.X:9009/k8s/clusters/c-m-jnt7pjdz/v1/metrics.k8s.io.nodes, obtain the node information by calling the interface and return it in json format. The program code filters the corresponding fields to get the CPU and memory information of the Node in advance. The threshold is stored in the cluster through ConfigMap. The threshold content of ConfigMap is defined as follows:

[App]

CpuDrainLimit=60

MemDrainLimit=60

Use the kubectl create command to create the cluster.

When the result of (real-time memory usage of the physical node where the business microservice POD is located / memory threshold of the physical node) is greater than 60%, and/or the result of (real-time number of CPU cores of the physical node where the business microservice POD is located / original CPU threshold of the physical node) is greater than 60%, it means that the physical node load is too high. The program will evict the POD on the node and automatically reschedule it to other physical nodes to ensure the stability and availability of the cluster.

It should be noted that the Node node to which a POD is scheduled to run is calculated by the scheduler component of K8S according to certain conditions and internal algorithms, and this process is not controlled by humans. After the POD scheduling is completed, it is impossible to reschedule according to the real-time resource usage of the node without human participation in rescheduling or node failure. This will cause the resource utilization rate of each Node node in the cluster to become very unbalanced, which will not only cause resource waste, but also may cause the resource utilization rate of some Nodes to soar, causing all services carried on it to freeze or even completely fail to respond. In order to avoid the problems caused by this "first time is lifelong" mode, this embodiment proposes a K8S platform that can carry a railway integrated video surveillance system, which can not only realize the dynamic expansion capability of resources at the POD level, but also realize the automatic rescheduling function between higher-level Node physical nodes, so that POD can be migrated from heavily loaded Node nodes to lightly loaded Node nodes, thereby balancing and optimizing the resource load of the entire cluster.

The technical features of the above embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, ordinary technicians in the field should understand that the specific implementation methods of the present invention can still be modified or some technical features can be replaced by equivalents without departing from the spirit of the technical solution of the present invention, which should be included in the scope of the technical solution for protection of the present invention.

Claims (10)

1. A method for dynamic and smooth expansion of POD resources based on K8S, characterized by comprising: Pre-deploy business microservice POD and application management program; wherein the business microservice POD is used to install video surveillance management software; Pre-configure the original memory threshold value, memory expansion threshold value, memory maximum value threshold value, original CPU threshold value, CPU expansion threshold value and CPU core maximum value threshold value corresponding to the business microservice POD; wherein the original memory threshold value and the original CPU threshold value are both POD resource dynamic parameters; During the operation of the business microservice Pod, the pre-deployed application management program is used to monitor the real-time memory usage and real-time CPU core number of the business microservice Pod; Based on the memory expansion threshold, real-time memory usage, original memory threshold and maximum memory threshold corresponding to the business microservice POD, determine whether to update the original memory threshold; If yes, the application management program obtains a new memory threshold value based on the preset step size I and the original memory threshold value, and updates the original memory threshold value based on the new memory threshold value; wherein the new memory threshold value>the original memory threshold value; Based on the CPU expansion threshold, the real-time number of CPU cores, the original CPU threshold and the maximum value threshold of the number of CPU cores corresponding to the business microservice POD, determine whether to update the original CPU threshold; If so, the application management program obtains a new CPU threshold value based on the preset step size II and the original CPU threshold value, and updates the original CPU threshold value based on the new CPU threshold value; wherein the new CPU threshold value>the original CPU threshold value. 2. According to the K8S-based POD resource dynamic and smooth expansion method of claim 1, it is characterized in that the memory expansion threshold is a percentage of the original memory threshold value, and the CPU expansion threshold is a percentage of the original CPU threshold value; When determining whether to update the original memory threshold value based on the memory expansion threshold, real-time memory usage, original memory threshold value, and maximum memory threshold value corresponding to the business microservice POD, execute: Calculate the real-time memory percentage, where the real-time memory percentage = the real-time memory usage of the business microservice Pod/the original memory threshold value; When the real-time memory percentage is greater than the memory expansion threshold and the original memory threshold value is less than or equal to the preset maximum memory threshold value, the application management program determines to update the original memory threshold value; otherwise, the application management program determines not to update the original memory threshold value; When determining whether to update the original CPU threshold value based on the CPU expansion threshold value, the real-time number of CPU cores, the original CPU threshold value, and the maximum threshold value of the number of CPU cores corresponding to the business microservice POD, the following is executed: Calculate the real-time CPU percentage, where the real-time CPU percentage = the number of real-time CPU cores of the business microservice POD/the original CPU threshold value; When the real-time CPU percentage is greater than the CPU expansion threshold and the original CPU threshold value is less than or equal to the preset CPU core maximum value threshold, the application management program determines to update the original CPU threshold value; otherwise, the application management program determines not to update the original CPU threshold value. 3. According to the K8S-based POD resource dynamic and smooth expansion method of claim 1, it is characterized in that the unit of the memory expansion threshold is consistent with the real-time memory usage, and the unit of the CPU expansion threshold is consistent with the real-time CPU core number; When determining whether to update the original memory threshold value based on the memory expansion threshold, real-time memory usage, original memory threshold value, and maximum memory threshold value corresponding to the business microservice POD, execute: When the real-time memory usage of the business microservice POD is greater than the memory expansion threshold, and the original memory threshold value is less than or equal to the preset maximum memory threshold value, the application management program determines to update the original memory threshold value; otherwise, the application management program determines not to update the original memory threshold value; When determining whether to update the original CPU threshold value based on the CPU expansion threshold value, the real-time number of CPU cores, the original CPU threshold value, and the maximum threshold value of the number of CPU cores corresponding to the business microservice POD, the following is executed: When the real-time number of CPU cores of the business microservice POD is greater than the CPU expansion threshold, and the original CPU threshold value is less than or equal to the preset maximum value threshold of the number of CPU cores, the application management program determines to update the original CPU threshold value; otherwise, the application management program determines not to update the original CPU threshold value. 4. The method for dynamic and smooth expansion of POD resources based on K8S according to any one of claims 1 to 3, characterized in that it also includes: In the configuration files of the API server, key components and default scheduler in the K8S cluster, in-place vertical scaling parameters are respectively added, and the in-place vertical scaling parameters are pre-configured to true to enable smooth expansion operations. 5. A rescheduling method for POD under heavy load based on K8S, characterized by comprising: Set the target physical node for the Node node where the business microservice POD is deployed, and pre-configure the rescheduling threshold of the target physical node; In the process of executing the K8S-based POD resource dynamic smooth expansion method according to any one of claims 1 to 4, the target physical node is monitored by the application management program, and whether the target physical node is in a heavy load state is determined based on the rescheduling threshold; If the target physical node is in a heavy load state, the business microservice POD on the Node node will be rescheduled to other available nodes in the K8S cluster. 6. The rescheduling method for POD under heavy load based on K8S according to claim 5 is characterized in that before rescheduling the business microservice POD on the Node node to other available nodes in the K8S cluster, it also includes: Determine whether the node where the target business microservice POD is located is the same physical node as the node in a heavy load state. If so, reschedule the target business microservice POD to other available nodes in the K8S cluster; The target business microservice POD refers to a business microservice POD for which the original memory threshold value and/or the original CPU threshold value need to be updated. 7. A K8S cluster, characterized in that: it comprises N master nodes and M Node nodes, and a load proxy component Nginx is compiled and installed on the master node; wherein the configuration file of the load proxy component Nginx includes proxy information of the N Master nodes; Pre-deploy a business microservice POD on the K8S cluster; wherein the business microservice POD is used to install the video surveillance management software; An application management program is pre-installed on the K8S cluster; wherein the application management program is used to monitor the real-time memory usage and the real-time number of CPU cores of the business microservice POD; Pre-configure the original memory threshold value, memory expansion threshold value, memory maximum value threshold value, original CPU threshold value, CPU expansion threshold value and CPU core maximum value threshold value corresponding to the business microservice POD; wherein the original memory threshold value and the original CPU threshold value are both POD resource dynamic parameters; During the operation of the business microservice Pod, the real-time memory usage and the real-time number of CPU cores of the business microservice Pod are monitored using the pre-installed application management program; Based on the memory expansion threshold, real-time memory usage, original memory threshold and maximum memory threshold corresponding to the business microservice POD, determine whether to update the original memory threshold; If yes, the application management program obtains a new memory threshold value based on the preset step size I and the original memory threshold value, and updates the original memory threshold value based on the new memory threshold value; wherein the new memory threshold value>the original memory threshold value; Based on the CPU expansion threshold, the real-time number of CPU cores, the original CPU threshold and the maximum value threshold of the number of CPU cores corresponding to the business microservice POD, determine whether to update the original CPU threshold; If so, the application management program obtains a new CPU threshold value based on the preset step size II and the original CPU threshold value, and updates the original CPU threshold value based on the new CPU threshold value; wherein the new CPU threshold value>the original CPU threshold value. 8. The K8S cluster according to claim 7 is characterized in that in the configuration files of the API server, key components and default scheduler in the K8S cluster, in-place vertical scaling parameters are pre-added respectively, and the in-place vertical scaling parameters are pre-configured to true to enable smooth expansion operations. 9. The K8S cluster according to claim 7 is characterized in that the Node node where the business microservice POD is deployed in the K8S cluster is set as a target physical node, and the rescheduling threshold of the target physical node is preconfigured; Monitoring the target physical node using the application management program, and determining whether the target physical node is in a heavy load state based on the rescheduling threshold; If the target physical node is in a heavy load state, the business microservice POD on the Node node will be rescheduled to other available nodes in the K8S cluster. 10. The K8S cluster according to claim 9, characterized in that before rescheduling the business microservice POD on the Node node to other available nodes in the K8S cluster, it also includes: Determine whether the node where the target business microservice POD is located is the same physical node as the node in a heavy load state. If so, reschedule the target business microservice POD to other available nodes in the K8S cluster; The target business microservice POD refers to a business microservice POD for which the original memory threshold value and/or the original CPU threshold value need to be updated.