CN114064225A - Self-adaptive scheduling method, device, computer storage medium and system - Google Patents

Abstract

The embodiment of the application provides a self-adaptive scheduling method, a self-adaptive scheduling device, a computer storage medium and a self-adaptive scheduling system, wherein the method comprises the steps of acquiring a task to be processed in a task list; sequentially judging whether a plurality of task executors included in the integrated module are in an idle state or not according to the tasks to be processed; when the plurality of task executors are all in a non-idle state, the number of the task executors in the integrated module is expanded; scheduling a newly added task executor from the expanded integrated module, and controlling the newly added task executor to execute the task to be processed; in this way, the integration module can adaptively adjust the number of task executors according to the service scale in the task list, thereby providing a dynamic telescopic continuous integration environment and improving the task concurrent processing capacity; the defect that the number of task processors must be kept fixed after the continuous integrated system is started is avoided, and the problem of solidification of continuous integrated processing capacity is solved.

Description

An adaptive scheduling method, device, computer storage medium and system

technical field

The present application relates to the technical field of computer software development, and in particular, to an adaptive scheduling method, device, computer storage medium and system.

Background technique

Jenkins is an open source continuous integration system. Through this platform, you can compile, test and release your own projects and codes on a large scale, which brings great convenience to the software development team. Here, Jenkins provides an open and easy-to-use software platform that makes continuous software integration possible.

In actual use, the Jenkins system generally adopts the master-slave mode to realize the integrated environment. In the master-slave mode, the master node is the control node of the Jenkins system, and the slave node is the task node of the Jenkins system to perform specific tasks. Equivalent to Jenkins' task executor. Among them, the slave node can contain multiple slaves to achieve the function of parallel processing tasks, and the number of slaves can be increased or decreased according to requirements. Although this structure can achieve a certain degree of parallel processing, it has the following disadvantages: continuous integration environment Relatively fixed, in the face of different continuous integration task requirements, individual configuration of each node is required. Therefore, there is still a need for an adaptive and scalable continuous integration environment to cope with dynamic business scales.

SUMMARY OF THE INVENTION

To solve the above technical problems, the embodiments of the present application provide an adaptive scheduling method, device, computer storage medium and system, which can adaptively adjust the number of task executors according to the business scale, thereby providing a dynamically scalable continuous integration environment.

The technical solution of the present application is realized as follows:

In a first aspect, an embodiment of the present application provides an adaptive scheduling method, and the method includes:

Get the pending tasks in the task list;

According to the to-be-processed task, sequentially determine whether the multiple task executors included in the integrated module are in an idle state;

When the multiple task executors are all in a non-idle state, expanding the number of task executors in the integrated module;

The newly added task executor is scheduled from the expanded integration module, and the newly added task executor is controlled to execute the to-be-processed task.

In a second aspect, an embodiment of the present application provides an adaptive scheduling device, where the adaptive scheduling device includes an acquisition unit, a judgment unit, an extension unit, and a scheduling unit; wherein,

An acquisition unit, configured to acquire pending tasks in the task list;

a judging unit, configured to sequentially judge whether a plurality of task executors included in the integrated module are in an idle state according to the tasks to be processed;

an expansion unit, configured to expand the number of task executors in the integrated module when the multiple task executors are all in a non-idle state;

The scheduling unit is configured to schedule the newly added task executor from the expanded integrated module, and control the newly added task executor to execute the to-be-processed task.

In a third aspect, an embodiment of the present application provides an adaptive scheduling apparatus, where the adaptive scheduling apparatus includes a memory and a processor; wherein,

the memory for storing a computer program executable on the processor;

The processor is configured to execute the steps of the method according to the first aspect when running the computer program.

In a fourth aspect, an embodiment of the present application provides a computer storage medium, where the computer storage medium stores an adaptive scheduler, and when the adaptive scheduler is executed by at least one processor, implements the steps of the method according to the first aspect .

In a fifth aspect, an embodiment of the present application provides a system, where the system at least includes the adaptive scheduling apparatus described in the second aspect or the third aspect.

Embodiments of the present application provide an adaptive scheduling method, device, computer storage medium, and system. The method includes acquiring tasks to be processed in a task list; and according to the tasks to be processed, sequentially judging multiple Whether the task executor is in an idle state; when the multiple task executors are all in a non-idle state, expand the number of task executors in the integrated module; schedule the newly added task executor from the expanded integrated module, and control the The newly added task executor executes the to-be-processed task; in this way, the integration module can adaptively adjust the number of task executors according to the business scale in the task list, thereby providing a dynamically scalable continuous integration environment, making The concurrent processing capability of tasks is improved; it avoids the disadvantage that a fixed number of task processors must be maintained after the continuous integration system is started, and solves the problem of curing the processing capability of continuous integration; avoids adding additional cluster management tools to manage multiple task executors, which simplifies the The cost of the connection process and connection management.

Description of drawings

1 is a schematic structural diagram of an integrated system in a master-slave mode provided in a related technical solution;

2 is a schematic structural diagram of an integrated system in another master-slave mode provided in the related technical solution;

FIG. 3 is a schematic flowchart of an adaptive scheduling method provided by an embodiment of the present application;

4 is a schematic structural diagram of an integrated system provided by an embodiment of the present application;

FIG. 5 is a schematic flowchart of another adaptive scheduling method provided by an embodiment of the present application;

FIG. 6 is a schematic flowchart of another adaptive scheduling method provided by an embodiment of the present application;

FIG. 7 is a schematic flowchart of still another adaptive scheduling method provided by an embodiment of the present application;

FIG. 8 is a schematic diagram of the composition and structure of an adaptive scheduling apparatus provided by an embodiment of the present application;

FIG. 9 is a schematic diagram of the composition and structure of another adaptive scheduling apparatus provided by an embodiment of the present application;

FIG. 10 is an example of a specific hardware structure of an adaptive scheduling provided by an embodiment of the present application;

FIG. 11 is a schematic structural diagram of a system provided by an embodiment of the present application.

Detailed ways

The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present application.

Jenkins is an open source continuous integration system. Through this platform, projects and codes can be compiled, tested and released on a large scale, bringing great convenience to software development teams. Jenkins provides an open and easy-to-use software platform that makes continuous software integration possible.

In the development process of the Jenkins platform, a combination of master nodes and slave nodes is generally used to complete the corresponding tasks. Among them, the master node is mainly responsible for control work, such as managing slave nodes and assigning tasks to slave nodes. In actual use, in the face of large-scale continuous integration requirements, the continuous integration structure of a single master node or a single master node with a single slave node (that is, a master only corresponds to a single slave) is difficult to meet the needs of the business. , so usually the structure of continuous integration environment is distributed and clustered.

Referring to FIG. 1, it shows a schematic structural diagram of an integrated system in a master-slave mode provided in a related technical solution. As shown in FIG. 1, when the master-slave mode is used to implement the integrated environment, the master node is the Jenkins system. The control node, the slave node is the task node of the Jenkins system, the number of slaves in the slave node can be increased or decreased according to the demand; in addition, some developers choose to use Docker instead of the slave node, see Figure 2, which shows related technologies Another schematic diagram of the structure of the integrated system in master-slave mode provided in the scheme, as shown in Figure 2, the master node is still used as the control node, the slave node is replaced with a Docker container, and the Docker container is connected to the master through jnlp or ssh The connection can achieve stable environment isolation for the parallel execution of different continuous integration tasks, and these integration systems have the following two disadvantages:

(1) The continuous integration environment is relatively fixed; for a fixed integration system, after the configuration is completed and started, the number of slaves cannot be increased or decreased during the operation process; therefore, in the face of different continuous integration task requirements, each node is required individual configuration;

(2) Once the slave node of this structure is faulty or wrong, it needs to be searched one by one and repaired manually, which wastes a lot of time.

Docker is an open source container application engine that allows developers to package applications and dependencies into a portable image, which can then be distributed to any popular environment, and can also be virtualized. Docker containers are completely sandboxed and do not have any interface with each other. The application scenarios of Docker mainly include the following four categories: (1) automated packaging and release of web applications; (2) automated testing and continuous integration and release; (3) deployment and adjustment of databases or other background applications in a service-based environment; (4) Compile or extend the existing OpenShift or OpenShift platform from scratch to build a separate platform service environment, where both OpenShift and OpenShift are open platforms for open source developers.

Based on this, an embodiment of the present application provides an adaptive scheduling method, by acquiring the tasks to be processed in the task list; according to the tasks to be processed, it is sequentially judged whether multiple task executors included in the integration module are in an idle state ; When the multiple task executors are in a non-idle state, expand the number of task executors in the integrated module; schedule the newly added task executor from the expanded integrated module, and control the execution of the newly added task In this way, the number of task executors in the integration module can be adaptively adjusted according to the business scale in the task list, thereby providing a dynamically scalable continuous integration environment and improving the concurrent processing capability of tasks ;Avoid the disadvantage that a fixed number of task processors must be maintained after the continuous integration system is started, and solve the curing problem of continuous integration processing capacity; Avoid adding additional cluster management tools to manage multiple task executors, simplifying the connection process and connection management. cost.

In an embodiment of the present application, referring to FIG. 3 , it shows a schematic flowchart of an adaptive scheduling method provided by an embodiment of the present application. As shown in FIG. 3 , the method may include:

S101: Obtain pending tasks in the task list;

It should be noted that, in order to meet large-scale business requirements, the structure of continuous integration system is distributed and clustered. In the continuous integration system in the master-slave mode, there are master nodes for completing control tasks and slave nodes for completing specific services. In this embodiment, each specific slave is containerized to obtain a task executor, and the entirety of these multiple task executors is containerized into an integrated module. That is to say, in this embodiment, the original separate slave is replaced by a task executor, but compared to the master node, the integrated module is directly connected and managed instead of specific multiple task executors, so , the control task of the master node does not become more complicated, but multiple task executors can complete the assigned tasks more efficiently through the adaptive scheduling method. At the same time, since multiple task executors are scheduled through the adaptive scheduling method described in this embodiment, there is no need to resort to complex cluster management tools, and it is not necessary to consider the combination of cluster management tools and master nodes, thus reducing the system workload. Complexity.

It should be noted that the continuous integration environment exists to complete the assigned tasks, and the information used to record these tasks forms the task list. For the task list, the tasks in it are in order. In general, the tasks in the task list are sorted according to the time of receipt, that is to say, the tasks received first are in the front of the task list, and the tasks received later are in the back of the task list. In the system of the mechanism, the tasks may be ordered according to the degree of urgency, which is also within the protection scope of the present application. Specifically, when the received tasks are written into the task list, the received tasks are assigned values, so as to identify and sort the tasks.

S102: According to the to-be-processed task, sequentially determine whether the multiple task executors included in the integrated module are in an idle state;

It should be noted that, according to the tasks to be processed, it is necessary to sequentially determine whether the multiple task executors included in the integrated module are in an idle state; here, "sequentially" means in a certain order, and this order is a preset setting The most common one is to follow the number of the task executor from small to large, and the number is generally generated when the task executor is created.

It should also be noted that here, the connotation of "sequentially" also includes that after judging each task executor, it is necessary to check whether the judgment result meets the conditions of the next step, and if so, proceed to the next step instead of the next step. A task executor continues to make judgments.

Further, in some embodiments, the task executor includes a Docker container.

It should be noted that Docker is an open source container application engine that allows developers to package applications and dependencies into a portable image, and then publish them to any popular environment, which can also be virtualized. Docker containers are completely sandboxed and do not have any interface with each other. The application scenarios of Docker mainly include the following four categories: (1) automated packaging and release of web applications; (2) automated testing and continuous integration and release; (3) deployment and adjustment of databases or other background applications in a service-based environment; (4) Compile or extend the existing OpenShift or Cloud Foundry platform from scratch to build a separate PaaS environment.

In this embodiment, Docker provides a sandbox environment for continuous integration, thereby avoiding manual configuration and repair of the environment. Docker also achieves stable environment isolation, and different continuous integration tasks can be implemented in different Docker containers. true parallelism.

It should be noted that, in this embodiment, the integration module includes multiple task executors, that is, includes multiple Docker containers, and each Docker container is used to execute multiple specific business requirements. Among them, the integration module is actually equivalent to the single-point slave node in the existing technology, so it can also be called the Salve Docker. The Salve Docker is distributed again to form multiple independent task executors, namely Docker0, Docker1...to Dockern, and these Docker0, Docker1... to Dockern are integrated through the adaptive scheduling method described in this embodiment.

Referring to FIG. 4, it shows a schematic structural diagram of an integrated system in the embodiment of the present application. As shown in FIG. 4, the adaptive scheduling system at least includes a master host (equivalent to a master node) and a Salve Docker (ie an integrated module) , in which the master node is implemented by the master host, that is, the ordinary physical host of Jenkins, or the master constructed by containerized Jenkins+Docker; the Salve Docker is directly connected to the master host, that is, the master host does not need to perform each task executor. Managed to simplify connection operations and reduce the cost of connection management.

In addition, Salve Docker generally monopolizes a physical host, which is also containerized by Docker. At the same time, SalveDocker includes multiple independent Docker containerized task executors, which are distributed to form the layout of Docker0, Docker1... to Dockern, and many A Docker containerized task executor is managed using the aforementioned integrated approach, eliminating the need for additional cluster management tools.

Further, in some embodiments, before step S101, the method may further include:

The integration module is initialized such that the integration module includes at least one task executor.

It should be noted that, for a continuous integration system, the amount of tasks it receives cannot be determined at the beginning of its establishment, so if more task executors are opened at the beginning of establishment, it will lead to less task requirements. Waste of processing power and additional consumption of hardware; however, if fewer task executors are turned on, the processing power will be low when the task demand is large, and the task demand cannot be completed in time. The adaptive scheduling method in this embodiment can ensure that the number of task executors can be adaptively increased or decreased relative to the task requirements during operation, that is to say, the number of task executors in the integrated module changes, so the initialization is initially performed. , you can only get an integrated module containing one task executor, and then according to the business scale, the number of task executors will change accordingly, which can not only reduce the waste of processing power but also increase the maximum number of concurrent processing tasks. Of course, when the integration module is initialized, other numbers of task executors, such as two or three, can also be obtained; at the same time, when the integration module is initialized, two parameters, query variables and the number of task executors, need to be set to facilitate the Subsequent management of task executors. Generally, the query variable is represented by i, and i also represents the serial number of the task executor in the integrated module. After initialization, the value of i is 0; the number of task executors is represented by n, and i is a positive value greater than or equal to 0 and less than n. Integer.

S103: when the multiple task executors are all in a non-idle state, expand the number of task executors in the integrated module;

It should be noted that if all the task executors obtained in the integrated module are in a non-idle state, it means that the current number of task executors is not enough to meet the existing task requirements, so the integration module needs to be expanded, that is, the need to expand The number of task executors. At the same time, since the Docker container is sandboxed and completely independent, creating a new Docker container is an easy operation.

Further, in some embodiments, step S103 may specifically include:

A new task executor is established in the integration module, and the number of task executors in the integration module is increased by one, so as to expand the number of task executors included in the integration module.

It should be noted that, after the newly added task executor is established in the integrated module, the number of task executors in the integrated module needs to be changed at the same time, otherwise the newly added task executor cannot be managed correctly in the future.

S104: Schedule the newly added task executor from the expanded integration module, and control the newly added task executor to execute the to-be-processed task.

It should be noted that since the newly added task executor must be in an idle state, after the establishment is completed, the newly added task executor can be scheduled from the integration module, and the pending tasks are assigned to the newly added task executor for completion, and the end of this time. Task Assignment.

In actual use, although some developers choose to use Docker instead of slave nodes, as shown in Figure 2, in this case, only the slave and Docker containers are replaced. The consequence is that the multiple slaves originally controlled by the master node become In order to control multiple Docker containers, multiple Docker containers are not integrated, and the complexity of master node control increases, which can still be accomplished with the help of separate cluster management tools such as swarm or kubernetes. In addition, in the prior art, the topology of the Docker container is also converted, for example, the master node is containerized, and the continuous integration task is completed after the conversion, but the function of dynamic scaling still cannot be realized.

In essence, the containerization of these Jenkins continuous integration systems does not optimize scheduling for the continuous integration requirements of large-scale projects and codes, and does not use specific algorithms to improve the efficiency of continuous integration parallel processing. The specific shortcomings are as follows:

(1) Additional cluster management tools, such as swarm or kubernetes, need to be added to perform additional management of clustered slave nodes. This will add additional management resources and management costs to deploy;

(2) The number of slave nodes is fixed by default, and the processing capacity is relatively solid. If too many containers are started to process continuous integration tasks, there will be too many preset startup containers in the face of concurrent task requirements, resulting in a certain degree of resource consumption. waste;

(3) If fewer containers are started for adaptive scheduling tasks, in the face of larger-scale task requirements, fewer containers cannot fully improve the efficiency of parallel processing, resulting in insufficient processing capacity.

In this application, each specific task node of the continuous integration system is containerized by Docker to form a task executor, and the overall task executor is re-contained to form a Slave Docker node, that is to say, the SlaveDocker node is re-contained. Distributed containerization, forming the layout of Docker0, Docker1... to Dockern. In this way, the master node is still connected to a single Slave Docker node, which simplifies the complexity of the connection operation and reduces the cost of connection management, without the need for additional cluster management tools; while the Slave Docker node uses the adaptive adjustment algorithm described in this embodiment for internal Docker scheduling, by sensing the scale of the business volume, adjusts the number of task executors through an adaptive scheduling method. This maximizes the concurrent processing capability of the continuous integration platform, which has the following significant advantages:

(1) Compared with the relatively fixed processing capability of containerized continuous integration; this application designs dynamic processing capability; compared with the industry practice, this mechanism can realize flexible and efficient invocation of respective processing steps;

(2) The maximum adaptive processing efficiency according to processing requirements is achieved through efficient scheduling algorithms;

(3) Since the number of task executors is flexibly adjusted according to the processing requirements, the task executors in the idle state will be closed, so when the Slave Docker node fails, only the Docker in the open state can be fault located and To repair, there is no need to search for all Dockers one by one, which can improve the efficiency of fault repair.

This embodiment provides an adaptive scheduling method, by acquiring the tasks to be processed in the task list; according to the tasks to be processed, it is judged in turn whether the multiple task executors included in the integrated module are in an idle state; When multiple task executors are in a non-idle state, expand the number of task executors in the integrated module; schedule newly added task executors from the expanded integrated module, and control the newly added task executors to execute the described Tasks to be processed; in this way, the integration module can adaptively adjust the number of task executors according to the business scale in the task list, thereby providing a dynamically scalable continuous integration environment, which improves the concurrent processing capability of tasks; avoids continuous integration systems The disadvantage of having to maintain a fixed number of task processors after startup solves the problem of curing the processing capacity of continuous integration; avoids adding additional cluster management tools to manage multiple task executors, and simplifies the connection process and connection management costs.

In another embodiment of the present application, please refer to FIG. 5 , which shows a schematic flowchart of another adaptive scheduling method provided by an embodiment of the present application. The adaptive scheduling method may include:

S201: Determine whether the i-th task executor in the integrated module is in an idle state;

It should be noted that i represents the number of the task executor for the task executor, i is a positive integer greater than or equal to 0 and less than n, and n represents the number of character executors. The number is used to determine whether the task executor is in an idle state. Generally speaking, the number of the task executor with the earlier establishment time is smaller, and the number of the task executor with the later establishment time is larger.

It should also be noted that the task executors in the integrated module are judged one by one through i, and this step also corresponds to the concept of "sequentially" in the previous embodiment. When the ith task executor is in a non-idle state, it indicates that the ith task executor has been called, and at this time, it is necessary to judge the next task executor.

In this way, for step S201, if the judgment result is no, it indicates that the ith task executor in the integrated module is in a non-idle state, then step S202 can be executed; if the judgment result is yes, it indicates that the ith task executor in the integrated module executes If the device is in an idle state, steps S203 and S204 can be performed.

S202: when the i-th task executor in the integrated module is in a non-idle state, execute i=i+1, and return to step S201;

Here, if the ith task executor is in a non-idle state, the query variable is incremented by one, and the process returns to step S201 to query the next task executor.

S203: when the ith task executor in the integrated module is in an idle state, call the ith task executor from the integrated module;

Here, if the ith task executor is idle, the task executor can be called to execute the pending task.

S204: Control the i-th task executor to execute the to-be-processed task, and end judging whether the task executor that has not been judging an idle state in the integrated module is in an idle state.

It should be noted that if there are idle task executors in the integration module, it means that the current number of task executors is sufficient to meet the existing task requirements, and there is no need to increase the number of task executors, and the tasks to be processed can be directly allocated. Executes the task executor that is in the idle state. At this point, the task to be processed this time has been allocated, so there is no need to continue to judge whether the remaining task executors are in an idle state, and this process can be ended.

The embodiment of the present application provides an adaptive scheduling method, and the specific implementation of the foregoing embodiment is described in detail, from which it can be seen that the integration module can adaptively adjust the number of task executors according to the business scale in the task list, Thereby, a dynamic scaling continuous integration environment is provided, which improves the concurrent processing capability of tasks; avoids the drawback that a fixed number of task processors must be maintained after the continuous integration system is started, and solves the curing problem of continuous integration processing capabilities; avoids adding additional Cluster management tools to manage multiple task executors, simplifying the connection process and the cost of connection management.

In another embodiment of the present application, referring to FIG. 6 , it shows another adaptive scheduling method provided by the embodiment of the present application. As shown in FIG. 6 , the method includes:

S301: Determine whether there is a pending task in the task list;

Here, for step S301, if the judgment result is no, step S302 is executed; if the judgment result is yes, step S303 is executed.

It should be noted that, for the task list, there may or may not be tasks to be processed. In this case, different responses need to be made according to the two situations. In addition, step S301 is generally designed as a timed task, so as to prevent the excessively frequent opening process from causing excessive processing pressure.

S302: Determine whether the Nth task executor in the integrated module is in an idle state;

Here, for step S303, if the judgment result is yes, step S304 is executed.

It should be noted that, in step S303, the generation time of the Nth task executor is later than the generation time of the task executors except the Nth task executor in the integration module, and N is greater than 1. positive integer;

It should be noted that when considering whether to close the task executor, it mainly refers to whether the task executor with the latest establishment time needs to be closed. Generally speaking, the task executor with the latest establishment time is executed according to the preset number. It is determined that the generation time of the Nth task executor is later than the generation time of the task executors other than the Nth task executor in the integration module.

As for why only the task executors with the latest establishment time are considered, the reasons are as follows: task executors all have numbers for identification. Generally speaking, due to the characteristics of programming languages, the intervals between these numbers are fixed, such as 4 The task executors are respectively numbered 1, 2, 3, and 4. When using these numbers, the next target can be found as long as the search value is +1 each time, so the numbering interruption rarely occurs. Based on this, when considering the problem of task executor reduction, in order to reduce the complexity, only the working state of the last task executor is detected, and the problem of number interruption can be avoided. An example is as follows. There are multiple task executors with preset numbers 1, 2, 3, and 4. Only the task executor with preset number 4 is the target task executor, and then its working status is obtained to determine whether it is necessary to It is closed; if the working status of the task executor with the preset number 2 is also judged, then when the task executor with the preset number 2 is closed, there will be a problem of number interruption, which may cause the program to crash. .

It should be noted that, by judging the working state of the Nth task executor, it is decided whether to close it. If the Nth task executor is not in an idle state, it does not need to be closed, and this process can be ended directly.

S303: Turn off the Nth task executor, and reduce the number of task executors in the integrated module by one to reduce the number of task executors in the integrated module;

It should be noted that if the Nth task executor is in an idle state, it can be closed, and the number of task executors is reduced by 1, so that the number of task executors in the integration module can correspond to current business requirements in real time.

S304: Acquire the to-be-processed task from the task list;

It should be noted that if there is a task to be processed, the task to be processed may be obtained from the task list to control a suitable task executor for execution.

S305: According to the task to be processed, sequentially determine whether the multiple task executors included in the integrated module are in an idle state;

It should be noted that, after acquiring the tasks to be processed, the statuses of the task executors in the integration module are judged one by one to determine whether the number of task executors needs to be expanded.

S306: when the multiple task executors are all in a non-idle state, expand the number of task executors in the integrated module;

It should be noted that when the multiple task executors are all in a non-idle state, it means that the number of existing task executors is insufficient to meet the task requirements, so the number of task executors in the integrated module needs to be expanded.

S307: Schedule the newly added task executor from the expanded integration module, and control the newly added task executor to execute the to-be-processed task.

It should be noted that, the task to be processed can be executed by using the newly added task executor, which also ends the process of this adaptive scheduling.

In this way, through the adaptive scheduling method described in this embodiment, an adaptive and scalable continuous integration solution is realized, a continuous integration structure is designed, and dynamic processing is provided by dynamically scaling the number of Docker containers according to the business scale The capability mechanism provides the structural topology of continuous integration applying the dynamic scaling mechanism, which overcomes the following shortcomings of the existing continuous integration environment:

(1) The common node structure of the master-slave topology of the distributed structure cannot realize the isolation of the continuous integration environment. The destruction of the node environment of a single point slave still requires manual node recovery operations, thus affecting the overall adaptive scheduling task. execution efficiency;

(2) In the traditional adaptive scheduling system as shown in Figure 2, although each Docker container simply implements a stable environment-isolated slave function node, additional cluster management tools need to be added, adding additional clustered slave nodes. There are the following disadvantages: (a) multiple containers are pre-set to process adaptive scheduling tasks, and in the face of concurrent task requirements, it is possible to pre-set and start too many containers, resulting in a waste of resources to a certain extent; (b) ) or start fewer containers for adaptive scheduling tasks, and in the face of larger-scale task requirements, fewer containers for execution cannot fully improve the efficiency of parallel processing;

(3) Although the continuous integration system containerized by Docker achieves the isolation of the environment, it lacks the algorithm coordination of the dynamic scheduling tool and the mobilization of the related algorithm mobilization mechanism;

(4) The existing continuous integration system lacks the ability to dynamically call the number of slave nodes, that is to say, there is no mechanism for the number of Docker containers to dynamically change according to the needs of the task volume.

The embodiment of the present application provides an adaptive scheduling method, and the specific implementation of the foregoing embodiment is described in detail, from which it can be seen that the integration module can adaptively adjust the number of task executors according to the business scale in the task list, Thereby, a dynamic scaling continuous integration environment is provided, which improves the concurrent processing capability of tasks; avoids the drawback that a fixed number of task processors must be maintained after the continuous integration system is started, and solves the curing problem of continuous integration processing capabilities; avoids adding additional Cluster management tools to manage multiple task executors, simplifying the connection process and the cost of connection management.

In still another embodiment of the present application, referring to FIG. 7 , it shows a schematic flowchart of still another adaptive scheduling method provided by the embodiment of the present application. As shown in FIG. 7 , the method includes:

S401: Initialize task queue list, initialize preset Docker0, iteration variable i=0; n=1;

It should be noted that the task queue list, task executor and iteration variables need to be initialized when the system starts. The task queue list is used to record the tasks to be assigned. Docker0 is the task executor preset at startup, and the value of n corresponds to It depends on the number of task executors in the system, so when initializing the preset Docker0, the value of n needs to be initialized to 1, and the value of i is the variable for rotation query in the scheduling task, which is initialized to 0. That is to say, in this embodiment, there is only one Docker node at the beginning of the integrated service, and there is no need to initialize too many preset nodes. At the same time, the number of Docker nodes can be dynamically increased and decreased with the number of integrated services, so as to maximize the number of Docker nodes. The concurrent processing capability of the adaptive scheduling system.

It should also be noted that this embodiment does not limit that there is only one preset Docker node during initialization. If multiple Dockers are preset to be started during initialization, it is also within the protection scope of this application. For example, if two Dockers are preset node, then S401 may include:

Initialize the task queue list, initialize the preset Docker0 and initialize the preset Docker1, and iterate variables i=0, n=2.

S402: Determine whether there is a new task;

Here, for step S402, if the judgment result is yes, then step S403 can be executed; if the judgment result is no, then step S409 is executed;

It should be noted that to determine whether there is a new task, it can be set as a timed task, or a monitoring mechanism can be used, that is, to determine whether there is a new task regularly, or to monitor whether there is a new task event.

S403: Insert a new task into the task queue list and assign values;

It should be noted that when a new task is detected, the new task is inserted into the task queue list, generally in a first-come, first-served order. The purpose of assignment is to identify different new tasks and serve as a reference to the new task.

S404: take out the next pending task in the list queue;

It should be noted that the tasks to be processed are generally the header tasks of the list queue, that is, the tasks to be processed are taken out in a certain order.

It should also be noted that this step S404 is performed continuously. After each execution of step S404, the next task to be processed needs to be taken out again after the task to be processed is allocated, until all tasks in the list queue are allocated. .

S405: Determine whether Docker0+i is free;

Here, for step S402, if the judgment result is yes, execute step S406; if the judgment result is no, execute step S407;

It should be noted that, at this time, the status of multiple Docker containers is judged sequentially according to the number, and then it is determined whether the number of task executors needs to be expanded.

S406: Allocate the task to be processed to Docker0+i, and reset i=0;

It should be noted that if it is determined that the Docker is in an idle state, the tasks to be processed can be allocated, and the allocation of the tasks to be processed is ended at this time.

It should also be noted that since step S404 is performed continuously, from an overall point of view, S406 only represents the end of this process.

S407: Control i=i+1, and judge whether Docker0+i exists;

Here, corresponding to step S407, if the judgment result is yes, then step S405 is executed; if the judgment result is no, then step S408 is executed;

It should be noted that if the current Docker0+i is not in an idle state, then it is necessary to judge the working state of the next Docker; this needs to consider that the current Docker0+i is the last Docker in the system, then it does not exist at this time. The next Docker, so before judging the working status of the next Docker, it is necessary to judge whether it exists.

S408: Create a new Docker0+i, assign the pending tasks to Docker0+i, and reset i=0;

It should be noted that if there is no next Docker, the current number of Docker is not enough to cope with the current task volume, and the number of Docker needs to be increased. Specifically, create a new Docker, and assign the tasks to be processed to the newly created Docker. At the same time, it is implied here that n needs to be incremented by 1. Similarly, S408 also represents the end of this process.

It should also be noted that, for steps S406 and S408, after the task to be processed is allocated to the task executor for completion, the next task to be processed needs to be taken out from the task queue list and continued to be allocated until the task queue list does not exist. pending tasks.

S409: Determine whether Docker n-1 is idle;

Here, for step S409, if the judgment result is yes, then step S410 can be executed; if the judgment result is no, return to execute step S402;

It should be noted that if there are no pending tasks, it is necessary to consider whether to reduce the number of Dockers. Specifically, it is necessary to consider whether to close Docker n-1.

S410: Turn off Docker n-1, set n=n-1, and return to step S402.

It should be noted that, if Docker n-1 is in an idle state, in order to avoid waste of processing capacity, it can be closed, and the process returns to step S402.

The embodiment of the present application provides an adaptive scheduling method, and the specific implementation of the foregoing embodiment is described in detail, from which it can be seen that the integration module can adaptively adjust the number of task executors according to the business scale in the task list, Thereby, a dynamic scaling continuous integration environment is provided, which improves the concurrent processing capability of tasks; avoids the drawback that a fixed number of task processors must be maintained after the continuous integration system is started, and solves the curing problem of continuous integration processing capabilities; avoids adding additional Cluster management tools to manage multiple task executors, simplifying the connection process and the cost of connection management.

In yet another embodiment of the present application, referring to FIG. 8 , it shows a schematic structural diagram of an adaptive scheduling apparatus 50 provided by an embodiment of the present application. As shown in FIG. 8 , the adaptive scheduling apparatus 50 includes an acquisition unit 501, the judgment unit 502, the expansion unit 503 and the scheduling unit 504; wherein,

an obtaining unit 501, configured to obtain tasks to be processed in the task list;

The judging unit 502 is configured to sequentially judge whether the multiple task executors included in the integrated module are in an idle state according to the tasks to be processed;

an expansion unit 503, configured to expand the number of task executors in the integrated module when the multiple task executors are all in a non-idle state;

The scheduling unit 504 is configured to schedule the newly added task executor from the expanded integration module, and control the newly added task executor to execute the to-be-processed task.

In the above embodiment, the obtaining unit 501 is specifically configured to determine whether there is a task to be processed in the task list; if there is a task to be processed in the task list, obtain the task to be processed from the task list.

Referring to FIG. 9 , it shows a schematic structural diagram of another adaptive scheduling apparatus 50 provided by an embodiment of the present application. As shown in FIG. 9 , on the basis of the foregoing embodiment, the adaptive scheduling apparatus 50 further includes a reduction unit 505. Configure to determine whether the Nth task executor in the integration module is in an idle state if there is no task to be processed in the task list; wherein, the generation time of the Nth task executor is later than The generation time of the task executors other than the Nth task executor in the integrated module, N is a positive integer greater than 1; if the Nth task executor in the integrated module is in an idle state, all The Nth task executor is processed, and the number of task executors in the integration module is decremented by one, so as to reduce the number of task executors in the integration module.

In the above embodiment, the expansion unit 503 is specifically configured to create a new task executor in the integrated module, and increase the number of task executors in the integrated module by one, so as to expand the integrated module The number of task executors included.

As shown in FIG. 9 , on the basis of the above embodiment, the scheduling unit 504 is configured to call the i-th task executor from the integration module if the i-th task executor in the integration module is in an idle state task executor; wherein, i represents the number of the task executor, and i is a positive integer greater than or equal to 0; control the i-th task executor to execute the to-be-processed task, and end the integration module that is not idle Judging whether the task executor of the state judgment is in an idle state.

In the above embodiment, the adaptive scheduling apparatus 50 further includes an initialization unit 506 configured to initialize the integrated module, so that the integrated module includes at least one task executor.

In the above embodiment, the task executor includes a Docker container.

It can be understood that, in this embodiment, a "unit" may be a part of a circuit, a part of a processor, a part of a program or software, etc., of course, it may also be a module, and it may also be non-modular. Moreover, each component in this embodiment may be integrated into one processing unit, or each unit may exist physically alone, or two or more units may be integrated into one unit. The above-mentioned integrated units can be implemented in the form of hardware, or can be implemented in the form of software function modules.

If the integrated unit is implemented in the form of a software functional module and is not sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this embodiment is essentially or The part that contributes to the prior art or the whole or part of the technical solution can be embodied in the form of a software product, the computer software product is stored in a storage medium, and includes several instructions for making a computer device (which can be It is a personal computer, a server, or a network device, etc.) or a processor (processor) that executes all or part of the steps of the method described in this embodiment. The aforementioned storage medium includes: U disk, removable hard disk, Read Only Memory (ROM), Random Access Memory (Random Access Memory, RAM), magnetic disk or optical disk and other media that can store program codes.

Therefore, the present embodiment provides a computer storage medium storing an adaptive scheduler that implements the method described in any one of the preceding embodiments when the adaptive scheduler is executed by at least one processor. step.

Based on the composition of the above-mentioned adaptive scheduling apparatus 50 and the computer storage medium, see FIG. 10 , which shows a specific hardware structure example of an adaptive scheduling apparatus 50 provided by an embodiment of the present application, which may include: a communication interface 601 , memory 602 and processor 603 ; the various components are coupled together by a bus system 604 . It will be appreciated that the bus system 604 is used to implement connection communication between these components. In addition to the data bus, the bus system 604 also includes a power bus, a control bus and a status signal bus. However, for clarity of illustration, the various buses are labeled as bus system 604 in FIG. 10 . Among them, the communication interface 601 is used for receiving and sending signals in the process of sending and receiving information with other external network elements;

memory 602 for storing computer programs that can run on processor 603;

The processor 603 is configured to, when running the computer program, execute:

Get the pending tasks in the task list;

According to the to-be-processed task, sequentially determine whether the multiple task executors included in the integrated module are in an idle state;

When the multiple task executors are all in a non-idle state, expanding the number of task executors in the integrated module;

The newly added task executor is scheduled from the expanded integration module, and the newly added task executor is controlled to execute the to-be-processed task.

It can be understood that the memory 602 in this embodiment of the present application may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memory. Wherein, the non-volatile memory may be Read-Only Memory (ROM), Programmable Read-Only Memory (PROM), Erasable Programmable Read-Only Memory (Erasable PROM, EPROM), Erase programmable read-only memory (Electrically EPROM, EEPROM) or flash memory. The volatile memory may be random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as Static RAM (SRAM), Dynamic RAM (DRAM), Synchronous DRAM, SDRAM), double data rate synchronous dynamic random access memory (Double Data Rate SDRAM, DDRSDRAM), enhanced synchronous dynamic random access memory (Enhanced SDRAM, ESDRAM), synchronous link dynamic random access memory (Synchronous link DRAM, SLDRAM) And direct memory bus random access memory (Direct Rambus RAM, DRRAM). The memory 602 of the systems and methods described herein is intended to include, but not be limited to, these and any other suitable types of memory.

The processor 603 may be an integrated circuit chip with signal processing capability. In the implementation process, each step of the above-mentioned method can be completed by an integrated logic circuit of hardware in the processor 603 or an instruction in the form of software. The aforementioned processor 603 may be a general-purpose processor, a digital signal processor (Digital Signal Processor, DSP), an application specific integrated circuit (Application Specific Integrated Circuit, ASIC), a field programmable gate array (Field Programmable Gate Array, FPGA), or other Programmable logic devices, discrete gate or transistor logic devices, discrete hardware components. The methods, steps, and logic block diagrams disclosed in the embodiments of this application can be implemented or executed. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like. The steps of the method disclosed in conjunction with the embodiments of the present application may be directly embodied as executed by a hardware decoding processor, or executed by a combination of hardware and software modules in the decoding processor. The software modules may be located in random access memory, flash memory, read-only memory, programmable read-only memory or electrically erasable programmable memory, registers and other storage media mature in the art. The storage medium is located in the memory 602, and the processor 603 reads the information in the memory 602, and completes the steps of the above method in combination with its hardware.

It will be appreciated that the embodiments described herein may be implemented in hardware, software, firmware, middleware, microcode, or a combination thereof. For hardware implementation, the processing unit may be implemented in one or more Application Specific Integrated Circuits (ASIC), Digital Signal Processing (DSP), Digital Signal Processing Device (DSP Device, DSPD), programmable logic Devices (ProgrammableLogic Device, PLD), Field-Programmable Gate Array (Field-Programmable Gate Array, FPGA), general purpose processors, controllers, microcontrollers, microprocessors, other electronic units for performing the functions described in this application or a combination thereof.

For a software implementation, the techniques described herein may be implemented through modules (eg, procedures, functions, etc.) that perform the functions described herein. Software codes may be stored in memory and executed by a processor. The memory can be implemented in the processor or external to the processor.

Optionally, as another embodiment, the processor 603 is further configured to execute the steps of the method in any one of the foregoing embodiments when running the computer program.

Based on the composition and hardware structure example of the above-mentioned adaptive scheduling apparatus 50, see FIG. 11, which shows a schematic composition structure diagram of a system 70 provided by an embodiment of the present application. As shown in FIG. 11 , the system 70 includes at least the adaptive scheduling apparatus 50 described in any one of the foregoing embodiments. Among them, the system can adaptively adjust the number of task executors in the integration module according to the business scale in the task list, thereby providing a dynamic scaling continuous integration environment, improving the concurrent processing capability of tasks; avoiding continuous integration system startup Afterwards, the drawback of having to maintain a fixed number of task processors solves the problem of curing the continuous integration processing capability; it avoids adding additional cluster management tools to manage multiple task executors, and simplifies the connection process and connection management costs.

The above descriptions are only preferred embodiments of the present application, and are not intended to limit the protection scope of the present application.

It should be noted that, in this application, the terms "comprising", "comprising" or any other variation thereof are intended to encompass non-exclusive inclusion, such that a process, method, article or device comprising a series of elements includes not only those elements , but also other elements not expressly listed or inherent to such a process, method, article or apparatus. Without further limitation, an element qualified by the phrase "comprising a..." does not preclude the presence of additional identical elements in a process, method, article or apparatus that includes the element.

The above-mentioned serial numbers of the embodiments of the present application are only for description, and do not represent the advantages or disadvantages of the embodiments.

The methods disclosed in the several method embodiments provided in this application can be arbitrarily combined under the condition of no conflict to obtain new method embodiments.

The features disclosed in the several product embodiments provided in this application can be combined arbitrarily without conflict to obtain a new product embodiment.

The features disclosed in several method or device embodiments provided in this application can be combined arbitrarily without conflict to obtain new method embodiments or device embodiments.

The above are only specific embodiments of the present application, but the protection scope of the present application is not limited to this. should be covered within the scope of protection of this application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims (11)

1. An adaptive scheduling method, wherein the method comprises: Get the pending tasks in the task list; According to the to-be-processed task, sequentially determine whether the multiple task executors included in the integrated module are in an idle state; When the multiple task executors are all in a non-idle state, expanding the number of task executors in the integrated module; The newly added task executor is scheduled from the expanded integration module, and the newly added task executor is controlled to execute the to-be-processed task. 2. The method according to claim 1, wherein the obtaining the tasks to be processed in the task list comprises: Determine whether there are pending tasks in the task list; If there is a to-be-processed task in the task list, acquire the to-be-processed task from the task list. 3. The method according to claim 2, wherein after judging whether there is a task to be processed in the task list, the method further comprises: If there is no task to be processed in the task list, determine whether the Nth task executor in the integration module is in an idle state; wherein, the generation time of the Nth task executor is later than that of the integration module The generation time of the task executors except the Nth task executor in the N is a positive integer greater than 1; If the N th task executor in the integrated module is in an idle state, the N th task executor is closed, and the number of task executors in the integrated module is reduced by one, so as to reduce the integrated module The number of internal task executors. 4. The adaptive scheduling method according to claim 1, wherein the expanding the number of task executors included in the integration module comprises: A new task executor is established in the integration module, and the number of task executors in the integration module is increased by one, so as to expand the number of task executors included in the integration module. 5. The adaptive scheduling method according to claim 1, wherein the step of judging whether multiple task executors included in the integrated module are in an idle state in turn comprises: If the ith task executor in the integrated module is in an idle state, the ith task executor is called from the integrated module; wherein, i represents the number of the task executor, and i is greater than or equal to 0 positive integer of ; The i-th task executor is controlled to execute the to-be-processed task, and the judgment on whether the task executor that has not been judging the idle state in the integrated module is in an idle state is ended. 6. The adaptive scheduling method according to any one of claims 1-5, characterized in that, before acquiring the tasks to be processed in the task list, the method further comprises: The integration module is initialized such that the integration module includes at least one task executor. 7. The adaptive scheduling method according to claim 6, wherein the task executor comprises a Docker container. 8. An adaptive scheduling device, characterized in that the adaptive scheduling device comprises an acquisition unit, a judgment unit, an extension unit and a scheduling unit; wherein, The obtaining unit is configured to obtain the tasks to be processed in the task list; The judging unit is configured to sequentially judge whether the multiple task executors included in the integrated module are in an idle state according to the tasks to be processed; the expansion unit, configured to expand the number of task executors in the integrated module when the multiple task executors are all in a non-idle state; The scheduling unit is configured to schedule the newly added task executor from the expanded integration module, and control the newly added task executor to execute the to-be-processed task. 9. An adaptive scheduling device, characterized in that the adaptive scheduling device comprises a memory and a processor; wherein, the memory for storing a computer program executable on the processor; The processor is configured to execute the steps of the method according to any one of claims 1 to 7 when running the computer program. 10. A computer storage medium, characterized in that the computer storage medium stores an adaptive scheduler, and when the adaptive scheduler is executed by at least one processor, implements the method according to any one of claims 1 to 7. step. 11. A system, characterized in that the system comprises at least the adaptive scheduling apparatus according to claim 8 or 9.

Images