CN114661462A - Resource allocation method, system, computer-readable storage medium, and electronic device - Google Patents

Abstract

The application discloses a resource allocation method, a system, a computer readable storage medium and an electronic device. Wherein, the method comprises the following steps: acquiring operation characteristic information of target operation to be executed by a target computing cluster; determining a target computing resource type based on the job characteristic information; and allocating target computing resources for the target computing cluster, wherein the target computing resources are computing resources required by the target computing cluster to execute the target job, and the computing resource type of the target computing resources is the target computing resource type. The method and the device solve the technical problem of unreasonable resource allocation caused by selecting similar computing resources for all jobs in the prior art.

Description

Resource allocation method, system, computer-readable storage medium, and electronic device

technical field

The present application relates to the field of artificial intelligence, and in particular, to a resource allocation method, system, computer-readable storage medium, and electronic device.

Background technique

Cloud computing platform, also known as cloud platform, refers to services based on hardware resources and software resources, providing computing, network and storage capabilities. With the development of cloud computing and artificial intelligence, there are more and more cloud demands on cloud computing service platforms, and the requirements for computing specifications are also diversified, and the scale of a single cluster is also increasing.

At present, in the process of processing jobs, the relevant cloud computing service platform selects the same computing resources for all jobs for unified processing, resulting in unreasonable resource allocation and unable to take advantage of the computing resources on the cloud.

For the above problems, no effective solution has been proposed yet.

SUMMARY OF THE INVENTION

Embodiments of the present application provide a resource allocation method, system, computer-readable storage medium, and electronic device to at least solve the technical problem of unreasonable resource allocation caused by selecting the same computing resources for all jobs in the prior art.

According to an aspect of the embodiments of the present application, a resource allocation method is provided, including: acquiring job feature information of a target job to be executed by a target computing cluster; determining a target computing resource type based on the job feature information; allocating target computing to a target computing cluster resources, where the target computing resource is the computing resource required by the target computing cluster to execute the target job, and the computing resource type of the target computing resource is the target computing resource type.

According to another aspect of the embodiments of the present application, a resource allocation method is also provided, including: responding to a job creation instruction, creating a target job and a target computing cluster for executing the target job, and displaying job feature information of the target job; The target computing resource type determined by the feature information; in response to the resource allocation instruction, the target computing resources allocated to the target computing cluster are displayed, wherein the target computing resources are the computing resources required by the target computing cluster to execute the target job, and the computing resources of the target computing resources The resource type is the target computing resource type.

According to another aspect of the embodiments of the present application, a resource allocation device is further provided, including: an acquisition module, configured to acquire job feature information of a target job to be executed by a target computing cluster; a determination module, configured to determine based on the job feature information The target computing resource type; the allocation module is used to allocate target computing resources to the target computing cluster, wherein the target computing resources are the computing resources required by the target computing cluster to execute the target job, and the computing resource type of the target computing resources is the target computing resource type.

According to another aspect of the embodiments of the present application, a resource allocation apparatus is also provided, including: a first response module, configured to respond to a job creation instruction, create a target job and a target computing cluster for executing the target job, and display the target job's job feature information; a display module for displaying the target computing resource type determined based on the job feature information; a second response module for responding to a resource allocation instruction and displaying target computing resources allocated for the target computing cluster, wherein the target computing The resource is the computing resource required by the target computing cluster to execute the target job, and the computing resource type of the target computing resource is the target computing resource type.

According to another aspect of the embodiments of the present application, a resource allocation system is also provided, including: a scheduler, configured to receive a target job submitted by a target object; Job feature information, and determine the target computing resource type based on the job feature information; the resource management component is used to determine the target computing resource according to the target computing resource type, create a task queue for the scheduler, and then add the target computing resource to the scheduler based on the task queue; The scheduler also allocates the target computing resources to the task queues corresponding to the computing nodes of the target computing cluster based on the preset scheduling policy, so that the computing nodes in the target computing cluster execute the target job.

According to another aspect of the embodiments of the present application, a computer-readable storage medium is also provided, where a computer program is stored in the computer-readable storage medium, wherein the computer program is configured to execute the above-mentioned resource allocation method when running.

According to another aspect of the embodiments of the present application, there is also provided an electronic device, the electronic device includes one or more processors; a memory for storing one or more programs, when the one or more programs are processed by one or more The processor, when executed, causes one or more processors to be implemented for running a program, wherein the program is configured to execute the resource allocation method described above at runtime.

In the embodiment of the present invention, a method of allocating different types of computing resources to each job based on job feature information of the job is adopted, the job feature information of the target job to be executed by the target computing cluster is obtained, and then the target computing resource is determined based on the job feature information. The target computing resource is allocated to the target computing cluster, wherein the target computing resource is the computing resource required by the target computing cluster to execute the target job, and the computing resource type of the target computing resource is the target computing resource type.

In the above process, target computing resources are allocated to the target computing cluster, which realizes the processing of each job based on different types of computing resources, and avoids the phenomenon of resource waste or insufficiency in selecting the same computing resources for all jobs, thereby improving resources. The rationality of the allocation, thereby improving the operation efficiency of the job and improving the user experience. In addition, in the present application, the target computing resource type is determined based on the job feature information, so as to determine the most suitable computing resource type of computing resource for processing each target job, avoiding the difference and inaccuracy of manual selection of computing resources , which further improves the operation efficiency of the operation.

It can be seen that the solution provided by the present application achieves the purpose of allocating different types of computing resources to each job based on the job feature information of the job, thereby achieving the technical effect of improving the rationality of resource allocation, and further solving the problems caused by the prior art. The technical problem of unreasonable resource allocation caused by selecting the same computing resources for all jobs.

Description of drawings

The drawings described herein are used to provide further understanding of the present application and constitute a part of the present application. The schematic embodiments and descriptions of the present application are used to explain the present application and do not constitute an improper limitation of the present application. In the attached image:

1 is a block diagram of a hardware structure of an optional electronic device (or mobile device) according to an embodiment of the present application;

FIG. 2 is an embodiment in which an optional electronic device (or mobile device) is used as a transmitting end according to an embodiment of the present application;

3 is a flowchart of an optional resource allocation method according to an embodiment of the present application;

4 is a schematic diagram of a resource allocation method according to the prior art;

5 is a schematic diagram of a resource allocation method according to the prior art;

6 is a sequence diagram of an optional resource allocation method according to an embodiment of the present application;

7 is a schematic diagram of an optional feature collection component according to an embodiment of the present application;

8 is a flowchart of an optional resource allocation method according to an embodiment of the present application;

9 is a schematic diagram of an optional human-computer interaction operation according to an embodiment of the present application;

FIG. 10 is a schematic diagram of an optional resource allocation apparatus according to an embodiment of the present application;

FIG. 11 is a schematic diagram of an optional resource allocation apparatus according to an embodiment of the present application;

12 is a schematic diagram of an optional resource allocation system according to an embodiment of the present application;

FIG. 13 is a structural block diagram of an optional electronic device according to an embodiment of the present application.

Detailed ways

In order to make those skilled in the art better understand the solutions of the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only The embodiments are part of the present application, but not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work shall fall within the scope of protection of the present application.

It should be noted that the terms "first", "second", etc. in the description and claims of the present application and the above drawings are used to distinguish similar objects, and are not necessarily used to describe a specific sequence or sequence. It is to be understood that data so used may be interchanged under appropriate circumstances so that the embodiments of the application described herein can be practiced in sequences other than those illustrated or described herein. Furthermore, the terms "comprising" and "having" and any variations thereof, are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those expressly listed Rather, those steps or units may include other steps or units not expressly listed or inherent to these processes, methods, products or devices.

Example 1

According to the embodiments of the present application, an embodiment of a resource allocation method is also provided. It should be noted that the steps shown in the flowcharts of the accompanying drawings may be executed in a computer system such as a set of computer-executable instructions, and, Although a logical order is shown in the flowcharts, in some cases steps shown or described may be performed in an order different from that herein.

The method embodiment provided in Embodiment 1 of the present application may be executed in a mobile terminal, an electronic device, or a similar computing device. FIG. 1 shows a block diagram of a hardware structure of an electronic device (or mobile device) for implementing a resource allocation method. As shown in FIG. 1 , the electronic device 10 (or the mobile device 10 ) may include one or more (shown as 102a, 102b, . . . , 102n in the figure) processors (the processors may include but are not limited to microprocessors A processing device such as an MCU or a programmable logic device FPGA), a memory 104 for storing data, and a transmission module 106 for communication functions. In addition, may also include: display, input/output interface (I/O interface), universal serial bus (USB) port (may be included as one of the ports of the BUS bus), network interface, power supply and/or or camera. Those of ordinary skill in the art can understand that the structure shown in FIG. 1 is only a schematic diagram, which does not limit the structure of the above electronic device. For example, electronic device 10 may also include more or fewer components than shown in FIG. 1 , or have a different configuration than that shown in FIG. 1 .

It should be noted that the one or more processors and/or other data processing circuits described above may generally be referred to herein as "data processing circuits". The data processing circuit may be embodied in whole or in part as software, hardware, firmware or any other combination. Furthermore, the data processing circuitry may be a single, stand-alone processing module, or incorporated in whole or in part into any of the other elements in electronic device 10 (or mobile device). As referred to in the embodiments of the present application, the data processing circuit acts as a kind of processor control (eg, selection of a variable resistance termination path connected to an interface).

The memory 104 can be used to store software programs and modules of the application software, such as a program instruction/data storage device corresponding to the resource allocation method in the embodiment of the present application. This kind of function application and data processing, that is, to realize the above-mentioned resource allocation method. Memory 104 may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, memory 104 may further include memory located remotely from the processor, which may be connected to electronic device 10 through a network. Examples of such networks include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

Transmission means 106 are used to receive or transmit data via a network. The specific example of the above-mentioned network may include the wireless network provided by the communication provider of the electronic device 10 . In one example, the transmission device 106 includes a network adapter (Network Interface Controller, NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 106 may be a radio frequency (Radio Frequency, RF) module, which is used for wirelessly communicating with the Internet.

The display may be, for example, a touch screen-style liquid crystal display (LCD) that enables a user to interact with the user interface of the electronic device 10 (or mobile device).

The block diagram of the hardware structure shown in FIG. 1 can be used not only as an exemplary block diagram of the above electronic device 10 (or mobile device), but also as an exemplary block diagram of the above server. In an optional embodiment, FIG. 2 is a block diagram An embodiment of using the electronic device 10 (or mobile device) shown in FIG. 1 as the sending end is presented. As shown in FIG. 2, the electronic device 10 (or mobile device) may be connected or electronically connected to one or more servers 108 via a data network, wherein the aforementioned servers may be servers in the same server cluster, or alternatively, a server cluster It can be a server cluster such as a resource cluster and a computing cluster, and the server can be a server such as a resource server, a computing server, a security server, or a game server. In an optional embodiment, the above-mentioned electronic device 10 (or mobile device) may be a mobile phone, a tablet, a computer or a smart wearable device. The data network connection may be a local area network connection, a wide area network connection, an Internet connection, or other type of data network connection. The electronic device 10 (or mobile device) may execute to connect to a network service executed by a server (eg, a computing server) or a group of servers. The web server 110 is a web-based user service, such as social networking, cloud resources, email, online payment or other online applications, and may also be a cloud server.

Under the above operating environment, the present application provides a resource allocation method as shown in FIG. 3 . FIG. 3 is a flowchart of an optional resource allocation method according to an embodiment of the present application. And in this application, the resource allocation system is used as the execution body of this embodiment, and the resource allocation system at least includes a resource scheduling component, a feature collection component, and a resource management component, wherein the resource scheduling component, the feature collection component, and the resource management component are mutually mutually exclusive. connect.

Step S202: Obtain job feature information of a target job to be executed by the target computing cluster.

In step S202, the job feature information of the target job to be executed by the target computing cluster may be obtained through an electronic device, a server, an application system, etc., and in this embodiment, the target computing cluster may be obtained through a resource scheduling component in the resource allocation system The target job to be executed, and then obtain job characteristic information of the target job through the characteristic collection component in the resource allocation system. Among them, the resource allocation system can generate the target job based on the desired operations performed by the user in the target application on the mobile phone, computer, tablet and other terminal devices, such as querying entries, browsing pages, switching programs, etc., and then the feature extraction component performs the target job. The job feature information is extracted. Optionally, the job feature information of the target job includes at least job information and system feature information; the target computing cluster can be used as a cloud computing service platform, and the target computing cluster can be an HPC (High Performance Computing, high-performance computing) cluster, which can be composed of The user creates the target computing resource in advance based on the resource allocation system before the target computing resource is allocated, and the target computing cluster includes a plurality of computing nodes, and each computing node is a server.

It should be noted that obtaining the job feature information can facilitate subsequent determination of the target computing resource type corresponding to the target job.

Step S204, determining the target computing resource type based on the job feature information.

In step S204, based on the feature collection component, the job feature information may be compared with the preset feature information in the feature database, the Internet, a cloud server, or other storage areas, so as to obtain the preset feature information that best matches the job feature information of the target job. The computing resource type corresponding to the feature information determines the target computing resource type. Optionally, the feature collection component may also directly compare the job feature information with each preset computing resource type, so as to determine the target computing resource type according to the comparison result. Optionally, the feature collection component may also input job feature information into the trained network learning model, so as to determine the target computing resource type based on the output result of the network learning model.

It should be noted that by determining the target computing resource type based on the job feature information, the determination of the computing resource type that is the most suitable computing resource for processing each target job is realized, and the difference and inaccuracy of manual selection of computing resources are avoided. This facilitates subsequent processing of each job based on different types of computing resources, and ensures the processing efficiency of each job.

Step S206: Allocate target computing resources to the target computing cluster, where the target computing resources are computing resources required by the target computing cluster to execute the target job, and the computing resource type of the target computing resources is the target computing resource type.

In step S206, target computing resources may be allocated to the target computing cluster based on the resource scheduling component in the resource allocation system. Optionally, before allocating a target computing resource to a target computing cluster, the resource management component in the resource allocation system can query the resource database, the Internet, a cloud server or other storage area based on the type of the target computing resource to find the type of the target computing resource. corresponding computing resources, so that the queried computing resources are sent to the resource scheduling component as the target computing resources, so that the resource scheduling component allocates the target computing resources to the target computing cluster.

It should be noted that, by allocating target computing resources to the target computing cluster, each job is processed based on different types of computing resources, avoiding the phenomenon of resource waste or insufficiency in selecting the same computing resources for all jobs, thereby improving The rationality of resource allocation, thereby improving the efficiency of job operation and improving the user experience.

In the prior art, there is an automatic scaling scheme on the public cloud, as shown in FIG. 4 , the scheme adjusts the computing scale of the computing cluster by performing automatic scaling according to the overall load such as CPU and memory. Similarly, in another automatic scaling solution for HPC solutions, as shown in Figure 5, this solution is bound to a single HPC cluster, and the computing resources are scaled to the cloud according to the load situation of the HPC cluster to adjust the computing The computing scale of the cluster. However, in the above two solutions, the resource elasticity capability of the scheduler of the HPC cluster is judged according to the system workload (load) or the job queue, that is, the aforementioned process is only based on the job queue for general processing, and it is often used for all Jobs are processed uniformly and similar computing resources are selected, without considering the computing resource requirements of the application characteristics of the job itself, nor will it collect and analyze the job characteristics, and select computing resources according to the analysis results, which will lead to unreasonable allocation problems.

Based on the solutions defined in the above steps S202 to S206, it can be known that, in this embodiment of the present invention, the method of allocating different types of computing resources to each job based on job feature information of the job is adopted, and the target computing cluster to be executed is obtained by obtaining the target computing resources. Job feature information of the job, and then determine the target computing resource type based on the job feature information, so as to allocate target computing resources to the target computing cluster, where the target computing resources are the computing resources required by the target computing cluster to execute the target job, and the computing resources of the target computing resources The resource type is the target computing resource type.

It is easy to notice that in the above process, target computing resources are allocated to the target computing cluster, which realizes the processing of each job based on different types of computing resources, and avoids the waste or shortage of resources that exist in selecting the same computing resources for all jobs. Therefore, the rationality of resource allocation is improved, the operation efficiency of the job is improved, and the user experience is improved. In addition, in the present application, the target computing resource type is determined based on the job feature information, so as to determine the most suitable computing resource type of computing resource for processing each target job, avoiding the difference and inaccuracy of manual selection of computing resources , which further improves the operation efficiency.

It can be seen that the solution provided by the present application achieves the purpose of allocating different types of computing resources to each job based on the job feature information of the job, thereby achieving the technical effect of improving the rationality of resource allocation, and further solving the problems caused by the prior art. The technical problem of unreasonable resource allocation caused by selecting the same computing resources for all jobs.

In an optional embodiment, in the process of obtaining job feature information of a target job to be executed by the target computing cluster, the feature collection component may obtain job information of the target job from a job list corresponding to the resource scheduling component, and then The system feature information corresponding to the target job is determined based on the target computing cluster. The resource scheduling component is used for receiving the target job submitted by the target object, and assigning the target job to the computing nodes of the target computing cluster, the job information at least includes: container dependency information when the target job is running, and the system feature information at least includes: computing The network bandwidth and memory usage of the node.

Optionally, as shown in FIG. 6 , after the resource scheduling component obtains the target job submitted by the user, the resource scheduling component may allocate the target job to the computing nodes of the target computing cluster. After that, the feature collection component can obtain job information of the target job from the job list corresponding to the resource scheduling component, where the job information at least includes container dependency information when the target job is running, job name, job detail information, software information, queue, etc., The job detail information may include information such as the size and content of the target job. Next, the feature collection component can collect the operation data of each computing node in the target computing cluster to obtain system feature information, so as to determine the job feature information based on the job information and the system feature information. The operation data represents the operation status of each computing node in the target computing cluster, and the system characteristic information at least includes information such as network bandwidth and memory occupancy rate of the computing node.

It should be noted that by acquiring the job information and system feature information of the target job, the job feature information of the target job is accurately determined, so that the target computing resource type determined based on the job feature information is more accurate, so that computing resources can be used efficiently. , improve the operation efficiency.

In an optional embodiment, in the process of determining the target computing resource type based on the job feature information, the feature collection component may query the computing resource type from a preset feature database based on the job feature information, obtain the query result, and store it in the The query result indicates that when there is a computing resource type corresponding to the job feature information of the target job in the preset feature database, determine that the computing resource type corresponding to the job feature information of the target job in the preset feature database is the target computing resource type, and The query result indicates that when there is no computing resource type corresponding to the job feature information of the target job in the preset feature database, the job feature information of the target job is compared with the target job feature information corresponding to the preset computing resource type, and Determine the target computing resource type according to the comparison result. Among them, the preset feature database stores at least job feature information of historical jobs, computing resource types, and an association relationship between job feature information of historical jobs and computing resource types, and computing resource types are required by the target computing cluster to execute historical jobs. The type of computing resource.

Optionally, as shown in FIG. 7 , the feature collection component can compare the job feature information of the currently running target job with the job feature information of all historical jobs in the preset feature database to determine the job feature of the target job. Job feature information of historical jobs whose information similarity exceeds a preset threshold, thereby obtaining query results. Optionally, the feature collection component can also compare the job feature information of the currently running target job with the job feature information of the historical jobs of the same industry in the preset feature database, so as to more quickly determine the job feature of the target job. Job feature information of historical jobs with similar information.

Further, when the query result indicates that there is a computing resource type corresponding to the job feature information of the target job in the preset feature database, that is, there is a difference between the job feature information of the historical job and the job feature information of the target job in the preset feature database. When the similarity exceeds the threshold, the feature collection component may determine the computing resource type corresponding to the job feature information of the historical job with the highest similarity to the job feature information of the target job as the computing resource type corresponding to the job feature information of the target job.

When the query result indicates that there is no computing resource type corresponding to the job feature information of the target job in the preset feature database, that is, the similarity between the job feature information of all historical jobs in the preset feature database and the job feature information of the target job When neither exceeds the threshold, the feature collection component may compare the job feature information of the target job with the target job feature information corresponding to the preset computing resource type, and determine the target computing resource type according to the comparison result. The preset computing resource types include at least a containerized instance computing resource type, a supercomputing cluster computing resource type, a physical computer computing resource type, and a virtual machine computing resource type.

It should be noted that by determining the target computing resource type by means of the job feature information of the historical job and the computing resource type, a more accurate judgment of the target computing resource type corresponding to each target job is achieved. In addition, in the case where the target computing resource type cannot be determined based on historical jobs, the target computing resource type is determined based on the preset target job feature information, which improves the applicability of the present application and avoids failure to obtain the target computing resource corresponding to the target job. The phenomenon of resource type occurs.

In an optional embodiment, in the process of comparing the job feature information of the target job with the target job feature information corresponding to the preset computing resource type, and determining the target computing resource type according to the comparison result, the feature The collection component can determine the job running form of the target job according to the container dependency information when the target job is running, and when the job running form characterizes the target job running in the container, determine the target computing resource type as the containerized instance computing resource type, where the container The instance computing resource type characterizes the target job's runtime dependency on the container.

Optionally, as shown in FIG. 7 , the feature collection component can determine the running form of the target job according to the container dependency information when the target job is running, and determine whether the target job runs in the container based on the running form of the target job, that is, Whether to run in a containerized manner, in this embodiment, the aforementioned container may be docker or singularity or other containers. Among them, docker is an open source application container engine, which allows developers to package their applications and dependencies into a portable image, and then publish them to machines of any operating system, which can also be virtualized; singularity is a An open source container platform, in particular, is one optimized for HPC workloads that allows untrusted users to run untrusted containers in a trusted manner. Further, when the running form of the target job indicates that the target job runs in the container, the feature collection component may determine that the target computing resource type is the containerized instance computing resource type.

It should be noted that determining whether the target computing resource type is a containerized instance computing resource type based on the container dependency information can accurately determine the relationship between the target computing resource type and the containerized instance computing resource type, so as to facilitate obtaining accurate data. Target compute resource type.

In an optional embodiment, when the job running form indicates that the target job is not running in the container, the feature collection component can detect whether the real-time maximum network bandwidth of the computing node is greater than the first preset bandwidth, and the corresponding Whether the average bandwidth is greater than the second preset bandwidth, and when the real-time maximum network bandwidth is greater than the first preset bandwidth, or, when the average bandwidth is greater than the second preset bandwidth, determine that the target computing resource type is a supercomputing cluster computing resource type, wherein , the computing resource type of supercomputing cluster indicates that the running of the target job is most affected by the network bandwidth.

Optionally, as shown in Figure 7, when the job running form indicates that the target job is not running in the container, the feature collection component can monitor the network bandwidth of the computing node corresponding to the target job, and detect the computing node corresponding to the target job. Whether the real-time maximum bandwidth (that is, the real-time maximum network bandwidth) of the network bottleneck of the node is greater than the first preset bandwidth and whether the average bandwidth (that is, the average bandwidth) of the network load pressure of the computing node corresponding to the target job is greater than the second preset bandwidth . When the real-time maximum network bandwidth of the aforementioned computing node is greater than the first preset bandwidth a (ie the threshold a in FIG. 7 ) or the average bandwidth is greater than the second preset bandwidth b (ie the threshold b in FIG. 7 ), the feature collection component It can be determined that the target computing resource type corresponding to the target job is the computing resource type of the supercomputing cluster.

It should be noted that whether the target computing resource type is a supercomputing cluster computing resource type is determined based on the real-time maximum network bandwidth and average bandwidth of the computing node corresponding to the target job, and the association between the target computing resource type and the supercomputing cluster computing resource type can be realized. Accurate determination of the relationship, so as to facilitate the acquisition of the accurate target computing resource type.

In an optional embodiment, when the real-time maximum network bandwidth is less than or equal to the first preset bandwidth, and the average bandwidth is less than or equal to the second preset bandwidth, the feature collection component can detect whether the memory occupancy rate of the computing node is not greater than the preset occupancy rate, and when the memory occupancy rate is greater than the preset occupancy rate, determine that the target computing resource type is a physical computer computing resource type, and when the memory occupancy rate is less than or equal to the preset occupancy rate, determine that the target computing resource type is virtual Computer computing resource type.

Optionally, as shown in FIG. 7 , when the real-time maximum network bandwidth is less than or equal to the first preset bandwidth, and the average bandwidth is less than or equal to the second preset bandwidth, the feature collection component can monitor the computing node corresponding to the target job. The CPU (central processing unit, central processing unit) occupancy rate, that is, the memory occupancy rate, is detected, and whether the CPU occupancy rate is greater than the preset occupancy rate is detected. Optionally, the feature collection component can detect whether the average CPU occupancy rate is greater than the preset occupancy rate, and can also detect whether the maximum CPU occupancy rate is greater than the preset occupancy rate.

Further, when the memory occupancy rate is greater than the preset occupancy rate, the feature collection component may determine that the target computing resource type is a physical computer computing resource type. Conversely, when the memory occupancy rate is less than or equal to the preset occupancy rate, the feature collection component may determine that the target computing resource type is a VM (Virtual Machine, virtual machine) computing resource type.

It should be noted that determining whether the target computing resource type is a physical computer computing resource type or a virtual machine computing resource type based on the memory occupancy rate corresponding to the target job can realize the comparison between the target computing resource type and the physical computer computing resource type and the virtual machine computing resource type. Accurate determination of the association relationship, so as to facilitate the acquisition of accurate target computing resource types.

It should be emphasized that the aforementioned judgment order for whether the target computing resource type is a containerized instance computing resource type, a supercomputing cluster computing resource type, a physical computer computing resource type, or a virtual machine computing resource type can be determined based on actual needs. changed.

In an optional embodiment, after comparing the job feature information of the target job with the target job feature information corresponding to the preset computing resource type, and determining the target computing resource type according to the comparison result, the feature collecting component The job detail information and system feature information corresponding to the target job can be stored in the preset feature database.

Optionally, as shown in FIG. 7 , the feature collection component can store the job detail information and system feature information corresponding to the target job and the association relationship between the target job and the target computing resource type in the preset feature database, so as to enrich the preset features. The data in the database improves the diversity of the data in the feature database, so that the next time the feature collection component obtains job feature information of a new target job, the corresponding target computing resource type can be obtained more quickly. The job detail information includes at least information such as size and content corresponding to the target job.

In an optional embodiment, in the process of allocating target computing resources to the target computing cluster, the resource allocation system may apply to the cloud server for target computing resources corresponding to the target computing resource type based on the resource management component, and then based on the resource management component The management component creates a task queue to the resource scheduling component, and adds target computing resources to the resource scheduling component based on the task queue, so as to allocate the target computing resources to the task queues corresponding to the computing nodes of the target computing cluster based on the scheduling policy corresponding to the resource scheduling component middle. The resource management component is used to manage the life cycle of the computing resources required by the target job, and the cloud server is used to perform production management of the computing resources required by the target job.

Specifically, as shown in FIGS. 6 and 7 , after the feature collection component determines the target computing resource type, the feature collection component sends a resource production request to the resource management component, and sends the target computing resource type to the resource management component. Optionally, the feature collection component may also send the target computing resource type to the resource management component after determining the target computing resource type and receiving the message of expansion of the target computing cluster. After that, the resource management component applies to the cloud server for the target computing resource corresponding to the target computing resource type, wherein the cloud server at least stores the computing resource corresponding to the aforementioned containerized instance computing resource type and the supercomputing cluster computing resource type. The corresponding computing resources, the computing resources corresponding to the physical computer computing resource types, and the computing resources corresponding to the virtual machine computing resource types.

Further, as shown in FIG. 6 , after the target computing resource application is successful, the resource management component can create a task queue to the resource scheduling component, and add target computing resources to the resource scheduling component based on the task queue, while maintaining all computing resources in the target computing cluster. The life cycle of the resource. Among them, at least before adding target computing resources to the resource scheduling component based on the task queue, the resource management component also needs to create a target computing cluster, that is, adding computing nodes to the resource scheduling component to ensure that jobs can be successfully run.

Further, the resource scheduling component can allocate target computing resources and target jobs to the task queues corresponding to the computing nodes of the target computing cluster based on its own scheduling strategy, so that the target computing cluster can run the corresponding target based on each target computing resource. Operation.

It should be noted that the task queue is created to the resource scheduling component based on the resource management component, and the corresponding task queue is allocated to the computing nodes of the target computing cluster based on the resource scheduling component, so that the target computing cluster can be based on each target computing resource. The target job is processed, thereby avoiding the phenomenon of resource waste or shortage of selecting the same computing resources for all jobs, and improving the rationality of resource allocation.

In an optional embodiment, after the target computing resource is allocated to the target computing cluster, and after the resource management component detects that the resource scheduling component completes the target job, the resource management component can remove the target computing resource from the task queue, And send a resource release message to the cloud server to release the target computing resources.

Optionally, as shown in FIG. 6 , when the execution of any target job in the resource scheduling component is completed, the resource scheduling component may send the execution completion information to the resource management component, or may be obtained by the resource management component from the resource scheduling component. Relevant information for the resource management component to determine that the resource scheduling component completes the target job. After that, the resource management component can remove the target computing resource corresponding to the completed target job from the task queue, and send a resource release message to the cloud server to release the target computing resource.

It should be noted that, by releasing the target computing resources corresponding to the target job after the target job is completed, it can avoid the situation that the target computing resource cannot be used successfully next time when the target computing resource needs to be used, thereby improving the utilization of each computing resource. utilization.

In an optional embodiment, after determining the target computing resource type based on the job feature information, the resource management component may create multiple task queues according to the job feature information of the multiple target jobs when the number of target jobs is multiple , and determine the target computing resources corresponding to each target job according to the computing resource types corresponding to the multiple task queues, so as to execute multiple target jobs based on the target computing resources. Wherein, each task queue corresponds to computing resources of different computing resource types.

Specifically, when there are multiple target jobs, the resource management component can create multiple task queues based on the categories of job feature information of the multiple target jobs. The lower job feature information is regarded as a different class, and the number of task queues is equal to the number of categories of the job feature information. Among them, each task queue corresponds to computing resources of different computing resource types, so that the target computing resources corresponding to each target job can be determined based on the computing resource types corresponding to the multiple task queues, and the corresponding target jobs can be executed based on the target computing resources. . Optionally, the resource management component can determine job feature information of multiple target jobs based on the aforementioned method, or can roughly determine the industry nature and characteristics of each target job based on testing and experience, and use it as job feature information.

It should be noted that by creating multiple task queues according to the job feature information of multiple target jobs, and determining target computing resources based on the multiple task queues, the work efficiency of the application can be improved when the number of target jobs is multiple. The running efficiency of each target job.

It should be noted that this application fully considers expansion or flexibility requirements, makes resource decisions based on job feature information of the target job, and automatically releases computing resources after the target job is executed, avoiding that statically distributed computing resources in the prior art cannot reflect cloud computing. The timeliness of the operation, there is a waste or shortage of resources, and the phenomenon that the operation cannot meet the needs of flexibility. At the same time, the decision of computing resource type based on job feature information can avoid differences caused by manual selection of computing resources, thereby improving user experience.

It can be seen that the solution provided by the present application achieves the purpose of allocating different types of computing resources to each job based on the job feature information of the job, thereby achieving the technical effect of improving the rationality of resource allocation, and further solving the problems caused by the prior art. The technical problem of unreasonable resource allocation caused by selecting the same computing resources for all jobs.

It should be noted that, for the sake of simple description, the foregoing method embodiments are all expressed as a series of action combinations, but those skilled in the art should know that the present application is not limited by the described action sequence. Because in accordance with the present application, certain steps may be performed in other orders or concurrently. Secondly, those skilled in the art should also know that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present application.

From the description of the above embodiments, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus a necessary general hardware platform, and of course can also be implemented by hardware, but in many cases the former is better implementation. Based on this understanding, the technical solution of the present application can be embodied in the form of a software product in essence or in a part that contributes to the prior art, and the computer software product is stored in a storage medium (such as ROM/RAM, magnetic disk, CD-ROM), including several instructions to make a terminal device (which may be a mobile phone, a computer, a server, or a network device, etc.) execute the methods of the various embodiments of the present application.

Example 2

Under the above operating environment, the present application provides a resource allocation method as shown in FIG. 8 . FIG. 8 is a flowchart of an optional resource allocation method according to an embodiment of the present application.

Step S302, in response to the job creation instruction, create a target job and a target computing cluster for executing the target job, and display job feature information of the target job.

In step S302, as shown in FIG. 9, the aforementioned resource allocation system can create a target job in response to the job creation instruction input by the user, and then the target job can be obtained by the resource scheduling component, and obtained from the resource scheduling component by the feature collection component target job to collect job characteristic information of the target job. At the same time, the resource management component in the resource allocation system can also create a target computing cluster for executing the target job.

Further, after the feature collection component completes the collection of job feature information of the target job, the resource allocation system can display the job feature information of the target job to the background operator or user through the display interface for viewing by the background operator or user.

It should be noted that, by responding to the user's job creation instruction, a target job and a target computing cluster are created, so as to ensure normal feedback of the user's operation. At the same time, by displaying the job feature information, the background operators can better supervise the resource allocation system, preventing the target job execution error caused by wrong instructions or wrong job feature information, thereby improving the stability of target job execution. . The erroneous job feature information may be blank job feature information or the like.

Step S304, displaying the target computing resource type determined based on the job feature information.

In step S304, as shown in FIG. 9, after the feature collection component determines the target computing resource type based on the aforementioned method, the resource allocation system can display each target job and its corresponding target computing resource type to the background operation through the display interface person or user.

It should be noted that by displaying the target computing resource type, it can further facilitate the background operator to supervise the resource allocation system, and prevent the wrong pairing of the target job and the target computing resource type, or the error caused by the wrong target computing resource type. The target job is executed incorrectly, thereby improving the stability of the target job execution. The wrong target computing resource type may be a blank computing resource type or a computing resource type that does not exist in a storage area such as a cloud server and a database.

Step S306, responding to the resource allocation instruction, displaying the target computing resources allocated by the target computing cluster, wherein the target computing resources are the computing resources required by the target computing cluster to perform the target job, and the computing resource type of the target computing resources is the target computing resource type .

In step S306, as shown in FIG. 9, the resource allocation system may respond to the resource allocation instruction input by the background operator or the user, allocate target computing resources to the target computing cluster, and display the target computing resources allocated to the target computing cluster (ie display assignment results in Figure 9).

It should be noted that by displaying the target computing resources allocated to the target computing cluster, it can further facilitate the background operators to supervise the resource allocation system, and prevent the resource allocation system from allocating the same type of target computing resources to all target jobs or multiple target jobs. The phenomenon that multiple target computing resources are allocated to the same target job occurs, thereby improving the stability of target job execution.

It should be noted that, for the sake of simple description, the foregoing method embodiments are all expressed as a series of action combinations, but those skilled in the art should know that the present application is not limited by the described action sequence. Because in accordance with the present application, certain steps may be performed in other orders or concurrently. Secondly, those skilled in the art should also know that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present application.

From the description of the above embodiments, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus a necessary general hardware platform, and of course can also be implemented by hardware, but in many cases the former is better implementation. Based on this understanding, the technical solution of the present application can be embodied in the form of a software product in essence or in a part that contributes to the prior art, and the computer software product is stored in a storage medium (such as ROM/RAM, magnetic disk, CD-ROM), including several instructions to make a terminal device (which may be a mobile phone, a computer, a server, or a network device, etc.) execute the methods of the various embodiments of the present application.

Example 3

According to an embodiment of the present invention, a resource allocation apparatus for implementing the above resource allocation method is also provided. As shown in FIG. 10 , the apparatus includes:

an obtaining module 402, configured to obtain job feature information of a target job to be executed by the target computing cluster;

a determining module 404, configured to determine the target computing resource type based on the job feature information;

The allocation module 406 is configured to allocate target computing resources to the target computing cluster, wherein the target computing resources are computing resources required by the target computing cluster to execute the target job, and the computing resource type of the target computing resources is the target computing resource type.

It should be noted here that the above-mentioned acquisition module 402, determination module 404 and allocation module 406 correspond to steps S202 to S206 in Embodiment 1, and the three modules have the same instances and application scenarios as the corresponding steps. It is limited to the content disclosed in the above-mentioned Embodiment 1. It should be noted that, as a part of the apparatus, the above-mentioned modules may run in the electronic device 10 provided in Embodiment 1.

Optionally, the acquisition module further includes: a sub-acquisition module, configured to acquire job information of the target job from the job list corresponding to the resource scheduling component, wherein the resource scheduling component is used to receive the target job submitted by the target object, and assign the target job to the target job. The job is allocated to the computing nodes of the target computing cluster; the first sub-determination module is used to determine the system characteristic information corresponding to the target job based on the target computing cluster, wherein the job characteristic information at least includes job information and system characteristic information, and the job information at least It includes: container dependency information when the target job is running, and the system feature information at least includes: network bandwidth and memory usage of computing nodes.

Optionally, the determining module further includes: a query module, configured to query the computing resource type from a preset feature database based on the job feature information, and obtain a query result, wherein the preset feature database stores at least job feature information of historical jobs. , computing resource type, and the relationship between job feature information of historical jobs and computing resource type, where the computing resource type is the type of computing resources required by the target computing cluster to execute historical jobs; the second sub-determination module is used for querying The result indicates that when there is a computing resource type corresponding to the job feature information of the target job in the preset feature database, the computing resource type corresponding to the job feature information of the target job in the preset feature database is determined as the target computing resource type; first The processing module is configured to, when the query result indicates that there is no computing resource type corresponding to the job feature information of the target job in the preset feature database, the target job corresponding to the job feature information of the target job and the preset computing resource type The feature information is compared, and the target computing resource type is determined according to the comparison result.

Optionally, the first processing module further includes: a third sub-determination module for determining the job running form of the target job according to the container dependency information when the target job is running; a fourth sub-determining module for characterizing the target in the job running form When the job runs in the container, it is determined that the target computing resource type is the containerized instance computing resource type, wherein the containerized instance computing resource type indicates that the target job depends on the container when running.

Optionally, the resource allocation device further includes: a second processing module, configured to detect whether the real-time maximum network bandwidth of the computing node is greater than the first preset bandwidth when the job running form indicates that the target job is not running in the container, and calculate Whether the average bandwidth corresponding to the node is greater than the second preset bandwidth; the fifth sub-determination module, when the real-time maximum network bandwidth is greater than the first preset bandwidth, or when the average bandwidth is greater than the second preset bandwidth, determine that the target computing resource type is The computing resource type of the supercomputing cluster, wherein the computing resource type of the supercomputing cluster indicates that the running of the target job is most affected by the network bandwidth.

Optionally, the resource allocation device further includes: a third processing module, configured to detect the memory of the computing node when the real-time maximum network bandwidth is less than or equal to the first preset bandwidth, and the average bandwidth is less than or equal to the second preset bandwidth. Whether the occupancy rate is greater than the preset occupancy rate; the sixth sub-determination module is used to determine that the target computing resource type is the physical computer computing resource type when the memory occupancy rate is greater than the preset occupancy rate; the seventh sub-determination module is used for memory When the occupancy rate is less than or equal to the preset occupancy rate, it is determined that the target computing resource type is the virtual machine computing resource type.

Optionally, the resource allocation apparatus further includes: a storage module, configured to store job detail information and system feature information corresponding to the target job in a preset feature database.

Optionally, the allocation module further includes: a fourth processing module, configured to apply to the cloud server for target computing resources corresponding to the target computing resource type based on the resource management component, wherein the resource management component is used for computing required by the target job. The life cycle of the resource is managed, and the cloud server is used for production management of the computing resources required by the target job; the fifth processing module is used to create a task queue to the resource scheduling component based on the resource management component, and send the resource scheduling component to the resource scheduling component based on the task queue. A target computing resource is added; a sub-allocation module is used to allocate the target computing resource to the task queue corresponding to the computing node of the target computing cluster based on the scheduling policy corresponding to the resource scheduling component.

Optionally, the resource allocation device further includes: a sixth processing module for removing the target computing resource from the task queue after the resource management component detects that the resource scheduling component completes the target job, and sends a resource release message to the cloud server, to free up target computing resources.

Optionally, the resource allocation device further includes: a creation module for creating multiple task queues according to job feature information of multiple target jobs when the number of target jobs is multiple, wherein each task queue corresponds to different computing resources. type of computing resources; the eighth sub-determination module is used to determine the target computing resources corresponding to each target job according to the computing resource types corresponding to the multiple task queues; the seventh processing module is used to execute multiple tasks based on the target computing resources target job.

Example 4

According to an embodiment of the present invention, a resource allocation apparatus for implementing the above resource allocation method is also provided. As shown in FIG. 11 , the apparatus includes:

a first response module 502, configured to respond to the job creation instruction, create a target job and a target computing cluster for executing the target job, and display job feature information of the target job;

A display module 504, configured to display the target computing resource type determined based on the job feature information;

The second response module 506 is configured to respond to the resource allocation instruction and display the target computing resources allocated to the target computing cluster, wherein the target computing resources are the computing resources required by the target computing cluster to execute the target job, and the computing resource type of the target computing resources Compute resource type for the target.

It should be noted here that the above-mentioned first response module 502, display module 504 and second response module 506 correspond to steps S302 to S306 in Embodiment 2, and examples and application scenarios implemented by the three modules and corresponding steps The same, but not limited to the content disclosed in Example 2 above. It should be noted that, as a part of the apparatus, the above-mentioned modules may run in the electronic device 10 provided in the first embodiment.

Example 5

According to an embodiment of the present invention, a resource allocation system for implementing the above resource allocation method is also provided. As shown in FIG. 12 , the system includes:

A scheduler for receiving target jobs submitted by target objects.

Optionally, the scheduler is responsible for receiving the target job submitted by the user, and according to different scheduling policies, assigns the target job to the specific computing resources of the target computing cluster for calculation, and monitors the running status of each target job to obtain each target job. The calculation execution result corresponding to the target job. Among them, the target job is the computing task of the target computing cluster. Different target jobs will be configured with different resource requirements, priorities, execution time and other parameters. The scheduler can use different scheduling strategies for the configuration parameters of different target jobs. Make the target computing cluster perform the target job better. When there are many target jobs corresponding to computing resources, the scheduler can queue and manage the target jobs. The scheduler can also monitor the calculation execution result of the target job, and resubmit for execution the target job whose calculation execution result indicates that the execution fails.

The feature collection component is used to obtain job feature information of the target job to be executed by the target computing cluster, and determine the target computing resource type based on the job feature information.

Optionally, the feature collection component is connected to the scheduler, and the feature collection component can access the scheduler and the target computing cluster on the target computing cluster side to collect container dependency information when the target job is running, and system feature information such as network bandwidth and memory usage. , according to a certain policy, determine the type of computing resource corresponding to the application, and send a resource application message to the resource management component. It should be noted that the feature collection component is different from the traditional performance collection tool. It can be combined with the cloud resource management component, scheduler and other units to jointly complete resource elasticity and management. In addition, the feature collection component can collect job feature information, give full play to the advantages of data on the cloud, and finally fully predict the type of computing resources required for computing resources.

The resource management component is used to determine the target computing resource according to the target computing resource type, create a task queue for the scheduler, and then add the target computing resource to the scheduler based on the task queue.

Optionally, the resource management component is connected to the feature collection component and the scheduler, and the resource management component is used to manage the life cycle of the computing resources required by the target job. Specifically, the resource management component is used for the computing resources of the entire target computing cluster. Life cycle management such as creation, deletion, addition and withdrawal of computing resources. The resource management component can apply to the cloud server for the target computing resources determined by it, and interact with the scheduler to notify the scheduler of the addition and withdrawal of computing resources, so that the scheduler can update its own scheduling policy, when the corresponding computing resources in the scheduler After the use is completed, the resource manager can also release the computing resources to the cloud server. Different from the composition of traditional related systems, since instances on the cloud can be flexibly applied for and released, the system of this application includes this part of specialized resource management components for life cycle management of cluster computing resources. The resource management component may be an independently deployed system, or may be deployed together with the scheduler or as an internal component of the scheduler.

The scheduler also allocates the target computing resources to the task queues corresponding to the computing nodes of the target computing cluster based on the preset scheduling policy, so that the computing nodes in the target computing cluster execute the target job.

Optionally, the scheduler is further configured to allocate the target computing resources to the task queue corresponding to the relevant server in the target computing cluster after applying for the corresponding target computing resource, so that the relevant server can execute the corresponding target job.

In an optional embodiment, the resource allocation system may further include a cloud server, the cloud server is connected to the resource management component, and is used for production management of the computing resources required by the target job, wherein the cloud server stores at least containers The computing resources corresponding to the computing resource types of the instance, the computing resources corresponding to the computing resource types of the supercomputing cluster, the computing resources corresponding to the computing resource types of the physical machines, and the computing resources corresponding to the computing resource types of the virtual machines.

It should be noted here that the above-mentioned scheduler, feature collection component, and resource management component can be used to implement the method provided in Embodiment 1, and examples and application scenarios and embodiments implemented by the scheduler, feature collection component, and resource management component. The examples and application scenarios implemented in 1 are the same, but are not limited to the content disclosed in the above-mentioned Embodiment 1.

Example 6

Embodiments of the present application may provide an electronic device, and the electronic device may be any electronic device in an electronic device group. Optionally, in this embodiment, the above electronic device may also be replaced by a terminal device such as a mobile terminal.

Optionally, in this embodiment, the above-mentioned electronic device may be located in at least one network device among multiple network devices of a computer network.

In this embodiment, the above-mentioned electronic device can execute the program code of the following steps in the resource allocation method: acquiring job feature information of a target job to be executed by the target computing cluster; determining the target computing resource type based on the job feature information; allocating to the target computing cluster Target computing resources, where the target computing resources are computing resources required by the target computing cluster to execute the target job, and the computing resource type of the target computing resources is the target computing resource type.

Optionally, FIG. 13 is a structural block diagram of an electronic device according to an embodiment of the present application. As shown in FIG. 13 , the electronic device 10 may include: one or more (only one is shown in the figure) a processor 102 , a memory 104 and a storage controller.

The memory can be used to store software programs and modules, such as program instructions/modules corresponding to the resource allocation method and device in the embodiments of the present application, and the processor executes various functional applications by running the software programs and modules stored in the memory. and data processing, that is, to implement the above-mentioned resource allocation method. The memory may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory may further include memory located remotely from the processor, and these remote memories may be connected to the terminal 10 through a network. Examples of such networks include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

The processor can call the information and application programs stored in the memory through the transmission device to perform the following steps: obtain job feature information of the target job to be executed by the target computing cluster; determine the target computing resource type based on the job feature information; allocate the target computing cluster to the target computing cluster. Target computing resources, where the target computing resources are computing resources required by the target computing cluster to execute the target job, and the computing resource type of the target computing resources is the target computing resource type.

Optionally, the above-mentioned processor may also execute the program code of the following steps: obtaining job information of the target job from the job list corresponding to the resource scheduling component, wherein the resource scheduling component is used for receiving the target job submitted by the target object, and assigning the job information to the target job. The target job is allocated to the computing nodes of the target computing cluster; the system characteristic information corresponding to the target job is determined based on the target computing cluster, wherein the job characteristic information at least includes job information and system characteristic information, and the job information at least includes: Container dependency information, and system feature information at least includes: network bandwidth and memory usage of computing nodes.

Optionally, the above-mentioned processor may also execute the program code of the following steps: based on the job feature information, perform a computing resource type query from a preset feature database to obtain a query result, wherein the preset feature database stores at least the jobs of the historical jobs. Feature information, computing resource types, and the relationship between job feature information of historical jobs and computing resource types, where the computing resource type is the type of computing resources required by the target computing cluster to execute historical jobs; the query results represent the preset feature database When there is a computing resource type corresponding to the job feature information of the target job, determine the computing resource type corresponding to the job feature information of the target job in the preset feature database as the target computing resource type; When there is no computing resource type corresponding to the job feature information of the target job, the job feature information of the target job is compared with the target job feature information corresponding to the preset computing resource type, and the target computing resource is determined according to the comparison result. Resource Type.

Optionally, the above-mentioned processor may further execute the program code of the following steps: determining the job running form of the target job according to the container dependency information when the target job runs; determining the target computing resource when the job running form represents that the target job is running in the container. The type is the containerized instance computing resource type, where the containerized instance computing resource type indicates that the target job depends on the container when running.

Optionally, the above-mentioned processor may also execute the program code of the following steps: when the job running form indicates that the target job is not running in the container, detect whether the real-time maximum network bandwidth of the computing node is greater than the first preset bandwidth, and the computing node Whether the corresponding average bandwidth is greater than the second preset bandwidth; when the real-time maximum network bandwidth is greater than the first preset bandwidth, or, when the average bandwidth is greater than the second preset bandwidth, determine that the target computing resource type is the supercomputing cluster computing resource type, Among them, the supercomputing cluster computing resource type indicates that the running of the target job is most affected by the network bandwidth.

Optionally, the above-mentioned processor may also execute the program code of the following steps: when the real-time maximum network bandwidth is less than or equal to the first preset bandwidth, and the average bandwidth is less than or equal to the second preset bandwidth, detect the memory occupation of the computing node. Whether the memory occupancy rate is greater than the preset occupancy rate; when the memory occupancy rate is greater than the preset occupancy rate, the target computing resource type is determined as the physical computer computing resource type; when the memory occupancy rate is less than or equal to the preset occupancy rate, the target computing resource type is determined as The virtual machine computing resource type.

Optionally, the above-mentioned processor may also execute the program code of the following steps: comparing the job feature information of the target job with the target job feature information corresponding to the preset computing resource type, and determining the target computing resource according to the comparison result. After the type, the job detail information and system feature information corresponding to the target job are stored in the preset feature database.

Optionally, the above-mentioned processor may also execute the program code of the following steps: apply to the cloud server for the target computing resource corresponding to the target computing resource type based on the resource management component, wherein the resource management component is used for computing required for the target job. The life cycle of resources is managed, and the cloud server is used for production management of computing resources required by target jobs; task queues are created to resource scheduling components based on resource management components, and target computing resources are added to resource scheduling components based on task queues; resource-based The scheduling policy corresponding to the scheduling component allocates the target computing resources to the task queues corresponding to the computing nodes of the target computing cluster.

Optionally, the above processor may also execute the program code of the following steps: after allocating the target computing resource to the target computing cluster, after the resource management component detects that the resource scheduling component completes the target job, remove the target computing resource from the task queue. , and send a resource release message to the cloud server to release the target computing resources.

Optionally, the above-mentioned processor may also execute the program code of the following steps: after determining the target computing resource type based on the job feature information, when the number of target jobs is multiple, create multiple target jobs according to the job feature information of the multiple target jobs. A task queue, wherein each task queue corresponds to computing resources of different computing resource types; the target computing resources corresponding to each target job are determined according to the computing resource types corresponding to the multiple task queues; multiple target jobs are executed based on the target computing resources .

Optionally, the above-mentioned processor may also execute the program code of the following steps: in response to the job creation instruction, create a target job and a target computing cluster for executing the target job, and display job feature information of the target job; display the job feature information determined based on the job feature information. Target computing resource type; in response to the resource allocation instruction, display the target computing resources allocated to the target computing cluster, where the target computing resources are the computing resources required by the target computing cluster to execute the target job, and the computing resource type of the target computing resources is the target computing resource. Resource Type.

Using the embodiment of the present application, an electronic device for implementing the resource allocation method in the foregoing embodiment is provided. By assigning different types of computing resources to each job based on the job feature information of the job, the purpose of improving the rationality of resource allocation is achieved, and the problem of unreasonable resource allocation caused by selecting the same computing resources for all jobs in the prior art is solved. technical problem.

Those of ordinary skill in the art can understand that the structure shown in FIG. 13 is for illustration only, and the electronic device can also be a smart phone (such as an Android mobile phone, an iOS mobile phone, etc.), a tablet computer, an applause computer, and a Mobile Internet Devices (MID) , PAD and other terminal equipment. FIG. 13 does not limit the structure of the above electronic device. For example, the electronic device 10 may also include more or fewer components than those shown in FIG. 13 (eg, network interfaces, display devices, etc.), or have a different configuration than that shown in FIG. 13 .

Those of ordinary skill in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructing the hardware related to the terminal device through a program, and the program can be stored in a computer-readable storage medium, and the storage medium can Including: flash disk, read-only memory (Read-Only Memory, ROM), random access device (RandomAccess Memory, RAM), magnetic disk or optical disk, etc.

Example 7

Embodiments of the present application also provide a computer-readable storage medium. Optionally, in this embodiment, the above-mentioned computer-readable storage medium may be used to store program codes executed by the resource allocation method provided in the above-mentioned first embodiment.

Optionally, in this embodiment, the computer-readable storage medium may be located in any electronic device in an electronic device group in a computer network, or in any mobile terminal in a mobile terminal group.

Optionally, in this embodiment, the storage medium is configured to store program codes for executing the following steps: acquiring job feature information of a target job to be executed by the target computing cluster; determining the target computing resource type based on the job feature information; The target computing cluster allocates target computing resources, wherein the target computing resources are computing resources required by the target computing cluster to execute the target job, and the computing resource type of the target computing resources is the target computing resource type.

Optionally, the above-mentioned computer-readable storage medium may also store program code for performing the following steps: obtaining job information of the target job from a job list corresponding to the resource scheduling component, wherein the resource scheduling component is used to receive the target object submission. and assign the target job to the computing nodes of the target computing cluster; determine the system feature information corresponding to the target job based on the target computing cluster, wherein the job feature information at least includes job information and system feature information, and the job information at least includes : Container dependency information when the target job is running. The system feature information includes at least: network bandwidth and memory usage of the computing node.

Optionally, the above-mentioned computer-readable storage medium may also store program codes for performing the following steps: querying the computing resource type from a preset feature database based on job feature information, and obtaining a query result, wherein at least one of the preset feature database Stores job feature information and computing resource types of historical jobs, as well as the relationship between job feature information of historical jobs and computing resource types. The computing resource type is the type of computing resources required by the target computing cluster to execute historical jobs; The result indicates that when there is a computing resource type corresponding to the job feature information of the target job in the preset feature database, it is determined that the computing resource type corresponding to the job feature information of the target job in the preset feature database is the target computing resource type; The result indicates that when there is no computing resource type corresponding to the job feature information of the target job in the preset feature database, the job feature information of the target job is compared with the target job feature information corresponding to the preset computing resource type. The comparison results determine the target computing resource type.

Optionally, the above-mentioned computer-readable storage medium may further store program codes for executing the following steps: determining the job running form of the target job according to the container dependency information when the target job is running; indicating that the target job runs in the container in the job running form. When , the target computing resource type is determined to be the containerized instance computing resource type, wherein the containerized instance computing resource type indicates that the runtime of the target job depends on the container.

Optionally, the above-mentioned computer-readable storage medium may also store program code for executing the following steps: when the job running form indicates that the target job is not running in the container, detecting whether the real-time maximum network bandwidth of the computing node is greater than a first preset bandwidth, and whether the average bandwidth corresponding to the computing node is greater than the second preset bandwidth; when the real-time maximum network bandwidth is greater than the first preset bandwidth, or when the average bandwidth is greater than the second preset bandwidth, determine that the target computing resource type is super The computing resource type of the computing cluster, wherein the computing resource type of the supercomputing cluster indicates that the running of the target job is most affected by the network bandwidth.

Optionally, the above-mentioned computer-readable storage medium can also store program codes for executing the following steps: when the real-time maximum network bandwidth is less than or equal to the first preset bandwidth, and when the average bandwidth is less than or equal to the second preset bandwidth, Detect whether the memory occupancy rate of the computing node is greater than the preset occupancy rate; when the memory occupancy rate is greater than the preset occupancy rate, determine that the target computing resource type is the physical computer computing resource type; when the memory occupancy rate is less than or equal to the preset occupancy rate, Determine the target computing resource type as the virtual machine computing resource type.

Optionally, the above-mentioned computer-readable storage medium may also store program codes for performing the following steps: when comparing the job feature information of the target job with the target job feature information corresponding to the preset computing resource type, according to the comparison After determining the target computing resource type from the result, the job detail information and system feature information corresponding to the target job are stored in a preset feature database.

Optionally, the above-mentioned computer-readable storage medium may also store program codes for performing the following steps: applying to the cloud server for a target computing resource corresponding to the target computing resource type based on the resource management component, wherein the resource management component is used for The life cycle of the computing resources required by the target job is managed, and the cloud server is used for production management of the computing resources required by the target job; a task queue is created to the resource scheduling component based on the resource management component, and added to the resource scheduling component based on the task queue. Target computing resources; the target computing resources are allocated to the task queues corresponding to the computing nodes of the target computing cluster based on the scheduling policy corresponding to the resource scheduling component.

Optionally, the above-mentioned computer-readable storage medium can also store program code for executing the following steps: after the target computing resource is allocated to the target computing cluster, after the resource management component detects that the resource scheduling component completes the target job, from the task queue Remove the target computing resources from the server, and send a resource release message to the cloud server to release the target computing resources.

Optionally, the above-mentioned computer-readable storage medium may also store program codes for performing the following steps: after determining the target computing resource type based on the job feature information, when the number of target jobs is multiple, according to the number of target jobs. Job feature information creates multiple task queues, wherein each task queue corresponds to computing resources of different computing resource types; the target computing resources corresponding to each target job are determined according to the computing resource types corresponding to the multiple task queues; The resource executes multiple target jobs.

Optionally, the above-mentioned computer-readable storage medium may also store program codes for executing the following steps: in response to a job creation instruction, creating a target job and a target computing cluster for executing the target job, and displaying job feature information of the target job; The target computing resource type determined by the job feature information; in response to the resource allocation instruction, display the target computing resources allocated to the target computing cluster, where the target computing resources are the computing resources required by the target computing cluster to execute the target job, and the target computing resources are The computing resource type is the target computing resource type.

The above-mentioned serial numbers of the embodiments of the present application are only for description, and do not represent the advantages or disadvantages of the embodiments.

In the above-mentioned embodiments of the present application, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference may be made to related descriptions of other embodiments.

In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. The apparatus embodiments described above are only illustrative, for example, the division of the units is only a logical function division, and there may be other division methods in actual implementation, for example, multiple units or components may be combined or Integration into another system, or some features can be ignored, or not implemented. On the other hand, the shown or discussed mutual coupling or direct coupling or communication connection may be through some interfaces, indirect coupling or communication connection of units or modules, and may be in electrical or other forms.

The units described as separate components may or may not be physically separated, and components displayed as units may or may not be physical units, that is, may be located in one place, or may be distributed to multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution in this embodiment.

In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically alone, or two or more units may be integrated into one unit. The above-mentioned integrated units may be implemented in the form of hardware, or may be implemented in the form of software functional units.

The integrated unit, if implemented in the form of a software functional unit and sold or used as an independent product, may be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of the present application can be embodied in the form of software products in essence, or the parts that contribute to the prior art, or all or part of the technical solutions, and the computer software products are stored in a storage medium , including several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present application. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, magnetic disk or optical disk and other media that can store program codes .

The above are only the preferred embodiments of the present application. It should be pointed out that for those skilled in the art, without departing from the principles of the present application, several improvements and modifications can also be made. It should be regarded as the protection scope of this application.

Claims (14)

1. A method for resource allocation, comprising:

acquiring operation characteristic information of a target operation to be executed by a target computing cluster;

determining a target computing resource type based on the job characterization information;

allocating target computing resources to the target computing cluster, wherein the target computing resources are computing resources required by the target computing cluster to execute the target job, and the computing resource type of the target computing resources is the target computing resource type.

2. The method of claim 1, wherein obtaining job characteristics information for a target job to be executed by a target compute cluster comprises:

acquiring the job information of the target job from a job list corresponding to a resource scheduling component, wherein the resource scheduling component is used for receiving the target job submitted by a target object and distributing the target job to the computing nodes of the target computing cluster;

determining system characteristic information corresponding to the target job based on the target computing cluster, wherein the job characteristic information at least comprises the job information and the system characteristic information, and the job information at least comprises: the container dependency information of the target operation runtime, the system characteristic information at least includes: and calculating the network bandwidth and the memory occupancy rate of the node.

3. The method of claim 2, wherein determining a target computing resource type based on the job characterization information comprises:

performing computing resource type query from a preset feature database based on the job feature information to obtain a query result, wherein the preset feature database at least stores job feature information and computing resource types of historical jobs and an incidence relation between the job feature information of the historical jobs and the computing resource types, and the computing resource types are types of computing resources required by the target computing cluster to execute the historical jobs;

when the query result represents that the computing resource type corresponding to the operation feature information of the target operation exists in the preset feature database, determining that the computing resource type corresponding to the operation feature information of the target operation in the preset feature database is the target computing resource type;

and when the query result indicates that the computing resource type corresponding to the operation characteristic information of the target operation does not exist in the preset characteristic database, comparing the operation characteristic information of the target operation with target operation characteristic information corresponding to a preset computing resource type, and determining the target computing resource type according to the comparison result.

4. The method according to claim 3, wherein comparing the job feature information of the target job with target job feature information corresponding to a preset computing resource type, and determining the target computing resource type according to the comparison result comprises:

determining the operation running form of the target operation according to the container dependence information of the target operation running;

when the job run form characterizes that the target job runs in a container, determining that the target computing resource type is a containerized instance computing resource type, wherein the containerized instance computing resource type characterizes that the target job runs in dependence on the container.

5. The method of claim 4, further comprising:

when the operation running form represents that the target operation does not run in the container, detecting whether the real-time maximum network bandwidth of the computing node is larger than a first preset bandwidth or not, and whether the average bandwidth corresponding to the computing node is larger than a second preset bandwidth or not;

when the real-time maximum network bandwidth is larger than the first preset bandwidth, or the average bandwidth is larger than the second preset bandwidth, determining that the target computing resource type is a super computing cluster computing resource type, wherein the super computing cluster computing resource type represents that the running of the target job is influenced by the network bandwidth to the maximum extent.

6. The method of claim 5, further comprising:

when the real-time maximum network bandwidth is smaller than or equal to the first preset bandwidth and the average bandwidth is smaller than or equal to the second preset bandwidth, detecting whether the memory occupancy rate of the computing node is larger than a preset occupancy rate;

when the memory occupancy rate is greater than the preset occupancy rate, determining that the target computing resource type is a physical machine computing resource type;

and when the memory occupancy rate is less than or equal to the preset occupancy rate, determining that the target computing resource type is the virtual machine computing resource type.

7. The method according to any one of claims 4 to 6, wherein after comparing the job characteristic information of the target job with target job characteristic information corresponding to a preset computing resource type and determining the target computing resource type according to the comparison result, the method further comprises:

and storing the operation detail information corresponding to the target operation and the system characteristic information into the preset characteristic database.

8. The method of claim 1, wherein allocating target computing resources for the target computing cluster comprises:

applying for a target computing resource corresponding to the target computing resource type to a cloud server based on a resource management component, wherein the resource management component is used for managing the life cycle of the computing resource required by the target job, and the cloud server is used for performing production management on the computing resource required by the target job;

creating a task queue to a resource scheduling component based on the resource management component, and adding the target computing resource to the resource scheduling component based on the task queue;

and allocating the target computing resource to a task queue corresponding to a computing node of the target computing cluster based on a scheduling policy corresponding to the resource scheduling component.

9. The method of claim 8, wherein after allocating target computing resources for the target computing cluster, the method further comprises:

after the resource management component detects that the resource scheduling component completes the target job, the target computing resource is removed from the task queue, and a resource release message is sent to the cloud server to release the target computing resource.

10. The method of claim 1, wherein after determining a target computing resource type based on the job characterization information, the method further comprises:

when the number of the target jobs is multiple, creating a plurality of task queues according to job characteristic information of the target jobs, wherein each task queue corresponds to computing resources of different computing resource types;

determining a target computing resource corresponding to each target operation according to the computing resource types corresponding to the plurality of task queues;

executing a plurality of the target jobs based on the target computing resources.

11. A method for resource allocation, comprising:

responding to a job creating instruction, creating a target job, executing a target computing cluster of the target job, and displaying job characteristic information of the target job;

displaying a target computing resource type determined based on the job characteristic information;

and responding to a resource allocation instruction, and displaying target computing resources allocated to the target computing cluster, wherein the target computing resources are computing resources required by the target computing cluster to execute the target job, and the computing resource type of the target computing resources is the target computing resource type.

12. A resource allocation system, comprising:

the scheduler is used for receiving a target job submitted by a target object;

the characteristic collection component is used for acquiring the job characteristic information of a target job to be executed by the target computing cluster and determining the type of the target computing resource based on the job characteristic information;

the resource management component is used for determining target computing resources according to the types of the target computing resources, creating a task queue for the scheduler, and then adding the target computing resources to the scheduler based on the task queue;

the scheduler further allocates the target computing resource to a task queue corresponding to a computing node of the target computing cluster based on a preset scheduling policy, so that the computing node in the target computing cluster executes the target job.

13. A computer-readable storage medium, in which a computer program is stored, wherein the computer program is arranged to perform the resource allocation method of any one of claims 1 to 11 when executed.

14. An electronic device, wherein the electronic device comprises one or more processors; memory for storing one or more programs which, when executed by the one or more processors, cause the one or more processors to implement a method for running a program, wherein the program is arranged to, when running, perform the resource allocation method of any of claims 1 to 11.

Images