CN116841731A - An FPGA virtualized resource scheduling system and method - Google Patents

Abstract

The invention belongs to the technical field of computers, and particularly relates to an FPGA (field programmable gate array) virtualized resource scheduling system and method. An FPGA virtualized resource scheduling system comprises a resource scheduling management module and a plurality of resource processing nodes, wherein each resource processing node comprises an FPGA virtualized module and an upper controller module which are in communication connection, and the upper controller module is in communication connection with the resource scheduling management module; the FPGA virtualization module comprises a static area and one or more dynamic reconfigurable areas which are formed by dividing a physical FPGA by adopting a local dynamic reconfiguration technology, wherein each dynamic reconfigurable area is used for deploying and realizing an FPGA resource scheduling task; the upper controller module is used for supervising the FPGA resource use condition of the dynamic reconfigurable area in the FPGA virtualization module; the resource scheduling management module is used for receiving resource scheduling requirements and scheduling and managing the PFGA resources of the dynamic reconfigurable areas in all nodes according to the FPGA resource use conditions of the dynamic reconfigurable areas in each FPGA virtualization module.

Description

An FPGA virtualized resource scheduling system and method

Technical field

The invention belongs to the field of computer technology, and specifically relates to an FPGA virtualized resource scheduling system and method.

Background technique

With today's society, the trend of informatization, digitization, networking, and intelligence continues to deepen. The amount of data calculation and the scale of information processing are growing explosively. This puts forward higher requirements for the computing and control capabilities of the platform. With space-borne computers, For example, a satellite computer is an embedded system, usually composed of microprocessor, storage, communication, acquisition and other modules. The satellite computer is the brain of the satellite and is mainly responsible for on-board data management, communication management, energy management, Attitude and orbit management and load management tasks. As satellite space missions become increasingly complex, they also place higher and higher requirements on the performance of on-board computers. At the same time, on the other hand, with the slowdown of semiconductor process progress and thermal problems caused by the increase in chip density, the growth rate of the number of integrated transistors on a single chip is also further decreasing, which makes it difficult to increase the computing power of a single chip. Adapt to the current complex production environment and massive data processing needs. Therefore, how to improve chip resource utilization is the main challenge currently faced.

For traditional spaceborne computers, their "computing power" is enclosed in independent spaceborne computers. The resources of each computer cannot be shared with each other, resulting in a waste of resources. The existing closed spaceborne computing systems with independent functions can no longer meet the needs of aerospace. intelligent computing needs. Therefore, it is necessary to adopt a "cloud" computing architecture while trying to improve single-point computing capabilities. Several heterogeneous computers cooperate with each other to complete a target computing task, share (rare) computing resources, improve computing efficiency, and balance computing loads. Improve the computing capabilities of spaceborne computing systems to meet the needs of intelligent computing.

Therefore, for high-performance computing devices such as FPGA, virtualization technology is used to realize dynamic loading and resource management of task-driven device functions. Virtualization technology can virtualize more virtual platforms on one physical platform, and each virtual platform can be used as an independent part. Compared with directly using physical platforms, virtualization has huge advantages in effective utilization of resources, dynamic allocation and high reliability.

FPGA has received increasing attention in both cloud computing and edge computing due to its hardware flexibility, superior computing throughput and low energy consumption. Although FPGAs have significant advantages over CPUs and GPUs, these advantages come from design and usability trade-offs. In the traditional FPGA development model, users usually use a hardware description language (HDL) to model application scenarios, and then map the hardware model to the FPGA through specific FPGA development tools, and finally generate a FPGA image that can be run. The main disadvantage of this development model is that FPGA can only be developed and used by a single user, regardless of the application scenario, FPGA product type, etc. For example, for an application that has low resource requirements and does not need to run continuously, most FPGA hardware resources will be idle most of the time. Obviously, it is difficult to fully utilize the FPGA in the space and time range.

In order to improve the development efficiency of FPGA, better utilize the logic resources of FPGA, and facilitate the large-scale deployment and application of FPGA, it is necessary to perform a certain degree of logic abstraction on FPGA so that top-level users do not need to pay too much attention to the implementation method and implementation of FPGA hardware logic. detail. As a result, FPGA resource virtualization technology was proposed. FPGA virtualization technology breaks the limitations of time and space dimensions, allowing users to easily fully schedule and use various resources of multiple FPGAs at different times.

The current mainstream FPGA virtualization technology includes FPGAOverlay technology, partially reconfigurable and virtualization manager technology.

FPGA Overlay technology provides upper-layer users with a programming architecture and interface that they are more familiar with by adding an Overlay layer. Users only need to care about the implementation of upper-layer applications without worrying about specific hardware circuit implementation, thus realizing the control of the underlying hardware resources of FPGA. abstraction and virtualization.

Overlay technology mainly abstracts and virtualizes underlying hardware resources, provides upper-level users with a more familiar programming architecture and interface, greatly simplifies the use of hardware resources by upper-level users, and plays a role in abstracting hardware resources. However, there is an obvious flaw in FPGA virtualization implemented through Overlay technology, which is that it cannot allow multiple tenants to use FPGA resources at the same time. Therefore, FPGA virtualization implemented through this technology cannot fully use and schedule FPGA resources.

Partial reconfigurability means that the FPGA can be divided into one or more areas, and these areas can be programmed and configured separately during the operation of the FPGA to change the logic of the circuits in the area, but it does not affect the normal operation of other circuits in the FPGA. run.

Partial reconfigurability allows FPGA to switch between multiple tasks directly by hardware in both dimensions of time and space. vFPGA is essentially one or more FPGA areas that can be dynamically reconfigured. They can belong to one user together or separately. Belongs to multiple users, running the same or different applications. When one vFPGA is dynamically reconfigured, the operation of other vFPGAs will not be affected.

Using partially reconfigurable technology, the FPGA can be divided into several sub-areas and used as virtual FPGAs for single or multiple users, while retaining some logical resources as non-reconfigurable areas to implement necessary infrastructure, such as memory management and Network communications, etc. FPGA virtualization implemented using partial reconfiguration technology and virtualization management technology can realize multi-tenant sharing of FPGA resources and improve FPGA resource utilization. However, this technology cannot abstract hardware resources and simplify user application development like Overlay technology. Difficulty.

Contents of the invention

The purpose of the present invention is to provide an FPGA virtualization resource scheduling system and method, which enables multiple users to concurrently use a single FPGA resource through FPGA virtualization, and simultaneously abstracts and manages FPGA hardware resources. Users can automatically deploy UI interfaces through automatic deployment. Complete task deployment and design resource scheduling strategies for multi-node situations, which can automatically schedule user tasks to run on appropriate nodes, reduce the complexity of users' use of FPGA resources, and effectively improve the flexibility of FPGA resource use and resource utilization. .

In order to achieve the above objectives, the present invention adopts the following technical solutions:

The first aspect of the embodiment of the present invention provides an FPGA virtualized resource scheduling system, which includes a resource scheduling management module and a plurality of resource processing nodes. Each resource processing node includes a FPGA virtualization module and an upper layer controller module that are communicated with each other. The controller module communicates with the resource scheduling management module;

The FPGA virtualization module includes the use of local dynamic reconfiguration technology to divide the physical FPGA into a static area and one or more dynamic reconfigurable areas. Each dynamic reconfigurable area is used to deploy and implement FPGA resource scheduling tasks;

The upper controller module is used to supervise the usage of FPGA resources in the dynamic reconfigurable area in the FPGA virtualization module;

The resource scheduling management module is used to receive resource scheduling requirements and schedule and manage the PFGA resources of the dynamic reconfigurable areas in all nodes according to the FPGA resource usage of the dynamic reconfigurable areas in each FPGA virtualization module.

As a preferred solution, each dynamically reconfigurable area includes a vFPGA used to deploy and implement FPGA resource scheduling tasks, and a vFPGA controller that manages the vFPGA; each vFPGA communicates with the system AXI- Interconnect connected;

The vFPGA controller includes a clutch, memory management unit, DMA module, and AXI4-AXIS bridge; the clutch is connected to the vFPGA through the AXI4 interface and AXIS interface; the memory management unit is connected to the clutch and AXI-Interconnect through the AXI4 interface respectively; the DMA module is respectively connected through the AXI4 interface Connected to the clutch and AXI-Interconnect, and at the same time connected to the memory management unit; the AXI4-AXIS bridge is connected to the clutch and AXI-Interconnect respectively through the AXIS interface;

Clutch, used to isolate the signal connection between vFPGA and static logic;

The memory management unit maps the virtual address space to the corresponding physical address through the page table;

DMA module, used to implement direct memory access;

AXI4-AXIS bridge is used to realize data transmission within the vFPGA controller and data transmission between vFPGAs.

As a preferred solution, the FPGA virtualization module also includes a PICE protocol stack unit. The PICE protocol stack unit is used to maintain the PICE protocol stack and is also used to communicate between the vFPGA in the FPGA virtualization sub-module and the upper-layer controller module;

The PICE protocol stack unit includes the application layer, transaction layer, data link layer, and physical layer that are connected in sequence;

The application layer is connected to the upper-layer controller module through the PCIe interface, and converts the obtained FPGA resource scheduling task requirements into data and sends it to the transaction layer;

The transaction layer receives data from the application layer and encapsulates it into data packets and sends them to the data link layer. It can also receive data packets from the data link layer and forward them to the application layer;

The data link layer receives data packets from the transaction layer, adds Sequence Number prefix and CRC suffix, and uses ACK/NAK to ensure the reliability of data packet delivery;

The physical layer is used to receive and forward various data packets, and is also used to create and decode specialized sequences for synchronizing and managing FPGA links in each dynamically reconfigurable area.

As a preferred solution, each upper-layer controller sub-module includes a vFPGA controller driver, system manager, system monitor, FPGA hardware manager, and hardware resource pool;

vFPGA controller driver, used to control the vFPGA controller;

The system manager is used to communicate with the resource scheduling management module, is scheduled and managed by the resource scheduling management module, and is responsible for the creation and destruction of vFPGA;

The system monitor is used to periodically monitor the working status of the FPGA, the operation of each vFPGA, the usage of FPGA virtualization resources, and the traffic statistics of the PCIE protocol stack module, and persistently record the monitored data information into the database;

Hardware resource pools, including vFPGA resource pools and FPGA memory pools, are used to manage and record the use of FPGA hardware resources;

The FPGA hardware manager communicates relevant operations with the FPGA virtualization module, and implements the allocated hardware resources on specific FPGA hardware through drivers.

As a preferred solution, the system manager conducts network communication with the resource scheduling management module and transmits the FPGA virtualized resource information of the computing node to the resource scheduling management module;

Receive scheduling requests from the resource scheduling management module, make corresponding processing according to different requests, and hand them over to various downstream managers for implementation;

Receive resource information monitoring requests from the resource scheduling management module, and upload the information detected by the system monitor to the resource scheduling management module.

As a preferred solution, the resource scheduling management module also includes an automated deployment UI;

The automated deployment UI is used to receive FPGA resource scheduling requirements and conduct network communication with the resource scheduling management module;

The resource scheduling management module includes an external interface, a node management unit, and a resource scheduling unit;

The external interface is used to provide an automated deployment interface for FPGA resource scheduling tasks based on the FPGA resource scheduling task requirements obtained by the automated deployment UI;

The node management unit is used to monitor the FPGA resource usage of the dynamically reconfigurable area in each node;

The resource scheduling unit is used to schedule the received FPGA resource scheduling task to the appropriate dynamic reconfigurable area according to the FPGA resource usage in the dynamic reconfigurable area in each node.

The second aspect of the embodiment of the present invention provides an FPGA virtualization resource scheduling method, including:

The resource scheduling management module receives the FPGA resource scheduling task, and schedules and deploys the FPGA resource scheduling task to the appropriate dynamic reconfigurable area according to the needs of the FPGA resource scheduling task and the occupancy of FPGA resources of each node;

The dynamically reconfigurable area used to deploy FPGA resource scheduling tasks creates a vFPGA for running FPGA resource scheduling tasks according to the needs of the FPGA resource scheduling tasks.

As a preferred solution, the steps for the resource scheduling management module to obtain the occupancy status of each node's FPGA resources include:

After the resource scheduling management module is run, the node management unit publishes services through multicast, and the services carry IP and port information;

The system manager of each upper-layer node controller receives the multicast message and establishes a network connection with the IP and port in the multicast;

After the connection is successfully established, the system manager of each upper-layer node controller will actively initiate a registration request. After receiving the registration request, the node management unit will include the resources of the node into management to complete the registration;

After completing the registration, the system manager of each upper-layer node controller periodically reports the FPGA resource usage of the respective node to the node management unit.

As a preferred solution, the dynamic reconfigurable area used to deploy FPGA resource scheduling tasks. According to the needs of the FPGA resource scheduling tasks, the steps of creating a vFPGA for running FPGA resource scheduling tasks include:

The clutch in the vFPGA controller in the dynamic reconfigurable area is turned on, the reconfigurable module is reset and the corresponding bit stream is downloaded for reconstruction, the memory management unit is configured, and the IO_LUT is configured;

Unreset the reconfigurable module and close the clutch to complete the creation of vFPGA to load the received FPGA resource scheduling task into the vFPGA for running.

As a preferred solution, the process of configuring the memory management unit for the dynamic reconfigurable area is:

S1. Determine whether the mapped address space is smaller than the required mapping space N. If not, end the configuration process. If so, perform step S2;

S2. Take out a memory resource from the FPGA memory pool and add a mapping record to the memory management unit of the vFPGA through the driver;

S3. Repeat steps S1 to S2 until the mapped address space is greater than or equal to the required mapped space N, then end the configuration process.

The beneficial effects of the present invention are:

1. The present invention realizes the concurrent use of single FPGA resources by multiple users through the FPGA virtualization method, solves the problem that FPGA can only be developed and used by a single user, and achieves the purpose of improving FPGA resource utilization. It can make full use of FPGA resources. As long as the remaining FPGA resources can meet the running requirements of the user task, the user task can be run in the vFPGA. There will not be only one user task running on the FPGA, and there will be a large number of FPGA resources in the vFPGA. The idle state causes the problem of low resource utilization.

2. The upper controller module can respond to the FPGA resource status request of the resource scheduling management module, and respond to the resource information recorded in the database to the resource scheduling management service. At the same time, the upper controller module supports the management of FPGA resources and responds to FPGA resource requests from the resource scheduling management module. If the resources meet the resource scheduling management service requirements, the user tasks will be scheduled to run at appropriate locations.

3. Reduces the difficulty of using FPGA resources. In the traditional method, the FPGA is used as a whole, and the FPGA program is loaded into the FPGA through the emulator to run. In the present invention, the user only needs to establish a network connection with the upper-layer controller. Operations such as resource application and FPGA task creation are performed through the request response mode, which lowers the threshold for using FPGA.

4. The entire chain from resource virtualization to resource management to resource scheduling is all self-developed using C/C++. There is no need to deploy K8S. It occupies small resources, is simple to deploy, lightweight and highly flexible.

Description of the drawings

In order to explain the embodiments of the present invention or the technical solutions in the prior art more clearly, the drawings needed to be used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings in the following description are only These are some embodiments of the invention. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without exerting creative efforts.

Figure 1 is a schematic structural diagram of an FPGA virtualized resource scheduling system provided in Embodiment 1.

Figure 2 is a schematic structural diagram of the FPGA virtualization module.

Figure 3 is a schematic structural diagram of the vFPGA controller.

Figure 4 is a schematic structural diagram of the PCIE protocol stack unit.

Figure 5 is a schematic structural diagram of the upper controller module.

Figure 6 is a workflow diagram of the system manager.

Figure 7 is a schematic structural diagram of the resource scheduling management module.

Figure 8 is a flow chart for creating vFPGA in a dynamically reconfigurable area.

Detailed ways

The following describes the implementation of the present invention through specific specific examples. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments. Various details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that, as long as there is no conflict, the following embodiments and the features in the embodiments can be combined with each other.

Example 1:

Referring to Figure 1, this embodiment provides an FPGA virtualized resource scheduling system, which includes a resource scheduling management module and multiple resource processing nodes. Each resource processing node includes a FPGA virtualization module and an upper layer controller module that are communicated with each other. The controller module communicates with the resource scheduling management module;

The FPGA virtualization module uses local dynamic reconfiguration technology to divide the physical FPGA into a static area and one or more dynamic reconfigurable areas. Each dynamic reconfigurable area is used to deploy and implement FPGA resource scheduling tasks. The FPGA virtualization module will be divided into static areas and dynamic reconfigurable areas. The hardware resources of the entire FPGA include dsp, pcie, lut, ff, bram, pblock, etc. The specific hardware resources of each FPGA virtualization module are determined by the actual Determined by hardware, except for some hardware resources used by the static area firmware program (the static area program accounts for a small proportion of resources, accounting for less than 5% of the total FPGA resources), all other hardware resources are used by the dynamic area.

The static area is the area divided on the FGPA to run the firmware program. This area cannot be dynamically reconfigured. In the actual application process, the mcs file corresponding to the static area can be solidified in flash. The dynamic reconfigurable area includes Microblaze (embedded soft core), HWICAP IP core, and Pcie interface. Microblaze is software programmed through SDK to realize the local dynamic configuration function of FPGA; the HWICAP IP core is used to realize the read and write operations of the configuration space by the FPGA through the AXI bus. When the user creates an FPGA virtualization task, the HWICAP IP core is used to complete the bit stream writing. into operations on the vFPGA. The internal structure of the FPGA virtualization module is shown in Figure 2.

The upper controller module is used to supervise the usage of FPGA resources in the dynamic reconfigurable area in the FPGA virtualization module;

The resource scheduling management module is independently implemented using C/C++. It is used to receive resource scheduling requirements and, based on the FPGA resource usage of the dynamic reconfigurable area in each FPGA virtualization module, adjust the PFGA of the dynamic reconfigurable area in all nodes. Resources are scheduled and managed.

Furthermore, each dynamically reconfigurable area includes a vFPGA used to deploy and implement FPGA resource scheduling tasks, and a vFPGA controller that manages the vFPGA; each vFPGA is connected to the system AXI-Interconnect through the AXI4 Master interface and AXI4Slave interface. ;

Referring to Figure 3, the vFPGA controller includes a clutch, a memory management unit, a DMA module, and an AXI4-AXIS bridge; the clutch is connected to the vFPGA through the AXI4 interface and the AXIS interface; the memory management unit is connected to the clutch and AXI-Interconnect through the AXI4 interface; the DMA module It is connected to the clutch and AXI-Interconnect respectively through the AXI4 interface, and is connected to the memory management unit; the AXI4-AXIS bridge is connected to the clutch and AXI-Interconnect respectively through the AXIS interface;

Clutch, used to isolate the signal connection between vFPGA and static logic. The interface mainly includes AXI4, AXIS and clock signals;

The memory management unit maps the virtual address space to the corresponding physical address through the page table;

DMA module, used to implement direct memory access;

AXI4-AXIS bridge is used to realize data transmission within the vFPGA controller and data transmission between vFPGAs.

specific:

vFPGA accesses the memory through the AXI4 interface. The AXIS interface has two groups, one transmitting and one receiving, used for data transmission between modules. The data transmission between the upper controller and FPGA is completed through the PCIE protocol stack. The data transmission between the two vFPGAs in the FPGA virtualization module is directly exchanged through the AXI4-Interconnect in the module.

Data exchange between vFPGAs is performed using the AXIS interface, while the system switching structure is implemented using AXI4, and relevant components need to be designed to complete the protocol conversion. When sending an AXIS packet, the DMA module queries the written destination address from the IO LUT through the TDEST signal value, thereby writing the AXIS packet to the address through the AXI4 interface. For the receiving interface, the AXI4-AXIS bridge is required to stream the AXI4 write operation into the AXIS protocol, and the value of TDEST is determined based on the address offset being written.

vFPGA can directly use the AXI4 protocol for memory access for read and write access. However, in order to ensure the isolation of memories between vFPGAs, the memory management unit needs to be accessed for management. Each vFPGA has an independent set of virtual address spaces. The virtual address space of each vFPGA is mapped to different physical addresses through the memory management unit. The address spaces that different vFPGAs can access are mutually exclusive at the physical addresses. The address mapping table in the memory management unit is configured by PS before the vFPGA is deployed. Considering the resource overhead of the memory management unit, the design uses a 4MB address range as one page, and the memory management unit records the virtual address of each page to the physical address. Offset.

Furthermore, the FPGA virtualization module also includes a PICE protocol stack unit. The PICE protocol stack unit is used to maintain the PICE protocol stack and is also used to communicate between the vFPGA in the FPGA virtualization sub-module and the upper-layer controller module. The PICE protocol stack unit is mainly controlled by the upper-layer controller. After the vFPGA initialization is completed, it will configure PCIE related resources for the vFPGA and create the required FIFO. The vFPGA can send and receive relevant data streams through the AXIS interface and complete the SEND/RECV operation;

Referring to Figure 4, the PICE protocol stack unit is implemented in layers, and it contains multiple layers, including the application layer, transaction layer, data link layer, and physical layer from top to bottom. Each level of the PICE protocol stack unit is implemented through hardware logic. When sending, the data message is first generated by the application layer, and then passes through the transaction layer, data link layer and physical layer and is finally sent out; the receiving end does the opposite step. , the data first passes through the physical layer, then is sent upward to the data link layer, transaction layer, and finally reaches the application layer. The PICE protocol stack unit is the PCI bus.

The application layer is connected to the upper-layer controller module through the PCIe interface, and converts the obtained FPGA resource scheduling task requirements into data and sends it to the transaction layer. PCI spec specifies a 256-byte configuration space. In order to be compatible with PCI devices, the PCIe bus almost completely retains the PCI bus configuration space. And the configuration space has been expanded to 4KB to support some new functions in the PCIe bus, such as Capability, power management, MSI, etc. PCIe expands the configuration space to 4KB. The original CF8/CFC access method can still access the first 256Byte of all PCIe configuration spaces, but cannot access the remaining space. Therefore, PCIe introduces an enhanced configuration space access mechanism, which maps the configuration space to the MMIO space so that access to the configuration space is like access to memory, so the complete 4KB configuration space can be accessed.

The transaction layer receives data from the application layer and encapsulates it into data packets and sends them to the data link layer. It can also receive data packets from the data link layer and forward them to the application layer. The transaction layer defines the transactions used by the PCIE protocol stack module, most of which are compatible with the PCI bus. In the PCIe Spec, four types of requests (Request) are specified: Memory, IO, Configuration and Messages. Among them, the first three are inherited from the PCI/PCI-X bus, and the fourth type of Messages is a newly added type of PCIe. The transaction layer receives data from the core layer and encapsulates it into TLP (Transaction Layer Packet) and sends it to the data link layer. In addition, the transaction layer can also receive data packets from the data link layer and then forward them to the core layer.

The data link layer receives data packets from the transaction layer, adds Sequence Number prefix and CRC suffix, and uses ACK/NAK to ensure the reliability of data packet delivery. In addition, it also defines a variety of DLLP (Data LinkLayer Pakcet). DLLP originates from the data link layer and ends at the data link layer.

The physical layer is used to receive and forward various data packets, and is also used to create and decode specialized sequences for synchronizing and managing FPGA links in each dynamically reconfigurable area. The physical layer is the lowest layer of the PCIe bus and connects PCIe devices to each other. The physical layer also implements link training and initialization functions, which is completed through LTSSM.

Furthermore, the FPGA virtualization module requires an upper-layer controller module to manage hardware resources and communicate with the resource scheduling management module through the upper-layer controller module. In this design, the upper-layer controller module program runs in FT2000/4.

Referring to Figure 5, each upper-layer controller sub-module includes a vFPGA controller driver, system manager, system monitor, FPGA hardware manager, and hardware resource pool.

The vFPGA controller driver is used to control the vFPGA controller. Since the entire life cycle of vFPGA is controlled by the upper-layer controller sub-module, the FT2000/4 processor on the upper-layer controller sub-module runs a Linux system. In order to facilitate the upper-layer controller sub-module to manipulate the underlying hardware, the vFPGA controller driver is designed.

For the vFPGA controller, its register-related information is shown in Table 1. The driver mainly needs to operate three parts: clutch status, IO_LUT mapping table, and memory mapping table of the memory management unit MMU.

Table 1vFPGA controller register table

The driver implementation of this device adopts character device driver implementation based on device tree. For each vFPGA controller device, there will be a related character device file exposed to the user, and the user can control the underlying controller hardware by operating the /dev/vFPGA_n device file. Users communicate with device drivers through ioctl system calls.

The calling method parameters are shown in Table 2:

Table 2vFPGA controller driver interface

Among them, for DECOUPLER_ON and DECOUPLER_OFF, the system call needs to be completed in conjunction with the interrupt service routine. When the user initiates these two commands, the driver modifies the value of DECOUPLER_CTRL respectively, and then the driver blocks and waits for a completion amount, which is provided by the interrupt service routine corresponding to the interrupt triggered after the Decoupler completes the relevant operation. The rest of the command sequences read and write registers in a non-blocking manner.

The system manager is used to communicate with the resource scheduling management module, is scheduled and managed by the resource scheduling management module, and is responsible for the creation and destruction of vFPGA. The main job of the system manager is to receive and process related operation requests of the resource scheduling management service, and implement these requests on the specific hardware resources of the current FPGA according to the current resource situation. In addition, the system manager can also receive a request from the resource scheduling management module to retrieve the monitoring results of the system monitor from the database and return the results.

Further, referring to Figure 6, the workflow of the system manager is: the system manager communicates with the resource scheduling management module over the network, and transmits the FPGA virtualized resource information of the computing node to the resource scheduling management module;

Receive scheduling requests from the resource scheduling management module, make corresponding processing according to different requests, and hand them over to various downstream managers for implementation;

Receive resource information monitoring requests from the resource scheduling management module, and upload the information detected by the system monitor to the resource scheduling management module.

The system monitor is used to periodically monitor the working status of the FPGA, the operation of each vFPGA, the usage of FPGA virtualization resources, and the traffic statistics of the PCIE protocol stack module, and persistently record the monitored data information into the database.

Hardware resource pools, including vFPGA resource pools and FPGA memory pools, are used to manage and record the use of FPGA hardware resources. FPGA hardware resources include reconfigurable blocks (such as CLB/Block RAM/DSP, etc.), resources in PCIE Stack, and FPGA physical memory resources. The management of these resources is relatively simple. You only need to record the ownership of these two resources by each task. The same task can only have one reconfigurable area. As for FPGA physical memory resources, since the underlying MMU has a page size of 4M, all physical memory in the FPGA physical memory resource pool needs to be divided and managed within a 4M size range. If the physical memory size of the FPGA in this system is 8GB, the number of divided memory resources is 8GB/4M, which is 2048.

When a task is deployed, resources will be taken from the idle resources in the three pools to deploy and run the task. It will first deploy the task to the reconfigurable area according to the reconfigurable area corresponding to the bitstream file of the task, and then allocate corresponding memory to it according to the needs of the task, which requires the memory in the task description. Demand allocation. If a task requires 4G of memory, that is, the vFPGA virtual address space is 0-4GB, then 1024 resources need to be taken out from the free area in the FPGA memory pool. When a task is offloaded, the resources allocated to the task will be reclaimed to the free area of each resource pool.

The FPGA hardware manager communicates relevant operations with the FPGA virtualization module, and implements the allocated hardware resources on specific FPGA hardware through drivers. These related operations include the configuration of the reconfigurable area, the control of the clutch during the reconfiguration process, and the configuration of MMU and IO_LUT, etc.

The database can be selected according to actual needs. It is recommended to use sqlite for data persistence records of other components in the upper controller module, because sqlite is a serverless lightweight database that takes up little resources and is easy to use.

Furthermore, the resource scheduling management module also includes automated deployment UI;

The automated deployment UI is used to receive FPGA resource scheduling requirements and conduct network communication with the resource scheduling management module. The user implements parameter input through the UI interface. The UI client calls the corresponding RestfulApi interface to implement task automation based on the parameters filled in by the user. deploy.

Referring to Figure 7, the resource scheduling management module includes an external interface, a node management unit, and a resource scheduling unit;

The external interface is used to provide an automated deployment interface for FPGA resource scheduling tasks based on the FPGA resource scheduling task requirements obtained by the automated deployment UI. The external interface mainly provides RestfrulAPI services, which can realize automated deployment of tasks by calling the RestfrulApi interface.

The node management unit is used to monitor the FPGA resource usage of the dynamically reconfigurable area in each node.

The resource scheduling unit is used to schedule the received FPGA resource scheduling task to the appropriate dynamic reconfigurable area according to the FPGA resource usage in the dynamic reconfigurable area in each node.

specific:

The resource scheduling management module mainly provides users with resource management services, especially in multi-node usage scenarios. Users do not need to care about which node the task should be deployed to and the resource occupancy of each node. Users only need to use the graphical UI to assign tasks and The resource information required by the task is sent to the resource scheduling management module to complete the automated deployment of the task. Users are unaware of the internal scheduling during the entire process, which can greatly improve the FPGA resource scheduling task deployment experience. The resource scheduling management module is deployed on any CPU motherboard operating system in the same local area network.

After the resource scheduling management software module is run, the node management unit will first publish the service through multicast. The service will carry IP and port information. After receiving the multicast message, the system manager of each upper-layer controller sub-module will communicate with the multicast internal Establish a network connection with the IP and port. After the connection is successfully established, each system manager will actively initiate a registration request to the node management unit. The request information includes the node's resource type, resource status, resource usage, etc. The node management unit receives the registration After the request, the resources of the node will be included in the management. After successful registration, the system manager will periodically report the FPGA resource usage to the node management unit, so that the resource scheduling management module can control the resource usage and status of each node. Once the user initiates a task deployment request, the user task can be scheduled to the appropriate node for execution according to the scheduling policy.

When the user creates an FPGA task, the resource scheduling unit will first traverse each node to find a node that meets the resource requirements. If there is no node that meets the requirements, the task deployment fails. If only one node meets the requirements, the task will be deployed to that node. If multiple nodes meet the requirements, the task will be scheduled to run on the appropriate node according to the scheduling policy. Here we take the resource maximum utilization strategy as an example. The user task requires 500 Block RAM. Assume that two nodes meet the requirements. One node has 600 Block RAM remaining and the other node has 2000 Block RAM remaining. Then the user task is scheduled to the remaining 600 Block RAM. Block RAM nodes, this can maximize the resource utilization of a single node. The scheduling strategy is customized and designed based on actual needs. The scheduling strategy is implemented in the form of a dynamic library. Developers only need to implement the corresponding strategy according to the interface requirements to replace the scheduling strategy.

Example 2:

This embodiment provides an FPGA virtualization resource scheduling method, including:

The resource scheduling management module receives the FPGA resource scheduling task, and schedules and deploys the FPGA resource scheduling task to the appropriate dynamic reconfigurable area according to the needs of the FPGA resource scheduling task and the occupancy of FPGA resources of each node;

The dynamically reconfigurable area used to deploy FPGA resource scheduling tasks creates a vFPGA for running FPGA resource scheduling tasks according to the needs of the FPGA resource scheduling tasks.

Further, the steps for the resource scheduling management module to obtain the occupancy status of each node's FPGA resources include:

After the resource scheduling management module is run, the node management unit publishes services through multicast, and the services carry IP and port information;

The system manager of each upper-layer node controller receives the multicast message and establishes a network connection with the IP and port in the multicast;

After the connection is successfully established, the system manager of each upper-layer node controller will actively initiate a registration request. After receiving the registration request, the node management unit will include the resources of the node into management to complete the registration;

After completing the registration, the system manager of each upper-layer node controller periodically reports the FPGA resource usage of the respective node to the node management unit.

Further, referring to Figure 8, the steps for dynamically reconfigurable areas to create vFPGAs for running FPGA resource scheduling tasks include:

The clutch in the vFPGA controller in the dynamic reconfigurable area is turned on, the reconfigurable module is reset and the corresponding bit stream is downloaded for reconstruction, the memory management unit is configured, and the IO_LUT is configured;

Unreset the reconfigurable module and close the clutch to complete the creation of vFPGA to load the received FPGA resource scheduling task into the vFPGA for running.

Furthermore, referring to Figure 8, the process of configuring the memory management unit for the dynamic reconfigurable area is:

S1. Determine whether the mapped address space is smaller than the required mapping space N. If not, end the configuration process. If so, perform step S2;

S2. Take out a memory resource from the FPGA memory pool and add a mapping record to the memory management unit of the vFPGA through the driver;

S3. Repeat steps S1 to S2 until the mapped address space is greater than or equal to the required mapped space N, then end the configuration process.

The above-described embodiments are only descriptions of preferred embodiments of the present invention and do not limit the scope of the present invention. Without departing from the design spirit of the present invention, those of ordinary skill in the art may make various modifications to the technical solutions of the present invention. All deformations and improvements shall fall within the protection scope of the present invention.

Claims (10)

1. An FPGA virtualized resource scheduling system, characterized in that it includes a resource scheduling management module and a plurality of resource processing nodes. Each resource processing node includes a communication-connected FPGA virtualization module, an upper-layer controller module, and an upper-layer controller. The module communicates with the resource scheduling management module; The FPGA virtualization module includes the use of local dynamic reconfiguration technology to divide the physical FPGA into a static area and one or more dynamic reconfigurable areas. Each dynamic reconfigurable area is used to deploy and implement FPGA resource scheduling tasks; The upper controller module is used to supervise the usage of FPGA resources in the dynamic reconfigurable area in the FPGA virtualization module; The resource scheduling management module is used to receive resource scheduling requirements and schedule and manage the PFGA resources of the dynamic reconfigurable areas in all nodes according to the FPGA resource usage of the dynamic reconfigurable areas in each FPGA virtualization module. 2. An FPGA virtualized resource scheduling system according to claim 1, characterized in that each dynamically reconfigurable area includes a vFPGA for deploying and implementing FPGA resource scheduling tasks, and a vFPGA for managing the vFPGA. vFPGA controller; each vFPGA is connected to the system AXI-Interconnect through the AXI4 Master interface and AXI4 Slave interface; The vFPGA controller includes a clutch, memory management unit, DMA module, and AXI4-AXIS bridge; the clutch is connected to the vFPGA through the AXI4 interface and AXIS interface; the memory management unit is connected to the clutch and AXI-Interconnect through the AXI4 interface respectively; the DMA module is respectively connected through the AXI4 interface Connected to the clutch and AXI-Interconnect, and at the same time connected to the memory management unit; the AXI4-AXIS bridge is connected to the clutch and AXI-Interconnect respectively through the AXIS interface; Clutch, used to isolate the signal connection between vFPGA and static logic; The memory management unit maps the virtual address space to the corresponding physical address through the page table; DMA module, used to implement direct memory access; AXI4-AXIS bridge is used to realize data transmission within the vFPGA controller and data transmission between vFPGAs. 3. An FPGA virtualization resource scheduling system according to claim 1, characterized in that the FPGA virtualization module also includes a PICE protocol stack unit, and the PICE protocol stack unit is used to maintain the PICE protocol stack and is also used in the FPGA The vFPGA in the virtualization sub-module communicates with the upper-layer controller module; The PICE protocol stack unit includes the application layer, transaction layer, data link layer, and physical layer that are connected in sequence; The application layer is connected to the upper-layer controller module through the PCIe interface, and converts the obtained FPGA resource scheduling task requirements into data and sends it to the transaction layer; The transaction layer receives data from the application layer and encapsulates it into data packets and sends them to the data link layer. It can also receive data packets from the data link layer and forward them to the application layer; The data link layer receives data packets from the transaction layer, adds Sequence Number prefix and CRC suffix, and uses ACK/NAK to ensure the reliability of data packet delivery; The physical layer is used to receive and forward various data packets, and is also used to create and decode specialized sequences for synchronizing and managing FPGA links in each dynamically reconfigurable area. 4. An FPGA virtualized resource scheduling system according to claim 1, characterized in that each upper-layer controller sub-module includes a vFPGA controller driver, a system manager, a system monitor, an FPGA hardware manager, and a hardware resource pool; vFPGA controller driver, used to control the vFPGA controller; The system manager is used to communicate with the resource scheduling management module, is scheduled and managed by the resource scheduling management module, and is responsible for the creation and destruction of vFPGA; The system monitor is used to periodically monitor the working status of the FPGA, the operation of each vFPGA, the usage of FPGA virtualization resources, and the traffic statistics of the PCIE protocol stack module, and persistently record the monitored data information into the database; Hardware resource pools, including vFPGA resource pools and FPGA memory pools, are used to manage and record the use of FPGA hardware resources; The FPGA hardware manager communicates relevant operations with the FPGA virtualization module, and implements the allocated hardware resources on specific FPGA hardware through drivers. 5. An FPGA virtualized resource scheduling system according to claim 4, characterized in that the system manager communicates with the resource scheduling management module over the network to transmit the FPGA virtualized resource information of the computing node to the resource scheduling management module. ; Receive scheduling requests from the resource scheduling management module, make corresponding processing according to different requests, and hand them over to various downstream managers for implementation; Receive resource information monitoring requests from the resource scheduling management module, and upload the information detected by the system monitor to the resource scheduling management module. 6. An FPGA virtualized resource scheduling system according to claim 1, characterized in that the resource scheduling management module further includes an automated deployment UI; The automated deployment UI is used to receive FPGA resource scheduling requirements and conduct network communication with the resource scheduling management module; The resource scheduling management module includes an external interface, a node management unit, and a resource scheduling unit; The external interface is used to provide an automated deployment interface for FPGA resource scheduling tasks based on the FPGA resource scheduling task requirements obtained by the automated deployment UI; The node management unit is used to monitor the FPGA resource usage of the dynamically reconfigurable area in each node; The resource scheduling unit is used to schedule the received FPGA resource scheduling task to the appropriate dynamic reconfigurable area according to the FPGA resource usage in the dynamic reconfigurable area in each node. 7. An FPGA virtualization resource scheduling method, based on an FPGA virtualization resource scheduling system according to any one of claims 4-6, characterized in that it includes: The resource scheduling management module receives the FPGA resource scheduling task, and schedules and deploys the FPGA resource scheduling task to the appropriate dynamic reconfigurable area according to the needs of the FPGA resource scheduling task and the occupancy of FPGA resources of each node; The dynamic reconfigurable area used to deploy FPGA resource scheduling tasks creates a vFPGA for running FPGA resource scheduling tasks according to the needs of the FPGA resource scheduling tasks. 8. An FPGA virtualized resource scheduling method according to claim 7, wherein the step of the resource scheduling management module obtaining the occupancy status of each node's FPGA resources includes: After the resource scheduling management module is run, the node management unit publishes services through multicast, and the services carry IP and port information; The system manager of each upper-layer node controller receives the multicast message and establishes a network connection with the IP and port in the multicast; After the connection is successfully established, the system manager of each upper-layer node controller will actively initiate a registration request. After receiving the registration request, the node management unit will include the resources of the node into management to complete the registration; After completing the registration, the system manager of each upper-layer node controller periodically reports the FPGA resource usage of the respective node to the node management unit. 9. An FPGA virtualized resource scheduling method according to claim 7, characterized in that the dynamic reconfigurable area used to deploy FPGA resource scheduling tasks is created for running FPGA resource scheduling according to the requirements of the FPGA resource scheduling tasks. The steps of task vFPGA include: The clutch in the vFPGA controller in the dynamic reconfigurable area is turned on, the reconfigurable module is reset and the corresponding bit stream is downloaded for reconstruction, the memory management unit is configured, and the IO_LUT is configured; Unreset the reconfigurable module and close the clutch to complete the creation of vFPGA to load the received FPGA resource scheduling task into the vFPGA for running. 10. An FPGA virtualization resource scheduling method according to claim 9, characterized in that the process of configuring a memory management unit for a dynamically reconfigurable area is: S1. Determine whether the mapped address space is smaller than the required mapping space N. If not, end the configuration process. If so, perform step S2; S2. Take out a memory resource from the FPGA memory pool and add a mapping record to the memory management unit of the vFPGA through the driver; S3. Repeat steps S1 to S2 until the mapped address space is greater than or equal to the required mapped space N, then end the configuration process.