CN104123173B - A kind of method and device for realizing inter-virtual machine communication - Google Patents

Abstract

The embodiment of the invention discloses a kind of method and system for realizing inter-virtual machine communication, the hardware resource consumption for reducing PCIe device, the efficiency high communicated between VM.Present invention method includes：The DMA engine reads data and by manager described in the data Cun Chudao after the reception for receiving the PF processors is instructed from described receive on the corresponding VM of instruction；The DMA engine is after the transmission for receiving the PF processors is instructed, from the manager reads data and the corresponding VM of instruction will be sent described in the data Cun Chudao.

Description

A method and device for realizing communication between virtual machines

technical field

The present invention relates to the field of communication technology, in particular to a method and device for realizing communication between virtual machines.

Background technique

On a physical machine, usually a central processing unit (Central Processing Unit, referred to as CPU), multiple virtual machines (Virtual Machine, referred to as VM) are generated through a virtual machine monitor (Virtual Machine Manager, referred to as VMM), and each VM runs on a In a completely isolated environment, it can be understood as simulating multiple virtual computers on a physical machine. The virtual machine is like a virtual computer and can work like a real computer.

PCIe (also called PCI-Express, the full name is Peripheral Component Interface Express) is the latest bus and interface standard, the main advantage is the high data transmission rate, and it has added a single-root node input/output virtualization specification (Single Root I /O Virtualization (SR-IOV for short) technology, which can realize the connection between multiple PCIe devices such as VM shared network cards and CPUs. The PCIe device is a peripheral device of the CPU. The SR-IOV specification allows a PCIe device to be divided into a PCIe sub-device with all its physical functions through virtualization technology, which is called a physical function (Physical Function, PF for short). Multiple virtual functions (Virtual Function, VF for short) are generated through virtualization technology, and each VF can be regarded as a virtual PCIe device. Among them, PF not only has all the functions of PCIe devices, but also has the configuration and management of SR-IOV extension Function; VF depends on a certain PF and has the lightweight functions of PCIe devices, including the resources necessary for data transmission and a small amount of configuration resources. One or more VFs can be assigned to a VM, and each VM can see an independent virtual PCIe device, that is, a VF. That is to say, one VM running on the CPU can correspond to one VF in the PCIe device.

For example, the CPU and the PCIe device are connected through the PCIe bus, and one or more VFs in the PCIe device are assigned to a VM in the CPU, and multiple VMs can share the PCIe device. When multiple VMs need to exchange data, it needs to be realized through communication between VFs, that is, the data of a VM is sent from the CPU to the VF corresponding to the VM in the PCIe device through the PCIe interface, and the VF sends it to the destination VF, and then The target VF sends it to the target VM in the CPU through the PCIe interface. Since the VF only has the necessary resources for data transmission and a small amount of configuration resources, the communication of the VF needs to be managed overall by the PF, for example, to determine whether two VFs need to have communication capabilities or to perform global adjustments, etc. The existing SR-IOV specification does not provide a specific implementation mechanism for the communication between the PF and the VF. As shown in Figure 1, N VMs are generated by the VMM in the physical machine, and include a PCIe manager and a PCIe composite interface, the VMM is connected to the VM and the PCIe manager respectively, and the VMM is connected to the PCIe composite interface; in the network card It includes a PF, N VFs, and PCIe terminal interfaces. The PCIe bus is connected to the PF and the VFs respectively, and the PCIe composite interface of the CPU is connected to the PCIe terminal interface through the PCIe bus. Assign a set of hardware-implemented Mailbox (mailbox) cache and Door Bell (doorbell) register to each VF to realize the communication between PF and VF, that is, configure a Mailbox cache and a Door Bell register for each VF in the network card, Implement a mechanism similar to "knock the doorbell". If the VF has data to send, it sends the data directly to the PF, and then the PF and the VF forward the data to the destination VF through the "doorbell" mechanism.

In the embodiment shown in Figure 1 above, one PF corresponds to N VFs. When a VF needs to send data, it directly sends data to the PF. In addition, because the memory size of the Mailbox cache is limited by the memory size of the network card, if the length of the communication between the PF and the VF is greater than 64 bytes, but the memory of the network card is less than 64 bytes, the data needs to be split into multiple times to reduce system performance.

Contents of the invention

In view of the above defects, the embodiments of the present invention provide a method and device for realizing communication between virtual machines, which can reduce the consumption of hardware resources of PCIe devices, support the transmission of large data between VMs, and improve the efficiency and reliability of communication between VMs.

The first aspect of the present invention provides a method for realizing communication between virtual machines, which is applied to a bus and interface standard PCIe device, and the PCIe device is connected to a physical machine through a PCIe bus, and the physical machine includes a plurality of virtual machines VM And a manager, multiple VMs and managers are implemented in the physical machine through virtualization technology; the PCIe device includes a direct memory access DMA engine, a physical function PF processor and a plurality of virtual function VFs The processor, the PF processor and multiple VF processors are implemented in the PCIe device through virtualization technology; wherein, one VM is associated with one or more VF processors, and the manager is associated with the PF processor; the method comprising:

After the DMA engine receives the receiving instruction from the PF processor, it reads data from the VM corresponding to the receiving instruction and stores the data in the manager;

After receiving the sending instruction from the PF processor, the DMA engine reads data from the manager and stores the data in the VM corresponding to the sending instruction.

With reference to the first aspect, in a first possible implementation manner, before the DMA engine receives the receiving instruction from the PF processor, the method includes: the PF processor receives the first notification information sent by the DMA engine, The first notification information is used by the DMA engine to notify the PF processor that a VM needs to send data; the PF processor detects whether the manager is ready to receive data sent by the VM corresponding to the VF processor ; If yes, the PF processor sends a receiving instruction to the DMA engine, and the receiving instruction includes a VM that needs to send data.

With reference to the first possible implementation manner of the first aspect, in a second possible implementation manner, the PF processor detecting whether the manager is ready to receive data sent by the VM corresponding to the VF processor includes: The PF processor detects whether the allocated receiving buffer in the manager is free to determine whether the manager is ready to receive the data sent by the VM corresponding to the VF processor, and the allocated receiving buffer in the manager Used to store data.

With reference to the first aspect, or the first possible implementation of the first aspect, or the second possible implementation of the first aspect, in a third possible implementation, the sending from the manager After the cache reads the data and stores the data in the receiving cache of the VM corresponding to the sending instruction, it includes: the DMA engine sends first response information to the PF processor, and the first response information is used to notify The PF processor has successfully stored the data sent by the VM corresponding to the received instruction into the manager.

In combination with the first aspect, or the first possible implementation of the first aspect, or the second possible implementation of the first aspect, or the third possible implementation, in the fourth possible implementation, After receiving the sending instruction from the PF processor, the DMA engine includes: the DMA engine detects whether the VM corresponding to the sending instruction is ready to receive the data; if so, executes reading data from the manager and A step of storing the data in the VM corresponding to the sending instruction.

With reference to the fourth possible implementation manner, in a fifth possible implementation manner, the DMA engine detecting whether the VM corresponding to the sending instruction is ready to receive the data includes: the DMA engine detecting that the VM corresponding to the sending instruction Whether the allocated receive buffer in the VM of the said sending instruction is free to determine whether the VM corresponding to the sending instruction is ready to receive the data, and the allocated receiving buffer in the VM corresponding to the sending instruction is used to store data.

With reference to the fourth possible implementation manner, or the fifth possible implementation manner, in a sixth possible implementation manner, in the reading data from the manager and storing the data in the sending instruction Afterwards, the corresponding VM includes: the DMA engine sends second response information to the PF processor, and the second response information is used to notify the PF processor that the data has been successfully read from the manager and stored to the VM corresponding to the sending instruction; the DMA engine sends second notification information to the VF processor associated with the VM corresponding to the sending instruction, and the second notification information is used to notify the VM corresponding to the sending instruction The associated VF processor has successfully stored the data of the manager into the VM corresponding to the sending instruction.

The second aspect of the embodiment of the present invention provides a direct memory access DMA engine, which is applied to a bus and interface standard PCIe device, and the PCIe device is connected to a physical machine through a PCIe bus, and the physical machine includes a plurality of virtual machines VM and a manager, multiple VMs and managers are implemented in the physical machine through virtualization technology; the DMA engine, a physical function PF processor and a plurality of virtual function VFs are included in the PCIe device processor, the PF processor and multiple VF processors are implemented in the PCIe device through virtualization technology, wherein one VM is associated with one or more VF processors, and the manager is associated with the PF processor, the DMA engine includes:

The first processing module is configured to read data from the VM corresponding to the received instruction and store the data in the manager after receiving the received instruction from the PF processor;

The second processing module is configured to read data from the manager and store the data in the VM corresponding to the sending instruction after receiving the sending instruction from the PF processor.

With reference to the second aspect, in a first possible implementation manner, the DMA engine further includes: a first response module, configured to send first response information to the PF processor, and the first response information is used to notify The PF processor has successfully stored the data sent by the VM corresponding to the received instruction into the manager.

With reference to the second aspect, or the first possible implementation of the second aspect, in the second possible implementation, the DMA engine further includes: a detection module, configured to detect whether the VM corresponding to the sending instruction is ready Good to receive the data.

With reference to the second possible implementation of the second aspect, in a third possible implementation, the DMA engine further includes: a second response module, configured to send second response information to the PF processor, the The second response information is used to notify the PF processor that it has successfully read data from the manager and stored it in the VM corresponding to the sending instruction; and, processing the VF associated with the VM corresponding to the sending instruction The manager sends second notification information, where the second notification information is used to notify the VF processor associated with the VM corresponding to the sending instruction that the data of the manager has been successfully stored in the VM corresponding to the sending instruction.

The third aspect of the embodiment of the present invention also provides a system for realizing communication between virtual machines, which may include: a bus and interface standard PCIe device and a physical machine, the PCIe device and the physical machine are connected through the PCIe bus, and in the physical machine Including a plurality of virtual machine VMs and a manager, the plurality of VMs and managers are implemented in the physical machine through virtualization technology; the PCIe device includes a physical function PF processor, a plurality of virtual function VF processing and a direct memory access DMA engine, the PF processor and multiple VF processors are implemented in the PCIe device through virtualization technology; wherein, one VM is associated with one or more VF processors, so The manager is associated with the PF processor;

The DMA engine includes:

The first processing module is configured to read data from the VM corresponding to the received instruction and store the data in the manager after receiving the received instruction from the PF processor;

The second processing module is configured to, after receiving the sending instruction from the PF processor, read data from the manager and store the data in the VM corresponding to the sending instruction;

The PF processor is configured to send the receiving instruction and the sending instruction to the DMA engine.

With reference to the third aspect, in a first possible implementation manner, the PF processor is further configured to: receive first notification information sent by the DMA engine, where the first notification information is used by the DMA engine to notify The PF processor has a VM that needs to send data; detect whether the manager is ready to receive the data sent by the VM corresponding to the VF processor; if so, execute sending the receiving instruction and the sending instruction to the DMA engine. In the step of instructing, the receiving instruction includes a VM that needs to send data.

With reference to the third aspect, or the first possible implementation of the third aspect, in a second possible implementation, the PF processor is further configured to: detect whether the allocated receive buffer in the manager is free To determine whether the manager is ready to receive data sent by the VM corresponding to the VF processor, the allocated receive buffer in the VM corresponding to the sending instruction is used to store data.

With reference to the second possible implementation manner of the third aspect, in a third possible implementation manner, the DMA engine further includes: a first response module, configured to send first response information to the PF processor, the The first response information is used to notify the PF processor that the data sent by the VM corresponding to the received instruction has been successfully stored in the manager.

With reference to the third aspect, in a fourth possible implementation manner, the DMA engine further includes: a detection module, configured to detect whether the VM corresponding to the sending instruction is ready to receive the data.

With reference to the fourth possible implementation of the third aspect, in a fifth possible implementation, the DMA engine further includes: a second response module, configured to send second response information to the PF processor, the The second response information is used to notify the PF processor that it has successfully read data from the manager and stored it in the VM corresponding to the sending instruction; and, processing the VF associated with the VM corresponding to the sending instruction The manager sends second notification information, where the second notification information is used to notify the VF processor associated with the VM corresponding to the sending instruction that the data of the manager has been successfully stored in the VM corresponding to the sending instruction.

From the above technical solutions, it can be seen that the method and device for realizing communication between virtual machines provided by the embodiments of the present invention have the following advantages: through the direct memory access (Direct Memory Access, referred to as DMA) engine in the management associated with the PF processor The data can be read and stored directly between the VM associated with the processor and the VF processor, so as to realize the communication between VMs. Compared with the existing technology, there is no need to allocate corresponding Cache can reduce the consumption of hardware resources of PCIe devices, and can directly transmit large data, effectively improving the efficiency of communication between VMs.

Description of drawings

In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following will briefly introduce the accompanying drawings required in the embodiments of the present invention. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For Those of ordinary skill in the art can also obtain other drawings based on these drawings without making creative efforts.

FIG. 1 is a schematic diagram of a virtual machine communication application provided by the prior art of the present invention;

FIG. 2a is a schematic flowchart of a method for implementing communication between virtual machines provided by an embodiment of the present invention;

FIG. 2b is a schematic diagram of an implementation of a method for implementing communication between virtual machines provided by an embodiment of the present invention;

FIG. 3 is a schematic diagram of a virtual machine communication application provided by an embodiment of the present invention;

FIG. 4 is a schematic flowchart of a method for implementing communication between virtual machines provided by another embodiment of the present invention;

FIG. 5 is a schematic flowchart of a method for implementing communication between virtual machines provided by another embodiment of the present invention;

FIG. 6 is a schematic structural diagram of a DMA engine provided by an embodiment of the present invention;

FIG. 7a is a schematic structural diagram of a system for implementing communication between virtual machines provided by an embodiment of the present invention;

Fig. 7b is an application schematic diagram of a system for implementing communication between virtual machines provided by an embodiment of the present invention.

detailed description

Embodiments of the present invention provide a method and device for realizing communication between virtual machines, which are used to reduce hardware resource consumption of PCIe devices and improve communication efficiency between VMs.

The following will clearly and completely describe the technical solutions in the embodiments of the present invention with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some, not all, embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by persons of ordinary skill in the art without creative efforts fall within the protection scope of the present invention.

The terms "first", "second", "third", "fourth", etc. (if any) in the description and claims of the present invention and the above drawings are used to distinguish similar objects, not to Describe a specific order or sequence. It is to be understood that the data so used are interchangeable under appropriate circumstances such that the embodiments of the invention described herein are, for example, capable of practice in sequences other than those illustrated or described herein. Furthermore, the terms "comprising" and "having", as well as any variations thereof, are intended to cover a non-exclusive inclusion, for example, a process, method, system, product or device comprising a sequence of steps or elements is not necessarily limited to the expressly listed instead, may include other steps or elements not explicitly listed or inherent to the process, method, product or apparatus.

Below with specific embodiment, the present invention is described in detail:

Please refer to FIG. 2a. FIG. 2a is a schematic flowchart of a method for implementing communication between virtual machines provided by an embodiment of the present invention; and in combination with FIG. 2b, FIG. 2b is a schematic diagram of an implementation of a method for implementing communication between virtual machines provided by an embodiment of the present invention; As shown in Figure 2a, a method for realizing communication between virtual machines is applied to a bus and interface standard PCIe device as shown in Figure 2b, the PCIe device is connected to a physical machine through a PCIe bus, and the physical machine includes N VMs and a manager, N VMs and managers are implemented in the physical machine through virtualization technology; the PCIe device includes a DMA engine, and a physical function PF processor and M virtual machines are configured in the PCIe device Function VF processors, PF processors and VF processors are implemented in PCIe devices through virtualization technology, wherein, N and M are both positive integers greater than or equal to 1; one VM is associated with one or more VF processors, The manager is associated with the PF processor.

As shown in Figure 2a, a method for implementing communication between virtual machines may specifically include:

S201. After receiving the receiving instruction from the PF processor, the DMA engine reads data from the VM corresponding to the receiving instruction and stores the data in the manager;

The PCIe device is connected to the physical machine through the PCIe bus, the DMA engine is connected to the PF processor and the VF processor, and the direct access of data between VMs is controlled by the DMA engine.

The receiving instruction includes at least the VM that needs to send data, indicating that the VM is ready to send the data, and the manager is ready to receive the data. The DMA engine can read the data from the VM corresponding to the receiving instruction and store it in the manager. middle.

Therefore, before step S201, the PF processor performs the following processing:

A1. The PF processor receives first notification information sent by the DMA engine, where the first notification information is used by the DMA engine to notify the PF processor that a VM needs to send data;

A2. The PF processor detects whether the manager is ready to receive data sent by the VM corresponding to the VF processor;

A3. If yes, the PF processor sends a receiving instruction to the DMA engine, and the receiving instruction includes a VM that needs to send data.

After successfully reading the data on the VM corresponding to the received instruction and storing it in the manager, the DMA engine sends the first response information to the PF processor, and the first response information is used to notify the PF processor that it has successfully translated the received The data sent by the VM corresponding to the instruction is stored in the manager.

S202. After receiving the sending instruction from the PF processor, the DMA engine reads the data from the manager and stores the data in a VM corresponding to the sending instruction.

The execution subject of the embodiment of the present invention is the DMA engine, and the DMA engine will read data from the VM corresponding to the received instruction, and store the data in the manager; if the DMA engine receives the sending instruction sent by the PF processor, it will read the data from the The manager reads the data and stores the data in the VM corresponding to the sending instruction. In the embodiment of the present invention, the manager is used as a transfer point, and the direct access between the VM and the manager is performed through the DMA engine, thereby realizing the communication between the VMs, without being limited by the hardware resources of the PCIe device, and supporting large data Processing, transmission efficiency is higher.

After the DMA engine receives the sending instruction of the PF processor, the DMA engine will also detect whether the VM corresponding to the sending instruction is ready to receive data, if so, read the data from the manager and store the data in the sending instruction in the corresponding VM.

After successfully reading the data in the manager and storing it in the VM corresponding to the sending command, the DMA engine will also send a second response message to the PF processor to notify the PF processor that it has successfully read data from the manager And store it in the VM corresponding to the sending instruction; then send the second notification information to the VF processor associated with the VM receiving the data, for notifying the VF processor associated with the VM receiving the data that it has successfully transferred the data of the manager Stored into its associated VM.

In a preferred embodiment of the present invention, each VM allocates two memory spaces with continuous addresses in its memory as VF buffers, such as a receive buffer and a send buffer. Among them, the receiving buffer is used to receive data from the manager, and the sending buffer is used to store data sent to the manager. Correspondingly, two memory spaces with continuous addresses are also allocated in the manager as PF caches, which are also named receive cache and send cache respectively. The size of the receiving buffer and sending buffer of the VM can be greater than 64Bytes. Correspondingly, the size of the receiving buffer and sending buffer in the manager is also larger than 64Bytes. The size of the receiving buffer and sending buffer in practical applications depends on the size of the physical machine memory. Since the manager is used as a transit point for communication between VMs, the receiving buffer and sending buffer of the manager can be allocated first, and then the receiving buffer and sending buffer in all VMs can be allocated according to the size of the receiving buffer and sending buffer in the manager .

At the same time, configure related register groups for the PF processor and each VF processor. Among them, the register group configured by the PF processor and its basic functions are as follows:

A1. PF_Mailbox_Size: The PF processor can perform read and write operations, and the PF processor writes the maximum cache size that can be supported.

A2. PF_RxMailbox_Addr: The PF processor can perform read and write operations. It is the starting address allocated by the PF processor in its memory for storing data from the VF (that is, the starting address of storing data in the receive buffer in the manager).

A3. PF_TxMailbox_Addr: The PF processor can perform read and write operations. It is allocated by the PF processor in its memory to store the start address of the data sent to the VF (that is, the start address of the send buffer in the manager).

A4. PF_TxLength: The PF processor can perform read and write operations, which is the effective length value of the data put into the transmit buffer by the PF processor.

A5. Dest_VF: The PF processor can perform read and write operations, and is written by the PF processor. It is the destination VF to which the data in the sending buffer is sent.

A6. VF_Req_Sts: PF can be read, and can also be cleared by writing 0. It is a multi-bit status register, and each bit corresponds to a VF processor. When the VF processor sets a register of the corresponding VF processor in the VF request register to 1, an interrupt is triggered and sent to the PF. The PF processor can know which VF processors are ready to send data to itself by reading this register.

A7. PF_Rx_En: PF can perform read operation and write 1 set operation. It is a multi-bit control register, and each bit corresponds to a VF processor. When the PF processor determines that the receive buffer of the manager is idle, and determines to receive data from a VF processor, it needs to set the corresponding bit register to 1 (the corresponding register in the PF_Rx_Done register can also be cleared at the same time) to Notify the DMA engine to start reading data from the send buffer of the VM associated with the VF processor to the receive buffer of the manager associated with the PF processor. Once the DMA engine reading is completed, the DMA engine needs to clear the corresponding bit of this register, and at the same time set a certain bit corresponding to the following PF reception completion register PF_Rx_Done to 1 (indicating that the data has been placed and needs to be processed by the PF processor) .

It is worth noting that the PF processor can only set one bit of PF_Rx_En to 1 at a time, that is, the DMA engine can only read data from one VF processor at a time.

A8. PF_Rx_Done: The PF processor can read and write 0 to clear. It is a multi-bit status register, and each bit corresponds to a VF processor. When the DMA engine has finished reading the data from the transmit buffer associated with a certain VF processor, it will send an interrupt to the PF processor, and set the corresponding bit of this register to 1 at the same time. The PF processor can read this register to know whether the data in its receive buffer is valid. After the PF processor has taken or used up the data in the receiving buffer, it can set the corresponding bit of this register to 0, so that it can be ready to receive new data.

A9. PF_Rx_Length: The PF processor can perform read operations. When the DMA engine writes the effective length value of the data stored in the receive buffer of the manager into it, the value in this register is only valid when PF_Rx_Done is 1.

A10, PF_Req: The PF processor can read and write 1 to set, but it cannot be cleared. Once the PF processor has completely put the data to be sent into the send buffer, write the target VF into Dest_VF, and write the length of the send data into PF_TxLength, you can set this register to 1 and notify the DMA engine PF The processor has data to send. When the DMA engine finds that the PF_Done register of the corresponding target VF is 0, it can start reading data. Once the reading is completed, the DMA engine clears PF_Req, and at the same time sets PF_Done of the corresponding VF processor to 1, and PF_TxLength of the VF processor writes the actual read data length value. After the PF processor finds that the PF_Req register is cleared, it can put new data into the receive buffer.

Correspondingly, the register set configured for VF and its basic functions are as follows:

B1. PF_Mailbox_Size: The VF processor can perform read operations. It is the mirror image of PF_Mailbox_Size in the PF processor. It is just a read-only register used to inform the VF processor of the cache size supported by the PF processor;

After the manager associated with the PF processor allocates the cache size, write its size value into PF_Mailbox_Size, and each VM obtains the cache size set by the manager by reading the image PF_Mailbox_Size in its associated VF processor, so as to set it correctly The size of the data sent by the VM to the PF to avoid data loss due to exceeding the cache size allocated by the PF processor.

B2. VF_RxMailbox_Addr: The VF processor can perform read and write operations. It is allocated by the VF processor in its memory to store the buffer start address of the data from the PF processor (that is, the start address of the receive buffer).

B3. VF_TxMailbox_Addr: The VF processor can perform read and write operations. It is allocated by the VF processor in its memory to store the buffer start address of the data sent to the PF processor (that is, the start address of the send buffer).

B4. VF_TxLength: The VF processor can perform read and write operations, which is the effective length value of the data put into the transmit buffer by the VF processor.

B5. VF_Req: The VF processor can read and write 1 to set, but it cannot be cleared. Once the VF processor has completely put the data to be sent into the send buffer, this register can be set to 1 to trigger an interrupt to be sent to the PF processor to notify the PF processor to receive the data. This register will be cleared after the PF processor has received data and cleared its PF_Rx_Done register. After the VF processor finds that this register is cleared, it indicates that the sending buffer can put new data.

B6. PF_Done: The VF processor can perform read operations and write 0 to clear, but the VF processor cannot write 1 to set. When the DMA engine reads the data from the send buffer in the manager associated with the PF processor into the receive buffer of the VF, it will set this register to 1 to trigger an interrupt to be sent to the VF processor to notify the VF The processor receives data; or the VF processor judges whether it needs to receive data by regularly checking whether this register is 1. After the VF processor has used up the data in the receiving buffer, it can write 0 to this register to clear it, so that new data can be received.

B7. PF_TxLength: The VF processor can only read but not write. When the DMA engine reads the data from the send buffer of the manager into the receive buffer of the VF, it will write the specific written length value into In this register, so that the VF processor can access it.

It is worth noting that the value in the PF_TxLength register is only valid when PF_Done is 1.

The following takes the X86 platform as an example to further introduce the present invention. Specifically, as shown in Figure 3, the CPU in the X86 platform is connected to the PCIe device through the PCIe bus. The CPU generates N VMs through the VMM, and configures the manager to allocate two consecutive blocks of memory for each VM and the manager, respectively. Receive cache and send cache, each VM is associated with one or more VF processors in the PCIe device, the manager is associated with PF, specifically, each VM has its operating system, and correspondingly, it is also installed with the The VF driver of the VF processor associated with the VM. Similarly, in addition to the operating system installed in the manager, the PF driver of the PF processor associated with the manager is also correspondingly installed.

Among them, the physical machine also includes a PCIe root node (English: PCIe Root Complex, which may be referred to as RC) specified by the PCIe protocol, and the PCIe root node is connected to to the PCIe bus. The PCIe device also includes a PCIe end point (English: PCIe Endpoint) stipulated in the PCIe protocol. The PCIe end point is connected to the PCIe bus and connected to the PF processor and each VF processor.

The PF processor also includes a register set, wherein the register set includes the above-mentioned registers; similarly, each VF processor includes a register set, and the register set includes the above-mentioned registers of the VF processor.

Based on the above architecture, it can be understood that the original communication between the VF processor and the PF processor can be realized through the communication between VMs. The following will introduce the implementation process of storing the data of the VM associated with the VF processor to the manager associated with the PF processor. Please refer to Figure 4 for details. Figure 4 is another embodiment of the present invention for implementing communication between virtual machines A schematic flow chart of the method, which may include:

S401. The VF processor determines whether the value of the register VF_Req is cleared;

Among them, whether the value of the register VF_Req is cleared, that is, whether the value of the register VF_Req is equal to 1’b0. When the sending buffer of the VM associated with the VF processor is put into data, the register VF_Req writes 1 to set, and after the data in the sending buffer of the VM is read, it is cleared. When it is not cleared, send and execute step S401; when it is cleared, execute step S402.

S402, the VF processor writes the data to be sent into the sending buffer, and writes the data length into the register VF_TxLength, and writes 1 to the register VF_Req to set;

Among them, the VF processor can write 1 to VF_Req, but it cannot be cleared. Writing 1 to VF_Req is to write 1’b1 to the register VF_Req, indicating that the VF processor has put the data to be sent into the sending buffer, and can notify the PF processor to receive the data.

S403. After the DMA engine detects that the register VF_Req is set, it sets the status register of the VF processor corresponding to the register VF_Req_Sts of the PF processor, and sends the first notification information to the PF processor;

The DMA engine sets the register corresponding to the VF processor in the multi-bit status register in the PF processor, and then sends the first notification message to the PF processor, so that the PF processor knows which VM associated with the VF processor needs to send data.

Specifically, the first notification information is an interrupt signal generated by the DMA engine after setting the status register of the VF processor corresponding to the register VF_Req_Sts.

S404, the PF processor detects whether the receive buffer in the manager is idle;

When the receiving buffer of the manager is free, it can be used to receive data from the VM; when the receiving buffer of the manager is not free, it continues to wait until the receiving buffer of the manager is free. Therefore, when the manager's reception buffer is free, go to step S405.

S405. The PF processor sets the corresponding position of the register PF_Rx_En to 1 according to the value of the register VF_Req_Sts, and then sends a receiving instruction to the DMA engine;

The PF processor knows which VM associated with the VF sends data through the bit set to 1 in VF_Req_Sts, and then sets the corresponding bit of PF_Rx_En to 1, indicating that the receiving buffer in the manager associated with the PF processor is ready to receive data ready, the DMA engine can read data.

S406. The DMA engine reads data from the sending buffer of the VM corresponding to the receiving instruction, and stores the read data in the receiving buffer of the manager;

S407. The DMA engine sets the register PF_Rx_Done, writes the data length into the register PF_Rx_Length, and clears the register PF_Rx_En.

After the DMA engine successfully reads the data and stores the data in the receive buffer of the manager, write 1 to the register PF_Rx_Done, and the PF processor knows that the data in the receive buffer of the manager is valid by reading the value of the register PF_Rx_Done. At the same time, the DMA engine also sends the first response information to the PF processor, that is, sends an interrupt signal, notifying the PF processor that the data has been successfully read and stored.

The length of the read data has been written into the register PF_Rx_Length, so that the PF processor can know the length of the data by reading the value of the register PF_Rx_Length.

Afterwards, after the PF processor knows that the data in the receive buffer of the manager is valid according to the value of the register PF_Rx_Done, if it is data interaction between VMs, the data in the receive buffer of the manager needs to be sent to the target VM, then the PF processor will The data in the manager's receiving buffer is put into the sending buffer, ready to be sent. If the data in the manager receiving buffer is the information that the VF processor associated with the VM requests resources from the PF processor, then the PF processor puts the data requested by the VF processor into the sending buffer according to the data received by the manager, and at the same time The data in the manager's receive buffer has been used up.

Then the PF processor clears the value of the register PF_Rx_Done, indicating that the data in the manager's receiving buffer has been read or used up, and it is ready to receive new data. Write 0 to clear the corresponding bit of the register VF_Req_Sts, indicating that the current VF processor corresponding to this bit has no data to send.

In addition, the DMA engine also clears the value of the register VF_Req of the VF processor. By reading the value of the register VF_Req, the VF processor can put new data into the sending buffer of the associated VM after the value of the register VF_Req is cleared. .

In another embodiment, the present invention will introduce the implementation process of reading data from the manager and storing it into the VM, which is similar to the process of Figure 4, please refer to Figure 5, Figure 5 is the realization of the virtual machine provided by another embodiment of the present invention Schematic flow diagram of the method for inter-communication, which may include:

S501. The PF processor determines whether the value of the register PF_Req is cleared;

Among them, whether the value of the register PF_Req is cleared, that is, whether the value of the register PF_Req is equal to 1’b0. After the value of the register PF_Req is cleared, data can be put into the sending buffer of the manager. Therefore, when it is not cleared, the transmission executes step S501 until it is cleared, and when it is cleared, it turns to step S502.

S502, the PF processor writes the data to be sent into the sending buffer, writes the target VF processor into the register Dest_VF and writes the length of the data into PF_Tx_Length;

The sending buffer refers to the sending buffer allocated in the manager associated with the PF processor.

The PF processor writes the ID number of the target VF processor to be sent to the register Dest_VF.

S503, the PF processor sets the register PF_Req, and sends a sending instruction to the DMA engine;

Among them, the PF processor can set the register PF_Req, but cannot clear it. PF_Req is set, that is, 1’b1 is written into the register PF_Req, indicating that the PF processor has put the data to be sent into the send buffer of the manager, and then notifies the DMA engine to read it.

S504, the DMA engine detects whether the PF_Done of the target VF processor is cleared;

Wherein, when the register PF_Done is cleared, it indicates that the receiving buffer in the VM associated with the VF processor is idle, and data can be put into the receiving buffer.

When the register PF_Done has not been cleared, continue to wait; after the register PF_Done is cleared, turn to step S505.

S505. The DMA engine reads data from the sending buffer of the manager, and stores the read data in the receiving buffer of the VM associated with the target VF processor;

When the DMA engine determines that the send buffer of the manager has data to be sent and the receive buffer of the VM associated with the target VF processor is free, it will read data from the send buffer of the manager and store it in the receive buffer of the VM associated with the target VF middle.

S506, the DMA engine sets the register PF_Done, writes the length of the data into the register PF_TxLength and writes 1'b0 to the register PF_Req to clear;

After the DMA engine finishes reading the data, it writes 1 to the register PF_Done in the target VF processor, and the target VF processor reads the register PF_Done to determine whether the data in the VM receive buffer is valid.

S507. The DMA engine sends second notification information to the target VF processor.

The second notification information is used to notify the target VF processor that the data of the manager has been successfully stored in the receive cache of its corresponding VM.

Specifically, the second notification information is an interrupt signal generated by the DMA engine.

After receiving the second notification information, the target VF processor performs corresponding processing on the data in the VM receiving buffer according to actual business needs. After that, write 0 to PF_Done to clear it, so that the receive buffer of the VM can be ready to receive new data.

Please refer to FIG. 6, the embodiment of the present invention also provides a DMA engine based on the above-mentioned method for realizing inter-virtual machine communication. The DMA engine is set in the PCIe device shown in FIG. 3 and communicates with the PF processor and the VF processor Connect respectively; As shown in Figure 6, a kind of DMA engine 600 can comprise:

The first processing module 610 is configured to, after receiving the receiving instruction from the PF processor, read data from the VM corresponding to the receiving instruction and store the data in the manager;

The second processing module 620 is configured to read data from the manager and store the data in the VM corresponding to the sending instruction after receiving the sending instruction from the PF processor.

Wherein, after the first processing module in the DMA engine receives the receiving instruction from the PF processor, it reads data from the VM associated with the VF processor corresponding to the receiving instruction and stores it in the manager; and the second processing module After the module receives the sending instruction from the PF processor, it reads the data from the manager and stores it in the VM corresponding to the sending instruction, without additionally allocating memory in the PCIe device, reducing the resource consumption of the PCIe device , read data through DMA, which effectively improves the communication efficiency between VMs.

In a possible implementation manner of the present invention, the above-mentioned DMA engine 600 also includes:

The first response module is configured to send first response information to the PF processor, the first response information is used to notify the PF processor that the data sent by the VM corresponding to the received instruction has been successfully stored in the in the manager.

In another possible implementation manner of the present invention, the above-mentioned DMA engine 600 also includes:

A detection module, configured to detect whether the VM corresponding to the sending instruction is ready to receive the data;

If yes, the above-mentioned second processing module executes the step of reading data from the manager and storing the data in the VM corresponding to the sending instruction.

In another possible implementation manner of the present invention, the above-mentioned DMA engine 600 also includes:

The second response module is configured to send second response information to the PF processor, and the second response information is used to notify the PF processor that the data has been successfully read from the manager and stored in the sending instruction In the corresponding VM; and, sending second notification information to the VF processor associated with the VM corresponding to the sending instruction, where the second notification information is used to notify the VF processor associated with the VM corresponding to the sending instruction, The data of the manager has been successfully stored in the VM corresponding to the sending instruction.

Please refer to FIG. 7a. FIG. 7a is a schematic structural diagram of a system for realizing communication between virtual machines provided by an embodiment of the present invention; as shown in FIG. 7a, the system includes a physical machine and a PCIe device, and the PCIe device and the physical The machines are connected through the PCIe bus, and the physical machine includes a plurality of virtual machine VMs and a manager, and the plurality of VMs and the manager are implemented in the physical machine through virtualization technology; the PCIe device includes a physical A function PF processor, a plurality of virtual function VF processors and a direct memory access DMA engine, the PF processor and a plurality of VF processors are implemented in the PCIe device through virtualization technology; wherein, one VM is associated One or more of the VF processors, the manager is associated with the PF processors;

The above-mentioned DMA engine is shown in FIG. 6 , and reference may be made to the above-mentioned detailed introduction, and details are not repeated here.

In addition, the above-mentioned PF processor is configured to send the receiving instruction and the sending instruction to the DMA engine.

In an implementable manner, the PF processor is further configured to: receive first notification information sent by the DMA engine, where the first notification information is used by the DMA engine to notify the PF processor that there is a VM Need to send data; detect whether the manager is ready to receive the data sent by the VM corresponding to the VF processor; if so, perform the steps of sending the receiving instruction and the sending instruction to the DMA engine, and the receiving Instructions include VMs that need to send data.

In an implementable manner, the PF processor is further configured to: detect whether the allocated receive buffer in the manager is free to determine whether the manager is ready to receive the message sent by the VM corresponding to the VF processor For data, the allocated receiving buffer in the VM corresponding to the sending instruction is used to store data.

Please refer to FIG. 7b. FIG. 7b is a schematic diagram of the application of a system for realizing communication between virtual machines provided by another embodiment of the present invention; in the above-mentioned manager and each VM, a sending cache and a receiving cache are allocated respectively, and at the same time, in the PF A set of registers is allocated to the processor, and a set of registers is allocated to each VF processor. The details are as described above and will not be repeated here.

In the foregoing embodiments, the descriptions of each embodiment have their own emphases, and for parts not described in detail in a certain embodiment, reference may be made to relevant descriptions of other embodiments.

Those skilled in the art can clearly understand that for the convenience and brevity of the description, the specific working process of the above-described system, device and unit can refer to the corresponding process in the foregoing method embodiment, which will not be repeated here.

In the several embodiments provided in this application, it should be understood that the disclosed system, device and method can be implemented in other ways. For example, the device embodiments described above are only illustrative. For example, the division of the units is only a logical function division. In actual implementation, there may be other division methods. For example, multiple units or components can be combined or May be integrated into another system, or some features may be ignored, or not implemented. In another point, the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces, and the indirect coupling or communication connection of devices or units may be in electrical, mechanical or other forms.

The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place, or may be distributed to multiple network units. Part or all of the units can be selected according to actual needs to achieve the goal of the solution of this embodiment.

In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, each unit may exist separately physically, or two or more units may be integrated into one unit. The above-mentioned integrated units can be implemented in the form of hardware or in the form of software functional units.

If the integrated unit is realized in the form of a software function unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the essence of the technical solution of the present invention or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium , including several instructions to make a computer device (which may be a personal computer, a server, or a network device, etc.) execute all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), magnetic disk or optical disk, and other media that can store program codes.

A method and system for realizing communication between virtual machines provided by the present invention has been introduced in detail above. For those skilled in the art, according to the idea of the embodiment of the present invention, there will be changes in the specific implementation and application scope. In summary, the content of this specification should not be construed as limiting the present invention.

Claims (17)

1. A method for realizing communication between virtual machines is applied in bus and interface standard PCIe equipment, and said PCIe equipment is connected with a physical machine by a PCIe bus, and said physical machine includes a plurality of virtual machine VMs and a manager, Multiple VMs and managers are implemented in the physical machine through virtualization technology; the PCIe device includes a direct memory access DMA engine, a physical function PF processor and a plurality of virtual function VF processors, the The PF processor and multiple VF processors are implemented in the PCIe device through virtualization technology; wherein, one VM is associated with one or more VF processors, and the manager is associated with the PF processor; It is characterized in that the method comprises: After the DMA engine receives the receiving instruction from the PF processor, it reads data from the VM corresponding to the receiving instruction and stores the data in the manager; After receiving the sending instruction from the PF processor, the DMA engine reads data from the manager and stores the data in the VM corresponding to the sending instruction. 2. The method according to claim 1, wherein the DMA engine comprises before receiving the receiving instruction of the PF processor: The PF processor receives first notification information sent by the DMA engine, where the first notification information is used by the DMA engine to notify the PF processor that a VM needs to send data; The PF processor detects whether the manager is ready to receive data sent by the VM corresponding to the VF processor; If yes, the PF processor sends a receiving instruction to the DMA engine, where the receiving instruction includes a VM that needs to send data. 3. The method according to claim 2, wherein the detecting by the PF processor whether the manager is ready to receive the data sent by the VM corresponding to the VF processor comprises: The PF processor detects whether the allocated receiving buffer in the manager is free to determine whether the manager is ready to receive the data sent by the VM corresponding to the VF processor, and the allocated receiving buffer in the manager Used to store data. 4. The method according to any one of claims 1 to 3, wherein the data is read from the sending buffer of the manager and stored in the receiving buffer of the VM corresponding to the sending instruction Include after: The DMA engine sends first response information to the PF processor, the first response information is used to notify the PF processor that the data sent by the VM corresponding to the received instruction has been successfully stored in the manager middle. 5. The method according to any one of claims 1 to 3, characterized in that, after receiving the sending instruction from the PF processor, the DMA engine includes: The DMA engine detects whether the VM corresponding to the sending instruction is ready to receive the data; If yes, execute the step of reading data from the manager and storing the data in the VM corresponding to the sending instruction. 6. The method according to claim 5, wherein the DMA engine detecting whether the VM corresponding to the sending instruction is ready to receive the data comprises: The DMA engine detects whether the allocated receiving buffer in the VM corresponding to the sending instruction is free to determine whether the VM corresponding to the sending instruction is ready to receive the data, and the allocated receiving buffer in the VM corresponding to the sending instruction Used to store data. 7. The method according to claim 5, characterized in that, after reading data from the manager and storing the data in the VM corresponding to the sending instruction, comprising: The DMA engine sends second response information to the PF processor, and the second response information is used to notify the PF processor that the data has been successfully read from the manager and stored in the corresponding to the sending instruction. in the VM; The DMA engine sends second notification information to the VF processor associated with the VM corresponding to the sending instruction, and the second notification information is used to notify the VF processor associated with the VM corresponding to the sending instruction that the The data of the manager is stored in the VM corresponding to the sending instruction. 8. A direct memory access DMA engine is applied in bus and interface standard PCIe equipment, and said PCIe equipment is connected with a physical machine by a PCIe bus, and said physical machine includes a plurality of virtual machines VM and a manager, multiple The VM and the manager are implemented in the physical machine through virtualization technology; the PCIe device includes the DMA engine, a physical function PF processor and a plurality of virtual function VF processors, and the PF processing The controller and multiple VF processors are implemented in the PCIe device through virtualization technology, wherein one VM is associated with one or more VF processors, and the manager is associated with the PF processor, wherein , the DMA engine consists of: The first processing module is configured to read data from the VM corresponding to the received instruction and store the data in the manager after receiving the received instruction from the PF processor; The second processing module is configured to read data from the manager and store the data in the VM corresponding to the sending instruction after receiving the sending instruction from the PF processor. 9. The DMA engine according to claim 8, wherein the DMA engine further comprises: The first response module is configured to send first response information to the PF processor, where the first response information is used to notify the PF processor that the data sent by the VM corresponding to the received instruction has been successfully stored in the in the manager. 10. The DMA engine according to claim 8 or 9, wherein the DMA engine further comprises: A detection module, configured to detect whether the VM corresponding to the sending instruction is ready to receive the data. 11. DMA engine according to claim 10, is characterized in that, described DMA engine also comprises: The second response module is configured to send second response information to the PF processor, and the second response information is used to notify the PF processor that the data has been successfully read from the manager and stored in the sending instruction In the corresponding VM; and, sending second notification information to the VF processor associated with the VM corresponding to the sending instruction, where the second notification information is used to notify the VF processor associated with the VM corresponding to the sending instruction, The data of the manager has been successfully stored in the VM corresponding to the sending instruction. 12. A system for realizing communication between virtual machines, characterized in that it comprises: a bus and interface standard PCIe device and a physical machine, the PCIe device is connected to the physical machine through the PCIe bus, and the physical machine includes a plurality of virtual machines VM and a manager, the multiple VMs and managers are implemented in the physical machine through virtualization technology; the PCIe device includes a physical function PF processor, multiple virtual function VF processors and direct memory storage Taking the DMA engine, the PF processor and multiple VF processors are implemented in the PCIe device through virtualization technology; wherein, one VM is associated with one or more VF processors, and the manager is associated with all The PF processor; The DMA engine includes: The first processing module is configured to read data from the VM corresponding to the received instruction and store the data in the manager after receiving the received instruction from the PF processor; The second processing module is configured to, after receiving the sending instruction from the PF processor, read data from the manager and store the data in the VM corresponding to the sending instruction; The PF processor is configured to send the receiving instruction and the sending instruction to the DMA engine. 13. The system of claim 12, wherein: The PF processor is also configured to: receive first notification information sent by the DMA engine, the first notification information is used by the DMA engine to notify the PF processor that there is a VM that needs to send data; detect the management Whether the device is ready to receive the data sent by the VM corresponding to the VF processor; if so, perform the step of sending the receiving instruction and the sending instruction to the DMA engine, and the receiving instruction includes the VM that needs to send data. 14. A system according to claim 12 or 13, characterized in that, The PF processor is also used to: detect whether the allocated receive buffer in the manager is free to determine whether the manager is ready to receive data sent by the VM corresponding to the VF processor, and the sending instruction corresponds to The allocated receive buffer in the VM is used to store data. 15. The system of claim 14, wherein: The DMA engine also includes: The first response module is configured to send first response information to the PF processor, where the first response information is used to notify the PF processor that the data sent by the VM corresponding to the received instruction has been successfully stored in the in the manager. 16. The system of claim 12, wherein: The DMA engine also includes: A detection module, configured to detect whether the VM corresponding to the sending instruction is ready to receive the data. 17. The system of claim 16, wherein: The DMA engine also includes: The second response module is configured to send second response information to the PF processor, and the second response information is used to notify the PF processor that the data has been successfully read from the manager and stored in the sending instruction In the corresponding VM; and, sending second notification information to the VF processor associated with the VM corresponding to the sending instruction, where the second notification information is used to notify the VF processor associated with the VM corresponding to the sending instruction, The data of the manager has been successfully stored in the VM corresponding to the sending instruction.