HK40035512A - Address caching in switches - Google Patents

Description

Address caching in a switch

Description of the cases

The present application belongs to the divisional application of Chinese patent application No.201680028182.0 with the application date of 2016, 6, month and 24.

Technical Field

This specification relates to switches, e.g., switch chips.

Background

For translation requests and for protection in a system running a virtual machine, an input/output memory management unit (IOMMU) walks page tables to translate device addresses. In some examples, the IOMMU may perform partial page walk caching, e.g., a) to save space when limited memory is available, b) when the IOMMU responds to translation requests to a large number of devices, or both.

Peripheral component interconnect express (pcie) Address Translation Services (ATS) allow devices to request address translations from the IOMMU and cache the translations locally on the device, e.g., in a Translation Lookaside Buffer (TLB). In some examples, the TLB of a device may not be large enough to store all of the virtual to physical address mappings that the device will use.

Disclosure of Invention

A switch chip for a bus, such as a PCI or peripheral component interconnect express (pcie) bus, receives the translated address and uses the translated address to validate memory access requests from devices coupled to the bus that are at least partially managed by the switch chip. For example, the switch chip may store page tables in a cache or a mapping of virtual addresses to physical addresses in a Translation Lookaside Buffer (TLB), and use addresses from the page tables or the TLB to respond to translation requests from devices connected to a bus at least partially managed by the switch chip, validate access requests from these devices, or both. For the sake of brevity, this specification may refer to a device "connected to" the switch chip, which is understood to mean connected to a bus or point-to-point connection that is controlled or managed, at least in part, by the switch chip.

The switch chip may include an input/output memory management unit that stores a page table or a portion of a page table for each device. When the swap chip receives a translation request from a device, the swap chip may provide the translation request to the integrated IOMMU, receive a physical address from the integrated IOMMU as a response, and provide the physical address to the device.

When the switch chip receives a memory access request from the device that includes a physical address, the switch chip uses a page table of the device stored in the integrated IOMMU to verify that the page table includes the physical address, and if the page table includes the physical address, allows the device to access the corresponding memory location. If the page table does not include a physical address, the switch chip prevents the device from accessing the memory location.

In some examples, the switch chip includes a TLB, e.g., a cache, in which the switch chip stores, e.g., per device, a table that maps virtual addresses to physical addresses. The switch chip may associate a particular table with a device using a port number that the device communicates with the switch chip.

When the switch chip receives an address translation request with a particular virtual address from a device, the switch chip determines whether the TLB includes a mapping for the particular virtual address, and if so, provides the corresponding physical address to the device. If not, the switch chip forwards the address translation request to another device, such as a root complex, and receives a response from the other device. The switch chip forwards the response to the requesting device and buffers the response in the TLB. When the switch chip includes a per-device TLB, the switch may look up the mapping only in the particular device's TLB, and store mappings received from other devices in the device's TLB if the mapping is not stored in the TLB.

When the switch chip receives an access request from a device, the switch chip uses the TLB of the device to verify whether the device has permission to access a memory location identified by a physical address included in the access request, e.g., whether the physical address is included in the TLB of the device. If the physical address is in the TLB of the device, the switch chip allows the device to access the memory location. If the physical address is not in the TLB of the device, the switch chip prevents the device from accessing the memory location and, for example, does not forward the request to another device.

In general, one innovative aspect of the subject matter described in this specification can be embodied in systems that include: a switch that receives and delivers packets from and to one or more devices connected to the bus, without any components on the bus between the switch and each device; a memory integrated into the switch, the memory for storing a mapping of virtual addresses to physical addresses; and a non-transitory computer readable storage medium integrated into the switch, the non-transitory computer readable storage medium storing instructions executable by the switch and that, when so executed, cause the switch to perform operations comprising: receiving, by a switch, a response to an address translation request of a device connected by a bus to the switch, the response including a mapping of a virtual address to a physical address; and storing a mapping of virtual addresses to physical addresses in a memory in response to receiving a response to an address translation request for a device. Other embodiments of this aspect include corresponding computer systems, apparatus, methods, and computer programs recorded on one or more computer storage devices, each configured to perform the actions of the described operations. A system of one or more computers may be configured to perform particular operations or actions by virtue of having software, firmware, hardware, or a combination thereof installed on the system that in operation causes the system to perform the actions. The one or more computer programs may be configured to perform particular operations or actions by virtue of comprising instructions that, when executed by a data processing apparatus, cause the apparatus to perform the actions.

In general, one innovative aspect of the subject matter described in this specification can be embodied in systems that include a switch that routes packets across a bus; a memory integrated into the switch, the memory for storing a mapping of virtual addresses to physical addresses; and a non-transitory computer readable storage medium integrated into the switch, the non-transitory computer readable storage medium storing instructions executable by the switch and that, when so executed, cause the switch to perform operations comprising: receiving, by a switch, a response to an address translation request of a device, the response including a mapping of a virtual address to a physical address; storing a mapping of virtual addresses to physical addresses in a memory in response to receiving a response to an address translation request of a device; receiving a memory access request including a particular physical address from a device; determining that a particular physical address is stored in memory; and in response to determining that the particular physical address is stored in the memory, forwarding the memory access request to another apparatus for servicing. Other embodiments of this aspect include corresponding computer systems, apparatus, methods, and computer programs recorded on one or more computer storage devices, each configured to perform the actions of the described operations. A system of one or more computers can be configured by having software, firmware, hardware, or a combination thereof installed on the system that in operation causes the system to perform actions. The one or more computer programs may be configured to perform particular operations or actions by virtue of comprising instructions that, when executed by a data processing apparatus, cause the apparatus to perform the actions.

In general, one innovative aspect of the subject matter described in this specification can be embodied in systems that include a switch that routes packets across a bus; a memory integrated into the switch, the memory for storing a mapping of virtual addresses to physical addresses; and a non-transitory computer readable storage medium integrated into the switch, the non-transitory computer readable storage medium storing instructions executable by the switch and that, when so executed, cause the switch to perform operations comprising: receiving, by a switch, a response to an address translation request of a device, the response including a mapping of a virtual address to a physical address; storing a mapping of virtual addresses to physical addresses in a memory in response to receiving a response to an address translation request of a device; receiving a memory access request including a particular physical address from a device; determining that the particular physical address is not stored in memory; and responsive to determining that the particular physical address is not stored in the memory, discarding the memory access request. Other embodiments of this aspect include corresponding computer systems, apparatus, methods, and computer programs recorded on one or more computer storage devices, each configured to perform the actions of the described operations. A system of one or more computers can be configured by having software, firmware, hardware, or a combination thereof installed on the system that in operation causes the system to perform actions. The one or more computer programs may be configured to perform particular operations or actions by virtue of comprising instructions that, when executed by a data processing apparatus, cause the apparatus to perform the actions.

Optionally, the foregoing and other embodiments may include one or more of the following features, either individually or in combination, respectively. The system may include an input/output memory management unit (IOMMU) integrated into the switch, and the IOMMU includes an IOMMU memory. The memory may comprise an IOMMU memory. Receiving, by the switch, a response to the address translation request of the device may include: a portion of a page table of a device is received by a switch. Storing the mapping of virtual addresses to physical addresses in memory may include: portions of the page table are stored in the IOMMU.

In some embodiments, the memory may include a Translation Lookaside Buffer (TLB). Storing the mapping of virtual addresses to physical addresses in memory may include: a mapping of virtual addresses to physical addresses is stored in the TLB. The system may include a predetermined number of ports of the switch, each port corresponding to a particular device, and the switch including a single TLB for each port. The operations may include: in response to receiving a response to an address translation request of a device, determining that the TLB does not include an empty location; removing the entry from the location in the TLB; and storing a new entry in the location of the TLB, the new entry mapping the virtual address to the physical address. The operations may include: determining that the device includes a TLB and that a setting of the device indicates that the device currently stores a mapping of virtual addresses to physical addresses in the TLB of the device; in response to determining that the device comprises a TLB and that a setting of the device indicates that the device currently stores a mapping of virtual addresses to physical addresses in the TLB of the device, sending a message to the device instructing the device to remove an entry from the TLB of the device; and in response to determining that the device includes a TLB and that the setting of the device indicates that the device currently stores a virtual address to physical address mapping in the TLB of the device, providing the virtual address to physical address mapping to the device. The operations may include: receiving, from the device, confirmation that the device has removed an entry from a TLB of the device, wherein providing the device with a mapping of virtual addresses to physical addresses comprises: in response to receiving confirmation that the device has removed an entry from the TLB of the device, a mapping of virtual addresses to physical addresses is provided to the device.

In some embodiments, the operations may include: receiving a memory access request including a particular physical address from a device; determining to store a particular physical address in memory; and in response to determining that the particular physical address is stored in the memory, forwarding the memory access request to another apparatus for servicing. The memory may be device specific and may not include any virtual to physical address mapping for other devices connected by the bus to the switch. The system may include one or more additional memories integrated into the switch, each of the additional memories and memories being specific to a particular device connected by the bus to the switch. Determining that the particular physical address is stored in the memory may include: determining that a memory access request has been received by a particular port in a switch; selecting a memory of a device using a particular port; and in response to the memory of the device being selected using the particular port, determining that the particular physical address is stored in the memory. The system may include a central processing unit and a cache. Receiving a response to the address translation request of the device may include: a mapping of virtual addresses to physical addresses is received from a cache. Forwarding the memory access request to another apparatus for servicing may comprise: the memory access request is forwarded to the central processing unit. The system may include: a controller on the second bus, the controller connecting the switch to the central processing unit and the cache via the second bus and routing the responses and requests to and from the switch and the central processing unit and the cache using the second bus. The controller may include a root complex. The system may include: a plurality of switches, the plurality of switches comprising a switch; and a controller on the second bus, the controller connecting each of the plurality of switches to the central processing unit and the cache via the second bus, and routing responses and requests to and from each of the plurality of switches and the central processing unit and the cache using the second bus.

In some embodiments, the system includes a motherboard and a bus integrated into the motherboard for which the switch is configured to route requests from a source device to a destination device to allow peripheral devices to connect to the motherboard. The operations may include: receiving a memory access request including a particular physical address from a device; determining that the device is not allowed to access a memory location identified by the particular physical address; and responsive to determining that the device is not allowed to access the memory location identified by the particular physical address, discarding the memory access request. The operations may include: the apparatus is reset in response to determining that the particular physical address is not stored in the memory.

In some embodiments, the operations may include: receiving, by a switch, an address translation request including a virtual address from a device; determining that the virtual address is not stored in memory; and requesting a mapping of the virtual address to the physical address, wherein receiving, by the switch, a response to the address translation request of the device comprises: a response is received in response to requesting a mapping of a virtual address to a physical address. The operations may include: determining a corresponding physical address of the virtual address using a virtual address to physical address mapping; and providing a response to the translation request to the device, the response including the corresponding physical address. The operations may include: it is determined whether the memory includes a virtual address to physical address mapping while requesting a virtual address to physical address mapping. Requesting a mapping of virtual addresses to physical addresses may include: a mapping of virtual addresses to physical addresses is requested from a central processing unit. Requesting a mapping of virtual addresses to physical addresses may include: a mapping of virtual addresses to physical addresses is requested from memory. Requesting a mapping of virtual addresses to physical addresses may include: a mapping of virtual addresses to physical addresses is requested from an input/output memory management unit (IOMMU).

In some embodiments, the system may include: a predetermined number of ports integrated into the switch, each port corresponding to a particular device for which the memory is configured to store a corresponding virtual address to physical address mapping, wherein the size of the memory corresponds to the predetermined number of ports. The switch may be a switching chip.

The subject matter described in this specification can be implemented in particular embodiments and can yield one or more of the following advantages. In some embodiments, a system including a switch chip having a memory storing a mapping of virtual addresses to physical addresses allows the system to scale with the addition of additional switch chips. For example, the switch chip allows the system to include: further devices connected to a bus managed at least in part by a switch chip, further devices connected to other buses managed at least in part by other switch chips, or both. In some embodiments, the systems and methods described below allow the system to respond to address translation requests faster than systems that do not include a switch chip with memory for virtual to physical address mapping. In some embodiments, the systems and methods described below allow the system to verify whether a device has the right to access a particular physical address included in a memory access request from the device.

The details of one or more embodiments of the subject matter described in this specification are set forth in the accompanying drawings and the description below. Other features, aspects, and advantages of the subject matter will become apparent from the description, the drawings, and the claims.

Drawings

FIG. 1 is an example of an environment in which a switch chip stores a mapping of virtual addresses to physical addresses.

FIG. 2 is a flow chart of a process for responding to a request for translation.

FIG. 3 is a flow chart of a process for determining whether a physical address is stored in memory.

Like reference numbers and designations in the various drawings indicate like elements.

Detailed Description

SUMMARY

In some embodiments, a switch chip, such as a peripheral component interconnect express (pcie) switch chip, represents device cache Address Translation Service (ATS) translation connected to a bus managed by the switch chip. For example, when a device issues a Peripheral Component Interconnect (PCI) read or write request on a bus, the switch chip checks a Translation Lookaside Buffer (TLB) included in the switch chip for a virtual address included in the request, and on a hit, translates the virtual address to determine a physical address. The switch chip may provide the physical address to the device or include the physical address in the request and forward the request to another device, e.g., for servicing.

On a TLB miss, for example, the switch chip issues an ATS request to an input/output memory management unit (IOMMU) in the root complex connected to the switch chip through another bus for translation of the virtual address, receives the translation, and buffers the translation in the TLB of the switch chip. In some embodiments, to achieve parallelism, the switch chip may send the untranslated original request to the root complex while checking the TLB for virtual addresses.

The switch chip may intercept PCIe configuration space access to devices connected to the bus managed at least in part by the switch chip to force the ATS capability bit to 1, e.g., an enable ATS capability bit. This may enable the operating system running on the hardware including the switch chip and the device or a device driver in the operating system to enable the ATS of the device. The IOMMU provides a mapping of virtual addresses to physical addresses for the switching chip and the device (if the device supports ATS) in response to address translation requests, read or write access requests, or both, when the ATS of the device is enabled.

In some examples, when a device driver in the operating system enables the ATS of the device, the device driver allows the operating system to whitelist known trusted switch chips. For example, a device driver may allow an operating system to execute on hardware that includes: a switch chip that includes memory (e.g., TLB) that stores virtual to physical address mappings and a switch chip that does not include memory that stores virtual to physical address mappings. The device driver whitelists switch chips that include memory to indicate that the switch chips can receive the mapping, to indicate that requests from the switch chips do not require authentication, or both.

For example, the switch chip may use a map stored in its memory to validate read requests, write requests, or both received from devices connected to a bus managed at least in part by the switch chip. When another hardware device, such as an IOMMU or a processor, receives a request from the switch chip, the other hardware device may use the whitelist to determine whether the request should be validated and whether the device that created the request should be allowed to access the corresponding memory location. When the other hardware device receives a request from a switch chip that is not on the white list, the other hardware device may validate the request to ensure that the device creating the request is not allowed to access memory locations to which the device does not have access, e.g., when the device creating the request is a malicious device, includes errant software, or both.

In some embodiments, a switch chip (e.g., PCIe switch chip) includes the IOMMU and, optionally, page table storage. For example, the switch chip includes a page table walker and onboard memory for storing page tables. The switch chip may receive an address translation request from a device connected to a bus managed at least in part by the switch chip and determine whether a corresponding physical address is stored in a page table. The switch chip provides the corresponding physical address to the requesting device if the corresponding physical address is stored in the page table. If the corresponding physical address is not stored in the page table, the switch chip may forward the request to another device, such as a cache or root complex IOMMU, and store the response received from the other device in a memory of the switch, such as page table storage. The switch may then respond to the original request.

Example System

FIG. 1 is an example of an environment 100 in which a switch chip A102a stores a mapping of virtual addresses to physical addresses. In some embodiments, the switch chip a102a includes a TLB 104, and the switch chip a102a stores the mapping in the TLB 104.

For example, when one of the devices A-C108 a-C is added to the environment 100 and connected to the switch chip A102a through the bus 101, the switch chip A102a assigns a port address to the device, such as device A108 a. When device A108a requests a virtual to physical address translation, the switch chip A102a determines whether the TLB 104 includes a translation for the requested virtual address. The switch chip A102a may initially determine that the TLB 104 does not include any address translations and request translations from the controller 110, e.g., the IOMMU 112 included in the controller 110, or another device such as the processor 114 or the cache 116.

In some embodiments, controller 110 may be a root complex when environment 100 includes a PCIe bus, e.g., bus 101 is a PCIe bus. For example, the root complex may be a switch that routes data between and manages requests to different switch chips A through C102a-C and to C102 a-C.

The switch chip a102a receives the response to the translation request, determines the physical address of the requested virtual address, and provides the physical address to device a108 a. In some examples, the response may include a single virtual address to physical address mapping. In some examples, the response may include a mapping of each of the plurality of different virtual addresses assigned to device a108a to a corresponding physical address to which device a108a has access rights.

When the device a108a is connected to the bus 101 managed by the switch chip a102a, the switch chip a102a may request mapping from the device a108 a. For example, when the device a108a is connected to the bus 101 managed by the switch chip a102a, the switch chip a102a may request a predetermined amount of address translation from another device, such as the controller 110. Address translation may be for low-numbered virtual addresses, high-numbered virtual addresses, virtual addresses most likely to be accessed by device a108a, or a combination of two or more of them.

Once the switch chip A102a stores the virtual to physical address mapping in the TLB 104, the switch chip A102a may respond to an address translation request from device A108a without requesting translation from another device. For example, the switch chip a102a may determine that the TLB 104 includes a mapping of a particular virtual address to a particular physical address for the particular virtual address and provide the particular physical address to the device a108 a.

The switch chip a102a may use the port number of device a108a, the bus device function number, or both to determine the mapping of device a108 a. For example, the TLB 104 may include mappings for device a108a and device B108B, and the switch chip a102a may use the port number and bus device function number of device a108a to only view the mappings for the particular functions of device a108a, but not device B108B.

In some examples, when the switch chip a102a receives a request for address translation from the device a108a, the switch chip a102a may forward the request to another device, such as the controller 110, for servicing, and concurrently determine whether the TLB 104 includes a response to the request. Here, the switch chip a102a does not wait to determine that the TLB 104 does not include the corresponding physical address before forwarding the request to another device for servicing.

For example, when the switch chip A102a cannot store additional mappings in the TLB 104, the switch chip A102a may use any suitable eviction policy to evict an entry from the TLB 104. For example, switch chip A102a may evict the least recently used entry or the most recently used entry.

In an embodiment when the device stores the mapping in its own memory and the switch chip a102a evicts an entry from the TLB 104, the switch chip a102a may send a message to the corresponding device, e.g., device a108a, instructing the device to evict the same entry from the device's memory. For example, when the switch chip A102a receives an address translation request from the device A108a whose TLB 104 does not include its mapping, the switch chip A102a may request translation from the controller 110. The switch chip A102a determines that the TLB 104 does not include any unused memory locations and determines that the least recently used entries should be evicted from the TLB 104. The switch chip a102a stores the translation received from the controller 110 in the newly cleared memory location and sends a message to device a108a instructing device a108a to evict the same least recently used entry from device a's memory. The switch chip a102a may receive an acknowledgement that device a108a evicted the entry. For example, in response to receiving an acknowledgement that device a108a evicted the least recently used entry, switch chip a102a sends the device a108a the translation received from controller 110.

The environment 100 may use an Address Translation Service (ATS) to indicate whether the switch chips A-C102 a-C and the devices A-C108 a-C may store address translation maps in memory. For example, the environment 100 may use the ATS bit to indicate whether device a108a may use ATS and whether device a108a is allowed to use ATS. When device a108a is connected to the bus 101 managed by switch chip a102a, the switch chip a102a may intercept the configuration settings of device a108a, such as PCIe configuration space settings, to "enable" the ATS of device a108a, e.g., set the bit to a value of "1".

In examples where device a108a does not support ATS, the switch chip a102a receives the access request from device a108a and forwards the request to the controller 110 for translation, or may translate one or more virtual addresses in the access request itself. For example, the switch chip a102a may enable the ATS of device a108a, receive an access request from device a108a, determine a virtual address in the request, send a message to the controller 110 of a mapping of the virtual address in the request to a corresponding physical address, receive the mapping, and use the received mapping to translate the virtual address received from device a108a to the corresponding physical address.

In some embodiments, when the device a108a does not support ATS, the switch chip a102a does not enable the ATS bit to indicate that the device a108a supports ATS. For example, when the switch chip A102a does not translate any virtual addresses in the request received from the device A108a to corresponding physical addresses, the switch chip A102a does not enable the ATS bit.

In an example where device a108a supports ATS, the environment 100 may or may not allow device a108a to use the ATS functionality. For example, when the environment 100 allows the device a108a to use the ATS function, the switch chip a102a may check the setting of the device a108a to ensure that the ATS bit is enabled, e.g., set to 1, and in response to determining that the ATS bit is enabled, not modify the ATS bit. Switch chip A102a may receive an address translation request from device A108a and respond to the address translation request as described above.

The switch chip a102a may validate a request from the device a108a, the request including a physical address, e.g., a translated address. For example, when switch chip a102a receives a memory access request from device a108a, switch chip a102a determines that the request is a memory access request and, for example, is not a translation request. The switch chip A102a determines one or more physical addresses included in the request and determines whether those addresses are included in the TLB 104. If these addresses are included in the TLB 104, the switch chip A102a forwards the request to another device, such as the processor 114, the cache 116, or the random access memory 118, for servicing.

If the switch chip A102a determines that one or more of the physical addresses included in the access request from the device A108a are not included in the TLB 104, the switch chip A102a does not forward the access request to another device. For example, the switch chip a102a determines that the device a108a does not have the authority to access a physical address that is not included in the TLB 104, and determines that the access request should not be serviced.

When the switch chip A102a determines that the memory access request should be serviced, the switch chip A102a may forward the request via the root complex. In these examples, the controller 110 may determine that the switch chip a102a verified the request, e.g., using an identifier of the switch chip a102a to determine that the switch chip a102a verified all memory access requests received by the switch chip a102 a. The controller 110 may forward the request to another device, such as a destination device that will service the request, e.g., a device that includes a memory location identified by a physical address in the request.

In an example where the device a108a supports ATS but the environment 100 does not allow the device a108a to use the ATS function, the switch chip a102a processes a request from the device a108a in a similar manner as when the device a108a does not support the ATS function. For example, the switch chip 102a may translate a virtual address included in a memory access request from device a108 a. In some examples, the switch chip 102a may examine the request from device a108a to determine that the request does not include a physical address, e.g., by using any suitable method.

In some embodiments, swap chip a102a includes IOMMU 106, where swap chip a102a stores the memory address in IOMMU 106. For example, switch chip A102a includes IOMMU 106 and does not include TLB 104. The IOMMU 106 includes a memory, such as a page table storage device, in which the IOMMU may store page tables.

When device B108B is connected to bus 101 managed by switch chip a102a, for example, when device B108B is connected to a motherboard using a physical connection including bus 101 and switch chip a102a on a path between device B108B and processor 114, switch chip a102a may request one or more page tables of device B108B. In some examples, switch chip a102a requests page tables from device B108B in response to receiving an address translation request from device B108B, a memory access request from device B108B, or both.

Switch chip A102a responds to the address translation request of device B108B by using another device-specific page table stored in the IOMMU 106 and connected by device B108B, and not, for example, switch chip A102a, via the bus 101. For example, the switch chip A102a receives an address translation request including a virtual address from the device B108B. The switch chip a102a determines whether the IOMMU 106 includes a mapping of virtual addresses to corresponding physical addresses, for example, using a page table walker that accesses page table storage in the IOMMU 106. When the IOMMU 106 includes a mapping, the switch chip a102a determines the corresponding physical address and provides the physical address to device B108B. When IOMMU 106 does not include a mapping, switch chip A102a requests additional page tables from another device, such as cache 116, random access memory 118, or IOMMU 112 in controller 110.

The IOMMU 106 may use an eviction policy for page tables stored in the memory of the IOMMU. For example, when the IOMMU 106 does not include space for another page table, the IOMMU may evict the least recently used page table or the most recently used page table, e.g., for any device connected to the bus 101 managed by the switch chip a102a or for the particular device for which the switch chip a102a stored the new page table in the IOMMU 106.

When the switch chip A102a receives a memory access request from the device B108B, the switch chip A102a may determine whether the memory access request includes a virtual address or a physical address. When the memory access request includes a virtual address, switch chip A102a determines the corresponding physical address by using the page table in the IOMMU 106 and replaces the instance of the virtual address in the memory access request with the corresponding physical address. The switch chip A102a may then forward the memory access request to another device, such as the processor 114, the cache 116, or the random access memory 118 for servicing.

When the switch chip a102a receives a memory access request including a physical address from the device B108B, the switch chip a102a verifies the physical address. For example, switch chip A102a determines the page table of device B108B in IOMMU 106 and determines whether the physical address is included in the determined page table. The switch chip A102a may use the port number or bus device function number of device B108B to determine the page table for device B108B in the IOMMU 106. The switch chip A102a uses the page table of device B108B to ensure that device B108B cannot access physical memory locations that device A108a or another device has access rights but device B108B does not.

In some embodiments, for each predetermined page range, e.g., 4k page range, the environment 100, e.g., switch chip a102a, has a bit indicating whether to map the page range to a device, e.g., whether this is mapped' or two bits indicating read and write permissions. The environment 100 may include a bit identifier, e.g., a single bit or two bits, per device bus device function number, e.g., for each bus device function combination.

When the physical address is included in the page table of device B108B, switch chip A102a forwards the memory access request to another device, such as the processor 114, cache 116, or random access memory 118 for servicing. When the physical address is not included in the page table of device B108B, switch chip a102a does not service the request. In some examples, the switch chip a102a may reset device B108B, suspend device B108B, or both, when the physical address is not included in the page table of device B108B.

In some embodiments, the environment 100 may include a motherboard, a computer, or multiple computers including, for example, a) one or more of the switch chips a-C102 a-C, b) the controller 110, C) the processor 114, d) the cache 116, the random access memory 118, or two or more of these devices. The environment 100 may include an operating system running on hardware, such as a motherboard, a computer, or both.

Enabling the ATS bit of device A108a, e.g., setting the value of the ATS bit to 1, may instruct an operating system running on hardware including switch chip A102a to enable the ATS of device A108a, e.g., to allow device A108a and switch chip A102a to cache a virtual to physical address mapping in the TLB 104. In response to the device a108a being connected to the bus 101 managed by the switch chip a102a, the switch chip a102a may enable the ATS bit of the device a108 a.

In some examples, a device driver included in the software may enable the ATS bit when device a108a is connected to the bus 101 managed by switch chip a108a, e.g., in response to a message from switch chip a102 a. In these examples, the device driver may allow the operating system to whitelist known switch chips that may cache address mappings to allow these switch chips to cache address mappings in a memory of the switch chip, e.g., in the TLB 104.

The environment 100 or operating system may use the white list to determine the functionality of the corresponding switch chip and whether the switch chip should be allowed to perform certain actions, such as caching address mappings in integrated memory, verifying addresses, or both. For example, environment 100 may include different types of switch chips and may determine which actions each of these switch chips is allowed to perform by using a white list.

In some examples, switch chip a102a may include a TLB, switch chip B102B may include an IOMMU, and switch chip C102C may not include either a TLB or an IOMMU. The environment 100 may connect devices using ATS to the bus 101 managed by the switch chip a102a, connect devices using another method to locally cache physical memory addresses on the devices to a second bus managed by the switch chip B102B, and connect devices not locally caching physical memory addresses on the devices to a third bus managed by the switch chip C102C. The environment 100 may allow any suitable type of device to connect to any suitable type of switch chip via a corresponding bus.

Example Address translation Process flow

Fig. 2 is a flow diagram of a process 200 for responding to a translation request. For example, the process 200 may be used by the switch chip a102a from the environment 100.

The switch receives an address translation request (202) from a device that includes a virtual address. For example, the switch receives an address translation request from an ATS-enabled device.

The switch determines that the virtual address is not stored in memory of the switch (204). For example, the switch checks a TLB or IOMMU included in the switch to determine whether the TLB or IOMMU includes a virtual address and a mapping of the virtual address to a corresponding physical address. The switch determines that the TLB or IOMMU does not include a virtual address and a corresponding physical address.

The switch sends an address translation request (206) requesting a mapping of virtual addresses to physical addresses. In some examples, the switch sends an address translation request to another apparatus in response to determining that the virtual address is not stored in the memory. For example, a switch sends an address translation request to the root complex, e.g., when the memory is a TLB, or to a cache or random access memory, e.g., when the memory is an IOMMU.

The switch receives a response to the address translation request, the response including a mapping of the virtual address to the physical address (208). The switch stores a mapping of virtual addresses to physical addresses in memory (210). For example, in response to sending an address translation request, the switch receives a response to the address translation request, e.g., from the root complex, and stores the response or data from the response in memory. The switch may use the data from the response to create an entry in the TLB. In some examples, the switch may receive a portion or all of the page table and store the portion of the page table in the IOMMU.

The switch determines the physical address of the virtual address using a virtual address to physical address mapping (212). For example, the switch uses the responses or data stored in memory to determine the physical address to which the virtual address maps.

The switch provides a response to the translation request including the physical address to the device (214). For example, the switch may forward a response received from the root complex to the device. In some examples, the switch creates a new response including the physical address and provides the new response to the apparatus.

The order of the steps in process 200 described above is merely illustrative and responses to conversion requests may be performed in a different order. For example, prior to determining that the virtual address is not stored in memory, e.g., performing step 204, the switch may send an address translation request, e.g., performing step 206. In some examples, the switch may perform steps 204 and 206 simultaneously.

In some embodiments, process 200 may include additional steps, fewer steps, or some steps may be divided into multiple steps. For example, the switch may receive the address translation request, determine that the virtual address is stored in memory, and determine the physical address using a mapping of the virtual address to the physical address, e.g., not performing steps 204 through 210.

In some embodiments, the switch may perform process 200 or a portion of process 200 for a plurality of different address translation requests. For example, the switch may receive a first address translation request from a first device, determine that a first physical address responsive to the first request is stored in memory, and provide the first physical address to the first device. The switch may receive a second address translation request from the first device or a second, different device, determine that a second physical address responsive to the second request is not stored in memory, e.g., determine that a virtual address from the second request is not stored in memory, and request a mapping of the virtual address from another device, e.g., a root complex or a cache. The switch receives a response to the request from the other device that includes the mapping and provides the second physical address to the requesting device.

The switch may include separate memory for each of the devices for which the switch stores a mapping of virtual addresses to physical addresses. In some examples, a switch may include both a TLB and an IOMMU.

Example verification Process flow

FIG. 3 is a flow diagram of a process 300 for determining whether a physical address is stored in memory. For example, the process 300 may be used by the switch chip a102a or the controller 110 from the environment 100.

The switch receives a memory access request from a device that includes a particular physical address (302). For example, a switch receives a read or write request.

The switch determines whether the particular physical address is stored in memory of the switch (304). The memory may be a TLB or IOMMU. The switch validates the request by determining whether the particular physical address is stored in memory and whether the device is allowed to access the particular physical address. For example, when a switch has one memory for each of the devices for which the switch stores a mapping of virtual addresses to physical addresses, the memories may be device specific. The memory may be used for all devices connected by the switch via the corresponding bus.

The switch determines that the particular physical address is stored in memory (306). If the switch determines that a particular physical address is stored in the memory of the device, the switch validates the request when there is a single memory for each device. The switch validates the request if the switch determines that the particular physical address is stored in a single memory for all devices and determines that the devices are allowed to access the particular physical address.

In response to determining that the particular physical address is stored in the memory of the switch, the switch forwards the memory access request to another device for service (308). For example, the switch forwards the request to a destination device having data responsive to the request. The forwarding request may route the request through one or more intermediary devices. In some examples, when the cache contains data responsive to the request, the switch may send the request to the cache via the root complex.

The switch determines that the particular physical address is not stored in memory (310). For example, the switch determines that the device is not allowed to access the memory location identified by the particular physical address. The switch does not validate the request when a particular physical address is stored in another of the switches memory not targeted for the device, when each device has its own memory, or when a particular physical address is stored in a single memory targeted for all devices but the particular physical address is not associated with the device, e.g., the device should not be allowed to access the particular physical address.

In response to determining that the particular physical address is not stored in memory of the switch, the switch discards the memory access request (312). For example, the switch does not forward the request to another device for service or respond to the request, or does not do either. In some examples, the switch may log an error identifying details of the discarded memory access request, e.g., a particular physical address not stored in memory of the switch and an identifier of the source device that sent the request to the switch.

The switch resets the means (314) in response to determining that the particular physical address is not stored in the memory of the switch. For example, the switch may determine that the device is a malicious device, has a software bug, or both, and reset the device, e.g., to prevent the device from accessing memory locations to which the device does not have access rights.

The order of the steps in process 300 described above is merely illustrative and determining whether a physical address is stored in memory may be performed in a different order. For example, the switch may reset the device and then discard the memory access request, e.g., perform step 314 and then perform step 312.

In some embodiments, process 300 may include additional steps, fewer steps, or some steps may be divided into multiple steps. For example, a switch may perform a portion or all of process 300 for a plurality of different memory access requests, a plurality of different types of memory access requests, such as read, write, or both, a plurality of different devices connected to a bus managed by the switch, or a combination of two or more of them. The switch may authenticate some of the requests, invalidate some of the requests, or both authenticate some of the requests and invalidate some of the requests.

In some embodiments, another device with a TLB or IOMMU validates requests. For example, the root complex may perform process 300 to validate the request.

Details of alternative embodiments

In some embodiments, the switch chip may include a TLB for each of the devices for which the switch chip stores address mappings, e.g., a TLB for each port that allows devices to connect to a bus managed by the switch chip and for which the switch chip maps addresses. The switch chip may use the port information to determine which TLB should be accessed, e.g., to determine an address mapping for a particular device or to validate a memory access request for a particular device. The switch chip may use any suitable method to determine which TLB to access for a particular device.

In some implementations, the switch chip may use an identifier, such as a port identifier, to identify the device and determine whether the device may cache the address mapping locally on the device. For example, the switch chip may include a bit for each port indicating whether a device connected to the port via the bus supports ATS and whether the switch chip should respond to an address translation request from the device.

The size of the memory, e.g., IOMMU, in the switch chip may be selected based on the number of devices for which the memory stores address mappings, e.g., the number of ports in the switch chip when each port corresponds to a device that may be connected to a bus managed by the switch chip and for which the IOMMU stores page tables. For example, a switch chip with functionality to connect via a bus with up to 16 devices and store its page tables may include a larger IOMMU than another switch chip with functionality to connect via another bus with up to 4 devices and store its page tables.

In some implementations, the IOMMU may store a percentage of the total number of page tables for one or more devices. For example, when a particular device may access a large number of memory locations, the IOMMU may store some, but not all, of the page tables of the particular device.

The switch chip may allocate a predetermined percentage of the page table storage of the IOMMU for each of the devices for which the IOMMU stores page tables. For example, when the switch chip may connect to up to 16 devices via a bus and store their page tables, and one device connects to the bus managed by the switch chip, the switch chip may allocate the entire IOMMU to store the page tables of the one device. When another device is connected to the bus managed by the switch chip, the switch chip may allocate half of the memory in the IOMMU to each device, and so on for additional devices. In some embodiments, the IOMMU may allocate one sixteenth of the memory to each device, regardless of how many devices are actually connected to the bus managed by the switch chip.

In some embodiments, the switch chip may include a Memory Management Unit (MMU) instead of an IOMMU. For example, a switch chip may use an MMU to translate virtual addresses to physical addresses of devices connected to a bus managed by the switch chip.

Additional implementation details

Embodiments of the subject matter described in this specification and the functional operations may be implemented in digital electronic circuitry, in tangibly embodied computer software or firmware, in computer hardware, including the structures disclosed in this specification and their structural equivalents, or in combinations of one or more of them. Embodiments of the subject matter described in this specification can be implemented as one or more computer programs, i.e., one or more modules of computer program instructions, encoded on a tangible, non-transitory program carrier for execution by, or to control the operation of, data processing apparatus. The computer storage medium may be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination of one or more of them.

The term "data processing apparatus" refers to data processing hardware and encompasses all kinds of apparatus, devices, and machines for processing data, including by way of example a programmable processor, a computer, or multiple processors or computers. The apparatus can also be or further include special purpose logic circuitry, e.g., an FPGA (field programmable gate array) or an ASIC (application-specific integrated circuit). In addition to hardware, the apparatus can optionally include code that creates an execution environment for the computer program, e.g., code that constitutes processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of them.

A computer program, which can also be referred to or described as a program, software, a software application, a module, a software module, a script, or code, can be written in any form of programming language, including compiled or interpreted languages, or declarative or procedural languages, and can be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. The computer program may, but need not, correspond to a file in a file system. A program can be stored in a portion of a file that holds other programs or data, such as one or more scripts stored in a markup language document, in a single file dedicated to the program in question, or in multiple coordinated files, such as files that store one or more modules, sub programs, or portions of code. A computer program can be deployed to be executed on one computer or on multiple computers that are located at one site or distributed across multiple sites and interconnected by a communication network.

The processes and logic flows described in this specification can be performed by one or more programmable computers executing one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows can also be performed by, and apparatus can also be implemented as, special purpose logic circuitry, e.g., an FPGA (field programmable gate array) or an ASIC (application-specific integrated circuit).

Computer readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media and memory devices, including by way of example: semiconductor memory devices such as EPROM, EEPROM, and flash memory devices; magnetic disks, such as internal hard disks or removable disks 'magneto-optical disks' CD-ROM disks and DVD-ROM disks. The processor and the memory can be supplemented by, or incorporated in, special purpose logic circuitry.

While this specification contains many specific implementation details, these should not be construed as limitations on the scope of what may be claimed, but rather as descriptions of features that may be specific to particular embodiments. Certain features that are described in this specification in the context of separate embodiments can also be implemented in combination in a single embodiment. Conversely, various features that are described in the context of a single embodiment can also be implemented in multiple embodiments separately or in any suitable subcombination. Moreover, although features may be described above as acting in certain combinations and even initially claimed as such, one or more features from a claimed combination can in some cases be excised from the combination, and the claimed combination may be directed to a subcombination or variation of a subcombination.

Also, while operations are shown in the drawings in a particular order, this should not be understood as requiring that such operations be performed in the particular order described or in sequential order, or that all illustrated operations be performed, to achieve desirable results. In some cases, multitasking and parallel processing may be advantageous. Moreover, the separation of various system modules and components in the embodiments described above should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

Specific embodiments of the present subject matter have been described. Other embodiments are within the scope of the following claims. For example, the actions recited in the claims can be performed in a different order and still achieve desirable results. As one example, the processes depicted in the accompanying figures do not necessarily require the particular order shown, or sequential order, to achieve desirable results. In some cases, multitasking and parallel processing may be advantageous.

Claims (24)

1. A system, comprising:

a switch to receive packets from and deliver packets to one or more devices connected to a bus without any components on the bus between the switch and each of the one or more devices;

a memory integrated into the switch to store a mapping of virtual addresses to physical addresses;

one or more additional memories integrated into the switch, each of the one or more additional memories and the memory specific to a particular device connected to the switch through the bus; and

a non-transitory computer readable storage medium integrated into the switch, the non-transitory computer readable storage medium storing instructions executable by the switch and upon such execution cause the switch to perform operations comprising:

receiving, by the switch, a memory access request including a particular physical address from a device connected to the switch through the bus;

in response to receiving the memory access request that includes the particular physical address, retrieving, by the switch, a mapping of virtual addresses to physical addresses from the memory;

determining that the particular physical address is stored in the memory using the mapping of the virtual address to a physical address, comprising:

determining that a particular port in the switch receives the memory access request;

selecting the memory for the apparatus from a group comprising the memory and the one or more additional memories using the particular port; and

in response to selecting the memory for the device using the particular port, determining that the particular physical address is stored in the memory; and

in response to determining that the particular physical address is stored in the memory, forwarding the memory access request to another apparatus for servicing.

2. The system of claim 1, wherein retrieving the mapping of the virtual address to the physical address from the memory comprises: retrieve the mapping of the virtual address to the physical address received in response to a request for an address translation request for the device from the memory.

3. The system of claim 1, wherein:

the memory comprises a translation lookaside buffer; and is

Retrieving the mapping of the virtual address to the physical address from the memory comprises: the mapping of the virtual address to the physical address is fetched from the translation lookaside buffer.

4. The system of claim 3, comprising:

a predetermined number of ports of the switch, each of the ports corresponding to a particular device, wherein the memory comprises a predetermined number of translation lookaside buffers, each of the predetermined number of translation lookaside buffers corresponding to one of the particular devices.

5. The system of claim 3, the operations comprising:

receiving, by the switch, a response to an address translation request for the device, the response comprising the mapping of the virtual address to the physical address;

in response to receiving the response to the address translation request for the device, determining that the translation lookaside buffer does not include an empty location;

removing an entry from a location in the translation lookaside buffer; and

storing a new entry in the location of the translation lookaside buffer, the new entry mapping the virtual address to the physical address.

6. The system of claim 3, the operations comprising:

determining that a setting specific to the device indicates that the device is currently storing a virtual address to physical address mapping in a translation lookaside buffer on the device;

in response to determining that the setting specific to the device indicates that the device currently stores a virtual address to physical address mapping in the translation lookaside buffer on the device, sending a message to the device instructing the device to remove an entry from a translation lookaside buffer of the device; and

in response to determining that the setting specific to the device indicates that the device currently stores a virtual address to physical address mapping in the translation lookaside buffer on the device, providing the mapping of the virtual address to the physical address to the device.

7. The system of claim 6, the operations comprising:

receiving, from the apparatus, an acknowledgement that the apparatus has removed the entry from a translation lookaside buffer of the apparatus, wherein providing the mapping of the virtual address to the physical address to the apparatus comprises: providing the mapping of the virtual address to the physical address to the device in response to receiving the acknowledgement that the device has removed the entry from a translation lookaside buffer of the device.

8. The system of claim 1, wherein:

the switch receiving packets from and delivering packets to two or more devices including the device, each of the two or more devices being connected to a bus between the switch and the respective device without any components on the bus;

the memory is specific to the device and does not include any virtual to physical address mappings for other devices connected to the switch by the bus and included in the two or more devices; and is

The system comprises:

one or more additional memories, each of the one or more additional memories: a) a particular device, other than the device, that is specific to the two or more devices, and b) does not include any virtual to physical address mapping for other devices connected to the switch through the bus and included in the two or more devices.

9. The system of claim 1, comprising:

a central processing unit; and

a cache, wherein:

receiving, by the switch, a response to an address translation request for the device from the cache, the response comprising the mapping of the virtual address to the physical address; and

forwarding the memory access request to another apparatus for servicing comprises: forwarding the memory access request to the central processing unit.

10. The system of claim 9, comprising:

a controller on a second bus, the controller connecting the switch to the central processing unit and the cache via the second bus and routing responses and requests to and from the switch and the central processing unit and the cache using the second bus.

11. The system of claim 10, wherein the controller comprises a root complex.

12. The system of claim 9, comprising:

a plurality of switches, the plurality of switches including the switch; and

a controller on a second bus, the controller connecting each of the plurality of switches to the central processing unit and the cache via the second bus and routing responses and requests to and from each of the plurality of switches and the central processing unit and the cache using the second bus.

13. The system of claim 1, comprising:

a motherboard; and

the bus integrated into the motherboard, the switch configured to route requests from a source device to a destination device for the bus to allow peripheral devices to connect to the motherboard.

14. The system of claim 1, the operations comprising:

receiving, by the switch, an address translation request including the virtual address from the device;

determining that the virtual address is not stored in the memory;

sending the address translation request for the mapping of the virtual address to the physical address;

in response to sending the address translation request for the mapping of the virtual address to the physical address, receiving the response to the address translation request; and

storing the mapping of the virtual address to the physical address in the memory in response to receiving the response to the address translation request for the device.

15. The system of claim 14, the operations comprising:

determining a corresponding physical address of the virtual address using the mapping of the virtual address to the physical address; and

providing a response to the address translation request to the device, the response including the corresponding physical address.

16. The system of claim 14, the operations comprising:

determining whether the memory includes the mapping of the virtual address to the physical address while sending the address translation request for the mapping of the virtual address to the physical address.

17. The system of claim 14, wherein sending the address translation request for the mapping of the virtual address to the physical address comprises: requesting, from a central processing unit, the mapping of the virtual address to the physical address.

18. The system of claim 14, wherein sending the address translation request for the mapping of the virtual address to the physical address comprises: requesting the mapping of the virtual address to the physical address from memory.

19. The system of claim 14, wherein sending the address translation request for the mapping of the virtual address to the physical address comprises: requesting the mapping of the virtual address to the physical address from an input/output memory management unit (IOMMU).

20. The system of claim 1, comprising:

a predetermined number of ports integrated into the switch, each of the ports corresponding to a particular device for which the memory is configured to store a corresponding virtual address to physical address mapping, wherein the size of the memory corresponds to the predetermined number of ports.

21. A system, comprising:

a switch to receive packets from and deliver packets to one or more devices connected to a bus without any components on the bus between the switch and each of the one or more devices;

a memory integrated into the switch to store a mapping of virtual addresses to physical addresses;

one or more additional memories integrated into the switch, each of the one or more additional memories and the memory specific to a particular device connected to the switch through the bus; and

a non-transitory computer readable storage medium integrated into the switch, the non-transitory computer readable storage medium storing instructions executable by the switch and upon such execution cause the switch to perform operations comprising:

receiving, by the switch, a memory access request including a particular physical address from a device connected to the switch through the bus;

in response to receiving the memory access request that includes the particular physical address, retrieving, by the switch, a mapping of virtual addresses to physical addresses from the memory;

determining, using the mapping of the virtual address to a physical address, that the device is not allowed to access a memory location identified by the particular physical address, comprising:

determining that a particular port in the switch receives the memory access request;

selecting the memory for the apparatus from a group comprising the memory and the one or more additional memories using the particular port; and

in response to selecting the memory for the device using the particular port, determining that the device is not allowed to access the memory location; and

in response to determining that the device is not allowed to access the memory location identified by the particular physical address:

discarding the memory access request; and

the device is reset.

22. The system of claim 21, wherein determining that the device is not allowed to access the memory location identified by the particular physical address comprises: determining that the particular physical address is not stored in the memory.

23. The system of claim 21, wherein:

receiving, by the switch, the memory access request including the particular physical address from the device connected to the switch through the bus comprises:

receiving the memory access request including an address; and

determining that the address included in the memory access request is a physical address; and

retrieving, by the switch, the mapping of the virtual address to the physical address from the memory is in response to determining that the address included in the memory access request is a physical address.

24. A computer-implemented method, comprising:

receiving, by a switch that receives a packet from one or more devices connected to a bus and delivers the packet to the one or more devices connected to the bus without any components on the bus between the switch and each of the one or more devices, a memory access request that includes a particular physical address;

in response to receiving the memory access request that includes the particular physical address, retrieving, by the switch, a virtual address to physical address mapping from memory integrated into the switch to store the virtual address to physical address mapping;

determining that the particular physical address is stored in the memory using the mapping of the virtual address to a physical address, comprising:

determining that a particular port in the switch receives the memory access request;

using the particular port to select the memory for the device from a group comprising the memory and one or more additional memories integrated into the switch, wherein each of the one or more additional memories and the memory is specific to a particular device connected to the switch through the bus; and

in response to selecting the memory for the device using the particular port, determining that the particular physical address is stored in the memory; and

in response to determining that the particular physical address is stored in the memory, forwarding the memory access request to another apparatus for servicing.