Electrical computers and digital processing systems: memory – Storage accessing and control – Shared memory area
Reexamination Certificate
2001-11-15
2004-06-08
Nguyen, Hiep T. (Department: 2187)
Electrical computers and digital processing systems: memory
Storage accessing and control
Shared memory area
C711S152000, C711S163000
Reexamination Certificate
active
06748499
ABSTRACT:
BACKGROUND OF THE INVENTION
1. Technical Field
The present invention relates to memory access in computer systems, and more specifically, how to efficiently provide shared access from multiple host channel adapters (HCA) to particular areas of memory.
2. Description of Related Art
In a System Area Network (SAN), the hardware provides a message passing mechanism which can be used for Input/Output devices (I/O) and interprocess communications between general computing nodes (IPC). Consumers access SAN message passing hardware by posting send/receive messages to send/receive work queues on a SAN channel adapter (CA). The send/receive work queues (WQ) are assigned to a consumer as a queue pair (QP). The messages can be sent over five different transport types: Reliable Connected (RC), Reliable datagram (RD), Unreliable Connected (UC), Unreliable Datagram (UD), and Raw Datagram (RawD). Consumers retrieve the results of these messages from a completion queue (CQ) through SAN send and receive work completions (WC). The source channel adapter takes care of segmenting outbound messages and sending them to the destination. The destination channel adapter takes care of reassembling inbound messages and placing them in the memory space designated by the destination's consumer. Two channel adapter types are present, a host channel adapter (HCA) and a target channel adapter (TCA). The host channel adapter is used by general purpose computing nodes to access the SAN fabric. Consumers use SAN verbs to access host channel adapter functions. The software that interprets verbs and directly accesses the channel adapter is known as the channel interface (CI).
A Memory Region is an area of memory that is contiguous in the virtual address space and for which the translated physical addresses and access rights have been registered with the HCA. Memory Registration is used to pin a Memory Region and make it accessible by the HCA. The HCA's software is provided with the Memory Region's protection domain (used to associate the memory region to a QP), access rights (local write, remote write, remote read), length, and virtual address. The HCA software translates the virtual address and length to the set of physical pages mapped to the virtual region. The HCA software then makes all of this information available to the HCA. This process requires considerable overhead, including kernel intervention, in order to set up the access rights, pinning or unpinning of memory, and setting up the address translation tables.
Unfortunately memory protection and address translation tables cannot be shared between CAs. Hence, nodes which support mulitple CAs will often have needlessly redundant entries in the memory protection and address translation tables, thus wasting valuable memory resources. Therefore, it would be desirable to provide a mechanism to allow sharing of these memory tables between channel adapters.
SUMMARY OF THE INVENTION
The present invention provides a method, computer program product, and data processing system for sharing memory protection tables and address translation tables among multiple Host Channel Adapters.
This invention utilizes a “Register Shared Memory Region” verb (command). When a particular memory region is to be shared among multiple HCAs, the consumer issues a “Register Shared Memory Region” verb to HCA software specifying the HCA, virtual address, length, protection domain, and access control parameters for each instance of this verb. The consumer further identifies the particular Memory Region that is to be shared. The HCA software returns to the consumer an “R_Key” value that must be associated with all queue pairs of each HCA that will access this shared memory region. The HCA software also keeps track of the HCAs that have been given access to the shared Memory Region so that it can determine when it is safe to deregister the Memory Region. Additionally, in response to the “Register Shared Memory Region” verb, the HCA software creates and manages an “HCA Access List” for each shared Memory Region. The HCA Access List identifies the HCAs that are granted access to the particular shared Memory Region.
The invention also employs a set of hardware registers within the HCA that point to the location of the memory protection and address translation tables in system memory. When the HCA software registers a memory region, it loads these hardware registers with the system memory location of the protection and address translation tables. Each HCA includes a hardware register containing the HCA's “HCA Identifier (HCA ID).” HCA software loads a unique value into this register prior to the HCA accessing any portion of either a shared or non-shared Memory Region.
When the HCA accesses a virtual address, via an RDMA (remote direct memory access) operation for instance, the HCA fetches the protection and address translation data from the tables located in system memory. The HCA checks the values of the protection domain for the memory region to ensure it matches the protection domain value of the queue pair so that it may access the memory region in question. The HCA further checks the HCA Access List for an instance of its HCA ID to determine whether this particular HCA has been granted access to this shared Memory Region.
Since the protection tables and address translation tables are located within the system memory of the host and thus are accessible by all HCAs attached to that host, sharing of the memory region can be accomplished without duplication of the protection and address translation tables.
When the consumer requests that a memory region be de-registered (via the “Deregister Memory Region” verb), it supplies the HCA software with the particular memory region that is being de-registered as well as the HCA from which it is being deregistered. The HCA software will invalidate the hardware registers that contain the system memory location of the protection and address translation tables associated with the shared memory region. The HCA software will further note that this particular HCA is no longer registered with the shared Memory Region, and will remove the HCA ID of this particular HCA from the HCA Access List associated with this Memory Region. The HCA software will not modify the system memory locations that contain the protection tables and address translation tables unless it determines that there are no HCAs currently using these facilities.
REFERENCES:
patent: 2002/0091841 (2002-07-01), Beukema et al.
patent: 2002/0124148 (2002-09-01), Beukema et al.
patent: 2003/0051103 (2003-03-01), Lardner et al.
Beukema Bruce Leroy
Craddock David F.
Gregg Thomas Anthony
Recio Renato John
Nguyen Hiep T.
Nichols Michael R.
Salys Casimer K.
Yee Duke W.
LandOfFree
Sharing memory tables between host channel adapters does not yet have a rating. At this time, there are no reviews or comments for this patent.
If you have personal experience with Sharing memory tables between host channel adapters, we encourage you to share that experience with our LandOfFree.com community. Your opinion is very important and Sharing memory tables between host channel adapters will most certainly appreciate the feedback.
Profile ID: LFUS-PAI-O-3315815