Context-aware address translation for high performance SMP cluster system
Recent work in low-latency, high-bandwidth communication systems has resulted in building user--level Network Interface Controllers (NICs) and communication abstractions that support direct access from the NIC to applications virtual memory to avoid both data copies and operating system intervention. Such mechanisms require the ability to directly manipulate user--level communication buffers for delivering data and achieving protection. To provide such abilities, NICs must maintain appropriate translation data structures. Most user--level NICs manage these data structures statically, which results both in high memory requirements for the NIC and limitations on the total size and number of communication buffers that a NIC can handle.In this paper, we categorize the types of data structures used by NICs and propose <i>dynamic handle lookup</i> as a mechanism to manage such data structures dynamically. We implement our approach in a modern, user--level communication system and evaluate our system, <i>miNL</i>, with both micro-benchmarks and real applications. We also study the impact of various cache parameters on system performance. We find that, with appropriate cache tuning, our approach reduces the amount of NIC memory required in our system by a factor of two for the total NIC memory and by more than 80% for the lookup data structures. Moreover, by pinning physical memory automatically and on demand, our approach eliminates the limitations and complexities imposed by static memory pinning that is used in most user--level communication systems. Our approach increases execution time by at most 3% for all but one applications we examine.