Abstract
As computer systems embrace heterogeneity in the memory system, applications decrease from non-uniform memory access (NUMA) effects. These mandates careful data placement for performance. However, performance optimizations on such systems have predominantly focused on user-space data for decades, while the layout of kernel data structures has not been considered crucial. Consequently, most kernel data structures are pinned and unmovable in a typical OS design. In this paper, we show that careful placement of kernel data structures is gaining importance in the context of page-tables: sub-optimal placement of page-tables causes severe slowdown (up to 3.1×) on virtualized NUMA servers.
Mirage is the first system to provide explicit NUMA management for page-tables on virtualized platforms for both the guest and extended page-tables. It enables faster address translation by migrating and replicating page-tables and supports two common virtualization configurations: a) the hypervisor exposes the NUMA architecture to the guest OS, and b) such information is hidden from the guest OS. Mirage is implemented in Linux/KVM, and our evaluation on a recent 1.5TB four-socket server shows that it effectively eliminates NUMA effects on 2D page-walks resulting in a speedup of 1.8 − 3.1× for Thin and 1.06 − 1.6× for Wide workloads respectively.