Abstract
We present TritonSort, a highly efficient, scalable
sorting system. It is designed to process large datasets,
and has been evaluated against as much as 100 TB of input
data spread across 832 disks in 52 nodes at a rate of 0.916
TB/min. When evaluated against the annual Indy GraySort
sorting benchmark, TritonSort is 60% better in absolute
performance and has over six times the per-node efficiency
of the previous record holder. In this paper, we describe
the hardware and software architecture necessary to operate
TritonSort at this level of efficiency. Through careful
management of system resources to ensure cross-resource
balance, we are able to sort data at approximately 80% of
the disks’ aggregate sequential write speed.
We believe the work holds a number of lessons for balanced
system design and for scale-out architectures in general.
While many interesting systems are able to scale linearly
with additional servers, per-server performance can
lag behind per-server capacity by more than an order of
magnitude. Bridging the gap between high scalability and
high performance would enable either significantly cheaper
systems that are able to do the same work or provide the
ability to address significantly larger problem sets with the
same infrastructure.