UC Santa Barbara

Deep learning (DL) has achieved impressive results in many tasks across recognition, decision, and content generation, generating significant economic growth and social benefits. The widespread adoption of DL requires efficient systems that can support large-scale training and reduce the cost of inference. As rigid hardware scaling starts to show diminishing returns for DL system performance improvement, the computation efficiency and algorithm efficiency becomes critical. This thesis investigates two key technologies in those two directions respectively, named DL compiler and DL sparsity.

The first part of this thesis tackles the challenge faced by current DL compiler. To bridge the performance gap from compiler-generated DL operators to hand-optimized operators, I present ALCOP: a framework that significantly advances the efficiency of compiler-generated operators by incorporating automatic software pipelining. The rest of this thesis address the challenges faced by current DL sparsity. To resolve the lack of high performance returns in adopting weight-sparse inference, I present ShflBW: a framework to make weight pruning both flexible and efficient on GPUs. To resolve the lack of versatile sparse architecture for all DL workloads, I present RM-STC: a tensor core architecture that harnesses weight and activation sparsity in both inference and training with high speedup and energy efficiency. To explore the acceleration of sparse operators in distributed settings, I present TRACI: a multi-GPU network architecture to accelerate the aggregation operator via in-network computing. This thesis is closed with a discussion about the methodology of designing high-performance DL systems.