Solid State Drives (SSDs) form the core of modern data center storage, propelled by rapid growth in capacity and improved performance and reliability over hard drives. However, with shrinking feature sizes, capacity gains come at the cost of performance degradation and reduced lifetime. NAND flash’s inability to support in-place updates further degrades performance and lifetime due to the overhead of remapping and garbage collection. Even in SSD-optimized log-structured systems, the lack of coordination can result in misaligned and interleaved logs across the storage stack, requiring garbage collection at multiple levels, increasing wear due to write amplification. Efforts to address these issues require rethinking the SSD interface, leveraging host insights to co-locate related data. Such co-location can improve write isolation across unrelated data streams and reduce write amplification by lowering data movement caused by garbage collection.

However, host-device coordination interfaces have struggled with large-scale adoption. The complexity introduced by integrating interface changes in applications and filesystems, coupled with the difficulty of grouping related data, makes adoption expensive and non-portable. Further, useful placement directives are hard to generate as applications are storage-unaware and filesystems application- unaware. Abstraction breaking changes can increase security risks and limit generalizability. Unless we tackle the complexity of abstraction changes and placement planning head-on, embracing these new interfaces will be limited to proof- of-concept applications or proprietary hyperscalar silos.

In this dissertation, we identify and address the pressing issues that limit host- device coordination: (i) Enabling host-device coordination without large-scale application or filesystem rewrites, (ii) Generating intelligent placement directives based on the application and workload, and (iii) Simplified data placement and storage management for current and future SSD interfaces. We present systems that can coordinate or work independently to support modern interfaces: (i) Shimmer allows applications to support requirements of changing interfaces without modifying the application or the storage system. (ii) Parakeet presents a rich API that can capture data relations and lifetimes and generate heuristic-based or dynamic data placement plans. (iii) Persimmon and shimmer-vfs can manage the placement and maintenance of flash storage and utilize the placement indicators from Parakeet. Our approach enables embracing host-device coordination without requiring changes to data management systems.

Together, these systems eliminate in-place updates and present a highly optimized data path for flash without kernel bypass or application-specific approaches. We demonstrate a 30-90% improvement in write throughput, a reduction in read and write latency, with a factor of 14 reduction in tail latency, and a large reduction in garbage collection overhead over traditional approaches. Further, we demonstrate that shim layers can keep up with highly-tuned per-application approaches, reducing application and filesystem complexity, reducing CPU, memory, and space utilization, and enabling more applications to unlock the full performance of SSDs.