Performance models for storage devices are an important part of simulations of large-scale computing systems. Storage devices are traditionally modeled using discrete event simulation. However, this is expensive in terms of computation, memory, and configuration. Configuration alone can take months, and the model itself requires intimate knowledge of the internal layout of the device. The difficulty in white-box model creation has led to the current situation, where there are no current, precise models. Automatically learning device behavior is a much more desirable approach, requiring less expert knowledge, fewer assumptions, and less time. Other researchers have created behavioral models of storage device performance, but none have shown low per-request errors. By making use of only a few high-level domain-specific assumptions, such as spatial periodicity and the existence of a logical-to-physical mapping, neural nets can learn to predict access time with low per-request errors. Providing neural nets with specific sinusoidal inputs allows them to generate periodic output, which is necessary for this problem. Weight sharing in the neural net accounts for regularities in the structure of the input, which reduces data requirements and error by reducing the number of free parameters. A trigonometric change of variables in the output of the neural net removes a discontinuity in the objective function, which makes the problem more amenable to neural nets and reduces error. Combining these approaches, we demonstrate that a neural net can predict access times with a mean absolute error of about 0.187 ms over a small portion of a hard disk drive, and a mean absolute error of about 2.120 ms over half of a hard disk drive.

Reproducibility is the cornerstone of the scientific method. Yet, in the computational and data-intensive branches of science, a gap exists between current practices and the ideal of having every new scientific discovery be _easily_ reproducible. At the root of this problem are the dysfunctional forms of communication between the distinct stakeholders of science: researchers, their peers, students, librarians and other consumers of research outcomes working in ad-hoc ways; groups of individuals organized as independent silos, sharing minimal information between them, all of them with the common task of publishing, obtaining, re-executing and validating experimentation pipelines associated to scientific claims contained in scholarly articles and technical reports. This dissertation characterizes the practical challenges associated to the research lifecycle (creation, dissemination, validation, curation and re-use of scientific explorations) and draws analogies with similar problems experimented by software engineering communities in the early 2000's. DevOps, the state-of-the-art software delivery methodology followed by companies and open source communities, appeared in late 2000's and addresses these analogous issues.

By framing the problem of research delivery (iterating the research lifecycle) as a problem of software delivery, it becomes possible to repurpose the DevOps methodology to address the practical challenges faced by experimenters across the domains of computational and data-intensive science. This thesis presents Popper, an experimentation protocol for writing articles and carrying out scientific explorations following DevOps principles. Popper brings agility to research delivery in a domain-agnostic way by considering the end-to-end research delivery cycle, making it easier for all the stakeholders of science to publish, access, re-execute and validate experimentation pipelines. This dissertation also presents reusable tools in the domain of computer systems research. In a research delivery context, this toolset covers the multiple phases of the research lifecycle and helps systems research practitioners carry out validations of scientific claims in this domain in an agile manner.

For this work, I was awarded the 2018 Better Scientific Software Fellowship, an initiative from the US Department of Energy whose goal is to foster best software development practices among the US scientific community. The Popper CLI tool, and its associated education materials, are currently being used to train new generations of scientists on how to be aware and guard against the multiple practical challenges in reproducibility.

Scientific workflows contain an increasing number of interacting

applications, often with big disparity between the formats of data

being produced and consumed by different applications. This mismatch

can result in performance degradation as data retrieval causes

multiple read operations (often to a remote storage system) in order

to convert the data. In recent years, with the large increase in the

amount of data and computational power available there is demand for

applications to support data access in-situ, or close-to simulation to

provide application steering, analytics and visualization.

Although some parallel filesystems and middleware

libraries attempt to identify access patterns and optimize data

retrieval, they frequently fail if the patterns are complex. It is

evident that more knowledge of the structure of the datasets at the

storage systems level will provide many opportunities for further

performance improvements.

For most developers of scientific applications, storing the

application data, and its particular format on disk, is not an

essential part of the application. Although they acknowledge the

importance of the I/O performance, their expertise lies mostly in

numerical simulations and the particular models their application

simulates. Most of their efforts are spent of ensuring that the

it produces correct numerical results. Ideally, they would like to be

able to have a library call that reads a subset of the data from storage (no

matter what its format is), and place it in the data structures the

simulation defines in the computer memory. Since the data needs to be

analyzed and visualized, and the data has to be accessible from

third-party tools, the scientists are forced to know more about the

data formats.

In this dissertation we investigate multiple techniques for utilizing

dataset description for improving performance and overall data

availability for HPC applications. We introduce a declarative data

description language that can be used to define the complete dataset

as well as parts of it. These descriptions are used to generate

transformation rules that allow data to be converted between different

physical layouts on storage and in memory.

First, we define the DRepl dataset description language and use it to

implement divergent data views and replicas as POSIX files. We

evaluate the performance for this approach and demonstrate its

advantages both because of the transparent application use, and

combined performance when the application is combined with analytics

and/or visualization code that reads the data in different format.

DRepl decouples the data producers and consumers and the data layouts

they use from the way the data is stored on the storage system.

DRepl has shown up to 2x for cumulative performance when data is

accessed using optimized replicas.

Second, we extend the previous approach to the parallel environment

used in HPC. Instead of using POSIX files, the new method allows data

to be accessed in larger chunks (fragments) in the way it will be laid

out in memory. The developers can define what data structures they

have in the process' memory and the overall format of the dataset on

storage, and the runtime will automatically take care of transforming

the data between the two. Both the formats in memory and on disk are

described with the DRepl language. Replacing the ability for reading

the data as an array of bytes with operations that use descriptions of

the data structure, provides better opportunities for the

storage system to optimize the access to the persistent data. The

integration of this technique in Ceph demonstrates the potential

advantages for this approach. The experiments show performance

improvements up to 5 times for writes and 10 times for reads, compared

to collective MPI I/O.

Third, we explore the future directions of extending the DRepl

language to support more complex datasets. The additions would allow

scientists to use different resolutions for different parts of a

multi-dimensional spaces, and define how to transform the data between

resolutions. The changes would also allow completely abstract

definitions of datasets not only for continuums, but also for

primitive types like real and integer numbers. The fragments of the

dataset that are present in memory or disk will have concrete

types that are compatible with the abstract types used in the dataset.

Finally, we provide foundations on how to extend the previous

functionality to the most complicated data structures used in

scientific applications -- unstructured meshes.

Storage system solutions have historically been dominated by proprietary offerings designed around a fixed set of common interfaces such as the POSIX file abstraction. However, as the scalability requirements of applications have grown, these interfaces and their implementations have not kept pace. This has forced developers to rely on middleware and external services to address these limitations. These solutions introduce new complexity into the system in the form of duplicated software and can lead to increased costs and reduced reliability. However, the recent availability of high-performance open-source storage systems is allowing developers to explore alternative storage interfaces that directly meet the needs of applications without the fear of vendor lock-in.

We introduce programmable storage as a means by which existing internal storage system abstractions can be generalized and reused to support applications through the creation of domain-specific interfaces. By reusing internal, code-hardened sub-systems applications can avoid duplicating complex software and increase reliability, as well as realize application-specific optimizations. We demonstrate programmable storage by mapping a wide range of common application and storage services requirements onto existing abstractions found in distributed storage systems.

We show that programmable storage introduces real challenges for issues of portability and maintenance, and that the design space for new storage interfaces is intractable for non-expert developers. To address this limitation we propose that new storage interfaces and services be expressed using a declarative language that abstracts across the differences in internal storage system interfaces to allow application developers to create new storage services without becoming storage system experts. While a declarative approach to building storage interfaces resolves many issues, it doesn't address the implications of developing and evolving storage interfaces in a real-world system. To address this we propose a set of abstractions for developing new interfaces that are aligned with existing software development workflows and source code control systems.

The non-uniform improvement of computer hardware performance poses a significant challenge for contemporary data processing in managing the growing volume of data. General-purpose systems encounter obstacles such as design, power, and heat management that hinder their computing power improvement. As data processing becomes more expensive and the increasing performance demands from applications, academia and industry are evincing interest in offloading data services to embedded systems (i.e., system software that runs on peripherals such as storage or network devices) to improve data processing efficiency. Given the domain-specific nature of embedded systems, this approach opens up abundant research opportunities, particularly as more applications rely on big data analysis for insights.

Efficiently leveraging embedded systems for data services requires answering three critical questions concerning why, what, and how. The ``why'' question pertains to the potential benefits of offloading a data service to an embedded system. Answering this question requires developing a methodology that can accurately quantify the benefits by taking into account the embedded system's domain nature and the data service workload. The ``what'' question pertains to what data services to offload to an embedded system. Answering this question requires a comprehensive understanding of the intended system and function to identify potential matches for successful offloading. In this thesis, I focus specifically on composable data services, not only because they serve as fundamental building blocks in applications, but also because their composability allows for more convenient migration to diverse systems. The ``how'' question pertains to determining the strategies to use for offloading. Given that embedded systems are designed to operate within a constrained environment, effective offloading strategies are required to prevent suboptimal performance resulting from incapable or overloaded embedded systems.

This thesis makes contributions to addressing the challenges associated with each of these research questions. First, I develop a practical methodology focused on cost-benefit quantification and a mathematical model to evaluate the data availability benefit of offloading data services into storage devices. Second, I examine and evaluate composable data services in high-performance scientific workflows to identify potential functions suitable for offloading. Finally, I explore strategies aimed at reducing data processing overhead and scheduling workloads dynamically to improve performance efficiency for data services running on embedded systems.

Global file system namespaces are difficult to scale because of the overheads of POSIX IO metadata management. The file system metadata IO created by today’s workloads subjects the underlying file system to small and frequent requests that have inherent locality. As a result, metadata IO scales differently than data IO. Prior work about scalable file system metadata IO addresses many facets of metadata management, including global semantics (e.g., strong consistency, durability) and hierarchical semantics (e.g., path traversal), but these techniques are integrated into ‘clean-slate’ file systems, which are hard to manage, and/or ‘dirty-slate’ file systems, which are challenging to understand and evolve.

The fundamental insight of this thesis is that the default policies of metadata management techniques in today’s file systems are causing scalability problems for specialized use cases. Our solution dynamically assigns customized solutions to various parts of the file system namespace, which facilitates domain-specific policies that shape metadata management techniques. To systematically explore this design space, we build a programmable file system with APIs that let developers of higher layers express their domain-specific knowledge in a storage-agnostic way. Policy engines embedded in the file system use this knowledge to guide internal mechanisms to make metadata management more scalable. Using these frameworks, we design scalable policies, inspired by the workload, for (1) subtree load balancing, (2) relaxing subtree consistency and durability semantics, and (3) subtree schemas and generators.

Each system is implemented on CephFS, providing state-of-the-art file system metadata management techniques to a leading open-source project. We have had numerous collaborators and co-authors from the CephFS team and hope to build a community around our programmable storage system.

Many storage systems are shared by multiple clients with different types of workloads and performance targets. To achieve performance targets without over-provisioning, a system must provide isolation between clients. Throughput-based reservations are challenging due to the mix of workloads and the stateful nature of disk drives, leading to low reservable throughput.

At the same time, virtualization and many other applications such as online analytics and transaction processing often require access to predictable, low-latency storage. Hard-drives have low and unpredictable performance under random workloads, while keeping everything in DRAM, in many cases, is still prohibitively expensive or unnecessary. Solid-state drives offer a balance between performance and cost, and are becoming increasingly popular in storage systems. Unfortunately, SSDs frequently block in the presence of writes, exceeding hard-drive latency and leading to unpredictable performance. Many systems with read/write workloads have low latency requirements or require predictable performance and guarantees. In such cases the performance variance of SSDs becomes a problem for both predictability and raw performance.

First, we present QBox, a new utilization-based approach for black box storage systems that enforces utilization (and, indirectly, throughput) requirements and provides isolation between clients, without specialized low-level I/O scheduling. Our experimental results show that our method provides good isolation and achieves the target utilizations of its clients.

Second, we present Rails, a flash storage system based on redundancy, which provides predictable performance and low latency for reads under read/write workloads by physically separating reads from writes.

More specifically, reads achieve read-only performance while writes perform at least as well as before. We evaluate our design using micro-benchmarks and real traces, illustrating the performance benefits of Rails and read/write separation in flash.

Compared to hard-drives, flash is a more expensive option in terms of raw storage space. We present eRails, a scalable flash storage system on top of Rails that achieves read/write separation using erasure coding without the storage cost of replication. To support an arbitrary number of drives efficiently we describe a design allowing us to scale eRails by constructing overlapping erasure coding groups that preserve read/write separation.

Programmable storage provides a means by which existing services in the storage system can be generalized, exposed, extended, combined and reused to support applications through the creation of domain-specific interfaces for use by external storage clients. Current work on programmable storage has shown how to embed user-defined functions that perform data management tasks into an object storage system. However, these functions are closely tied to the storage software code base, for instance they require an SDK and must be compiled against specific storage software versions, making them less portable and less future proof. We propose extending this capability by creating a dynamic object interface with WebAssembly - a portable binary format that facilitates generic code execution, and leveraging WebAssembly’s high-level goals to enable clients to add user-defined functionality to the OSD to support the needs of their applications. This thesis explores the design space of interfaces in a programmable storage system and introduces a method to embed a WebAssembly runtime environment, enabling dynamic injection of generic user-defined functions into a running storage system without requiring much knowledge of the internals of the storage layer.

In the last decade, our ability to store data has grown at a greater rate than our ability to process that data via existing methods. This has given rise to an entire industry focused on developing new approaches to large-scale data-intensive computing that is commonly referred to as "big data".

The scientific community has seen a similar growth in the sizes of the datasets that they must analyze in order to advance their respective fields. Unfortunately, the unique characteristics of scientific data and computations are sufficiently distinct from those that garner the focus of the larger "big data" community that the direct application of existing systems is not feasible. The absence of a suitable platform for scalable data-intensive scientific computing has already become a hindrance to some scientific communities and the negative impact on research will only increase as datasets continue to grow.

In response to this clear and present need, we present our research into designing and building a distributed computing platform appropriate for researchers in a variety of fields that are continually evolving their approaches to processing scientific data. Our work extends Hadoop, an existing open source system based on the research described in the seminal 2004 MapReduce paper, in order to benefit from the existing state of the art. We first identify the barriers to executing common scientific computations over data residing in domain-specific file formats. Once those issues are addressed, we then consider how the unique properties of scientific data and computations can be leveraged to improve the system. The resulting alterations are then evaluated, both in terms of their impact to the MapReduce design as well as their impact on the performance characteristics of the system. The culmination of our work is a viable platform for scientific computing that has been utilized and validated by external researchers.

We increasingly depend on well-behaved networks in the course of every-day activities for business, community, government, science, and recreation. And with more people demanding a greater variety of services comes sharp disagreement about which needs are most important. Unfortunately, today’s technology is inadequate to guarantee the performance of dynamic and diverse workloads in such a way that we are prevented from hurting each other’s goals.

No one likes waiting in traffic, whether on a road or on a computer network. Stuttering progress and slow interactive feedback annoy everyone and cost time and money. A network should aim to (1) minimize latency, (2) maximize bandwidth, (3) share resources according to agreements, (4) enable incremental deployment, and (5) minimize administrative overhead. Many technologies have been developed, but none yet satisfactorily address all five of these goals. The best performing solutions to date achieve some success in goals 1-3, but they often downplay the importance of goals 4-5. Solutions almost always require coordinated configurations across many pieces of a network, but they end up being either impractical or suffer poor performance when even a single piece of a network is not cooperating.

In this dissertation I present an overview of the multilayered problem surrounding network performance, a transport-level solution called TCP Inigo, and foundations for a clean-

slate queueing discipline solution based on a new combination of recent real-time and economic scheduling algorithms. TCP Inigo uses independent delay-based algorithms on the sender and receiver. In emulated experiments with single administrative domains, a situation common in data centers or a single HPC cluster, TCP Inigo’s fairness, bandwidth, and latency indices are up to 1.3× better than the best available solution. When deployed in a more complex environment, such as across administrative domains, TCP Inigo performs up to 42× better.