Networks provide functionality beyond just packet routing and delivery. Network functions such as firewalls, caches, WAN optimizers, etc. are crucial for scaling networks and in supporting new applications. While traditionally network functions were implemented using dedicated hardware middleboxes, recent efforts have resulted in them being implemented as software and deployed in virtualized environment . This move towards virtualized network function is commonly referred to as network function virtualization (NFV). While the NFV proposal has been enthusiastically accepted by carriers and enterprises, actual efforts to deploy NFV have not been as successful. In this thesis we argue that this is because the current deployment strategy which relies on operators to ensure that network functions are configured to correctly implement policies, and then deploys these network functions as virtual machines (or containers), connected by virtual switches are ill- suited to NFV workload.

In this dissertation we propose an alternative NFV framework based on the use of static techniques such as type checking and formal verification. Our NFV framework consists of NetBricks – a NFV runtime and programming environment, that uses type checking to provide isolation, and presents a novel dataflow based approach to writing high performance network functions; and VMN a verification tool that can automatically check whether a set of NFs correctly implement network policy. Finally, we also show that simplifying NF development and deployment enable new applications, both in the wide-area and within datacenters.

Although today's networking equipment has achieved high performance and low cost by embedding forwarding logic in hardware, this has come at the price of severely reduced flexibility. In this dissertation, we address the problem of achieving both flexibility and performance in two networking domains: packet forwarding and data center networking. In packet forwarding, we present Software Defined Forwarding, a hybrid design that attempts to combine the high speed and low cost of hardware with the superior flexibility of software. Within the data center context, we propose Ripcord, a platform for data center routing and management. Through simulation, prototype implementation and testbed experiments, we demonstrate that these solutions achieve both flexibility and high performance in their respective contexts.

Since the invention of packet switching networks, routing is an important, if not the most fundamental, component of networking. Over decades, both academia and industry put huge efforts to improve routing in various scenarios. However, with the explosive growth of Internet, wide adoption for critical business and services, and new environments like data center networks, routing has more difficulties to meet the increasingly stringent requirements.

We examine current routing schemes and discover that the problem is more fundamental, because they rely on many assumptions and trade-offs that are no longer applicable for today's networks. Believing that more radical changes are necessary to effectively address the challenge, we start the design process from scratch, with assumptions, trade-offs and design philosophies which better reflect the networks of today and tomorrow. The outcome of the design process is a unified routing framework which utilizes directed acyclic graph (DAG) as routing topology. And we name it Routing Along DAGs (RAD).

RAD separates route optimization and connectivity maintenance, handles both failures and congestions, and recovers fast and locally. We explain the details of RAD design and test its performance on various real world topologies. Then we improve RAD to achieve even better performance and cover more use cases. With these extensions, RAD becomes both complete and practical.

Cloud computing enables operators to spin up resources on-demand, and orchestrators enables the automatic deployment of complex distributed applications. As deployment of applications in datacenter environments grows increasingly automated and accessible, the logical next step is to consider how the infrastructure can support the operator in critical configuration tasks.

In this dissertation, we build and evaluate a number of infrastructure-based approaches that automate a few of these critical configuration tasks. First, we present AutoTune, an orchestrator-based tool that tunes an application's resource allocation in order to improve end-to-end performance and resource efficiency. Next, we describe Privoxy, a system which automatically checks that SQL queries issued by web applications are compliant with operator-defined privacy policies. Finally, we build a trace-driven packet simulator that evaluates how an architectural change to the datacenter networking fabric impacts the training time of convolutional neural networks for image recognition.

Until practical ad-hoc localization systems are developed, early deployments of wireless sensor networks will manually configure location information in network nodes in order to assign spatial context to sensor readings. In this paper, we argue that such deployments will use hierarchical location names (for example, a node in a habitat monitoring network might be said to be node number N in cluster C of region R), rather than positions in a two- or three-dimensional coordinate system. We show that these hierarchical location names can be used to design a scalable routing system called HLR. HLR provides a variety of primitives including unicast, scoped anycast and broadcast, as well as various forms of scalable rendezvous. These primitives can be used to implement most data-centric routing and storage schemes proposed in the literature; these schemes currently need precise position information and geographic routing in order to scale well. We evaluate HLR using simulations as well as an implementation on the Mica-2 motes.

The past few years have seen a major change in computing systems, as growing data volumes and stalling processor speeds require more and more applications to scale out to distributed systems. Today, a myriad data sources, from the Internet to business operations to scientific instruments, produce large and valuable data streams. However, the processing capabilities of single machines have not kept up with the size of data, making it harder and harder to put to use. As a result, a growing number of organizations---not just web companies, but traditional enterprises and research labs---need to scale out their most important computations to clusters of hundreds of machines.

At the same time, the speed and sophistication required of data processing have grown. In addition to simple queries, complex algorithms like machine learning and graph analysis are becoming common in many domains. And in addition to batch processing, streaming analysis of new real-time data sources is required to let organizations take timely action. Future computing platforms will need to not only scale out traditional workloads, but support these new applications as well.

This dissertation proposes an architecture for cluster computing systems that can tackle emerging data processing workloads while coping with larger and larger scales. Whereas early cluster computing systems, like MapReduce, handled batch processing, our architecture also enables streaming and interactive queries, while keeping the scalability and fault tolerance of previous systems. And whereas most deployed systems only support simple one-pass computations (e.g. aggregation or SQL queries), ours also extends to the multi-pass algorithms required for more complex analytics (e.g. iterative algorithms for machine learning). Finally, unlike the specialized systems proposed for some of these workloads, our architecture allows these computations to be combined, enabling rich new applications that intermix, for example, streaming and batch processing, or SQL and complex analytics.

We achieve these results through a simple extension to MapReduce that adds primitives for data sharing, called Resilient Distributed Datasets (RDDs). We show that this is enough to efficiently capture a wide range of workloads. We implement RDDs in the open source Spark system, which we evaluate using both synthetic benchmarks and real user applications. Spark matches or exceeds the performance of specialized systems in many application domains, while offering stronger fault tolerance guarantees and allowing these workloads to be combined. We explore the generality of RDDs from both a theoretical modeling perspective and a practical perspective to see why this extension can capture a wide range of previously disparate workloads.