Python-based database-backed web frameworks have gained wide adoption in developing online web applications in many commercial companies and open-source projects. They are used in e-commerce, banking, social networking, and many aspects of our daily life. These web applications have critical requirements on high dependability: any data corruption or service downtime could lead to significant business loss. Unfortunately, such incidents caused by application bugs or operator errors remain increasingly pervasive.

This dissertation explores the potential of building and maintaining more dependable Python-based web applications by providing tooling support for developers. To achieve this, we observe the two unique characteristics of these web applications: (1) They are backed by a database management system (DBMS), which enforces data integrity constraint checks. (2) The main application logic is developed in Python, an interpreted language with unique language features.

The first piece of this dissertation tackles the data integrity issues caused by application bugs or operator errors. Our study observes that application developers often do not take full advantage of the database-level integrity validations and miss specifying many database constraints, resulting in many severe consequences. We then build a tool called CFinder, to automatically infer the missing database constraints from application source code by leveraging the observation that many source code patterns imply certain data constraints.

Second, the dissertation tackles the demand of applying code changes for online diagnosing and fixing while keeping the service available. We observe the unique language features of Python, i.e. meta-object protocol and dynamic typing, which makes dynamic code change much easier for Python than for C or Java. We then propose a new framework called PyLive, to support dynamic logging, profiling, and bug-fixing without restarting. It can be easily adopted with no modification to the runtime system.

In cloud-scale systems, fault is a fact of life. To tolerate faults and provide highly-available service is arguably the single most important task for cloud builders. Yet, despite the considerable efforts into fault-tolerance and software engineering for reliability, all cloud scale services continue to experience costly failures. A natural question to ask is: why do cloud-scale services still fail despite the abundant fault-tolerance and how we can further improve? This thesis attempts to shed light on this question.

In the first part of this thesis, we study a set of 34 publically disclosed cloud service outages that we gathered and consider them from the point of view of fault-tolerance mechanisms. We present a novel taxonomy to categorize why the mechanisms may be ineffective; it includes faults that cannot be handled by replication, insufficient redundancy, and undetected faults. We also explore the root causes of failures, and investigate the interactions of system components in failures that were caused by multiple faults.

We find that, in many cases, while cloud systems are robust to tolerate traditional faults, they are fragile under misconfiguration, which is a major source of service unavailability. To further improve cloud service quality, it is crucial to reduce misconfiguration.

In the second half of this thesis, we propose a framework, ConfValley, to systematically validate configuration and catch errors before production. At the core of ConfValley is a language called CPL to allow experts to express configuration specifications declaratively. To further reduce operators' burdens of writing configuration specifications, our framework also includes a component to automatically infer specifications.

We evaluate ConfValley in a leading cloud service provider, Microsoft Azure, on its various types of configuration data. We rewrite existing configuration validation code in Microsoft Azure in CPL with more than 10x fewer lines of code. The framework also automatically infers thousands of CPL specs with high accuracy. With the translated and automatically generated specifications, we prevented a number of configuration errors from rolling out in production in Microsoft Azure.

Misconfigurations (a.k.a., configuration errors from a system’s standpoint) are among the dominant causes of today’s catastrophic system failures that turn down cloud-scale services and affect hundreds of millions of end users. Despite their wide adoption, traditional fault-tolerance and failure-recovery techniques are not effective in dealing with configuration errors, especially in large-scale software systems deployed in cloud and datacenters. To make the matters worse, even the tolerance and recovery mechanisms themselves are often misconfigured in the real world, which impairs the immune system of the entire cloud and datacenters.

This dissertation explores two fundamental questions towards the solutions for the inevitable misconfigurations—how to build reliable cloud and datacenter systems in the face of configuration errors; moreover, how to prevent misconfigurations in the first place by better configuration design. The goal is to enable software systems to proactively anticipate and defend against misconfigurations, rather than reacting to their manifestations and consequences.

This dissertation presents three key principles of systems design and implementation for hardening cloud and datacenter systems against misconfigurations—anticipating misconfigurations, early detection of configuration errors, and simplicity-oriented configuration design. The dissertation demonstrates that applying these principles can effectively defend cloud and datacenter systems against misconfigurations. Moreover, the dissertation presents the corresponding techniques and tool support that can automatically and systematically apply these principles to existing systems software.

The main technical insight is that configurations are essentially used by the systems, while configuration errors are mostly manifested through the faulty execution that uses erroneous configuration values. Therefore, by analyzing the system’s code that uses

configuration values, one can understand and make use of system-level information of configurations to build defense against potential errors. This dissertation first presents Spex that enables systems to anticipate misconfigurations. Spex automatically infers configuration constraints from a system’s source code, and then leverages the constraints to test the system’s resilience to misconfigurations and detect error-prone configuration design/handling. On step further, the dissertation introduces PCheck to automatically generate checking code which captures configuration errors at the system’s initialization phase to prevent their late manifestations and the corresponding failure damage.

Going beyond, this dissertation presents simplicity-oriented configuration design towards more usable and less error-prone software configuration. The key idea is to apply the user-centric philosophy to design configuration as an interface—configurations are essentially the interface for controlling and customizing system behavior, but have rarely been treated as it is. The dissertation shows that configurations in today’s systems software can be significantly simplified and effectively navigated, with the understanding of how they are actually used in the field.

Access control configurations are gatekeepers to block unwelcome access to sensitive data. Unfortunately, system administrators (sysadmins) sometimes over-grant permissions when they resolve unintended access-deny issues reported by legitimate users. The mistakes in the access control configurations can result in severe consequences, such as data breaches and system compromises. To make things worse, the access control misconfigurations may stay silent until the security incident happens.

This dissertation explores two approaches to help sysadmins diagnose the access-deny issues and reduce the permission over-granting mistakes. The first approach takes the problem of insufficient access-control logging in server applications. We designed an automated tool, SecLog, to automatically add missing access-deny log messages, and also enhance existing ones with relevant information to guide sysadmins to diagnose the access-deny issues. The second approach tackles the problem of blind spots in knowledge and system settings for sysadmins in solving access-deny issues. We propose a system, Multiview, to automatically mutate the system configurations to explore possible directions and let each direction grant as few permissions as possible. Multiview provides a detailed diagnosis report, including access-control configurations that are related to the denial, possible directions to allow the request, as well as the impact of each direction on the access-control state of the entire system to assist sysadmins during diagnosis.

Most of today’s mobile apps rely on the underlying networks to deliver key func- tions such as web browsing, file synchronization, and social networking. Compared to desktop-based networks, mobile networks are much more dynamic with frequent con- nectivity disruptions, network type switches, and quality changes, posing unique pro- gramming challenges for mobile app developers.

As revealed in this thesis, many mobile app developers fail to handle these inter- mittent network conditions in the mobile network programming. Consequently, network programming defects (NPDs) are pervasive in mobile apps, causing bad user experiences such as crashes, data loss, etc. Despite the development of network libraries in the hope of lifting the developers’ burden by providing flexible fault-tolerant APIs, we observe that many app developers often ignore such APIs or lack the capability to use them correctly and therefore still introduce NPDs.

In this thesis we make three contributions towards robust network programming for mobile apps. In the beginning, we study the characteristics of 90 real-world NPDs in Android apps towards a deep understanding of their impacts, root causes, and code patterns. This study exposes the common mistakes and pitfalls made by app developers when dealing with intermittent network. It also reflects that NPDs are pervasive despite the availability of fault-tolerant API from third-party network libraries.

Driven by the study, we build NChecker, a practical tool to detect NPDs by stat- ically analyzing Android app binaries. NChecker has been applied to hundreds of real Android apps and detected 4180 NPDs from 285 randomly-selected apps with a 94+% accuracy. Shockingly, NChecker detects network bugs in 99% of the evaluated apps. Our further analysis of these defects reveals the common mistakes of app developers in working with the existing network libraries’ abstractions, which provide insights for improving the usability of mobile network libraries.

Besides NPD detection, a more fundamental way to eliminate NPDs is to pro- vide developers easy-to-use interfaces for network programmings. We argue that the difficulty of avoiding NPDs can be mitigated through an annotation language that al- lows developers to declaratively state desired and actual properties of the application, largely without reference to fault-tolerant concepts, much less implementation. A pre- compiler can process these annotations, replacing calls to standard networking libraries with customized calls to a specialized library that enhances the reliability. The last com- ponent of the thesis presents ANEL, a declarative language and middleware for Android that enables non-experts. We demonstrate the expressiveness and practicability of ANEL

annotations through cases studies on real-world networked mobile apps. We also show that the ANEL middleware introduces negligible runtime performance overhead.

Bugs and security issues are primary concerns for software developers. Existing research have continuously focused on addressing these problems. However, the evolution of software engineering leads to increasingly complex software systems that are more susceptible to bugs. The rise of third-party services, cross-vendor libraries, and collaborative development introduces significant challenges for developers, making it difficult for them to have a comprehensive understanding of the entire codebase. Under the pressure of agile development timelines, developers often work with incomplete knowledge, leading to potential blind spots in software development. These blind spots can result in developers being unaware of certain constraints or security implications imposed by other components or authors, causing serious issues in access control, memory management, I/O operations, and business logic.

The dissertation investigates two aspects of these challenges. The first aspect focuses on cross-authorship blind spots. This part of the study identifies a specific pattern of bug-proneness, namely cross-authorship unused definitions. To address this, we introduce syntactic and semantic patterns that help identify such issues while filtering out false positives. Additionally, to accommodate the time pressures faced by developers, we use a code familiarity model to prioritize bug validation. Our implementation, named ValueCheck, has been evaluated on large-scale systems including Linux, MySQL, OpenSSL, and NFS-ganesha, successfully detecting 210 unknown bugs, with 154 confirmed. In comparisons with the state-of-the-art tools like Infer and Coverity, ValueCheck demonstrates greater effectiveness and lower false positive rates.

The second part studies cross-component blind spots. It focuses on blind spots in web applications with a client-server architecture, where client-side code is exposed. Relying solely on client-side security checks for authorization, identity verification, and user input validation is insufficient due to potential user manipulation. To address this, we propose a novel technique that enhances existing methods by altering client-side code to assess server-side security. This approach improves testing efficiency and detects complex vulnerabilities related to business logic, token-based defenses, and data preprocessing. Our testing tool, FenceHooper, identified 48 vulnerabilities in the top 300 websites from the Tranco dataset, including critical access control flaws affecting over 20 million user accounts.

Access-control misconfigurations are among the main causes of today’s security incidents. One main reason is that access-control configurations need to be frequently changed by system administrators (sysadmins) to accommodate dynamic information sharing. Unfortunately, to err is human—sysadmins often make mistakes (e.g., over-granting privileges) when changing access control configurations. Such mistakes can stay unnoticed for a long time until eventually being exploited by attackers, causing catastrophic security incidents.

This dissertation explores two validation approaches to detect access-control misconfigurations at different life-cycle stages of systems. The first approach is to test access-control configuration changes before they are deployed to production. This can help sysadmins detect access-control misconfigurations before they bring any real harm to production systems. The second approach is to monitor access-control behavior changes after the configuration changes are deployed to production. This can help sysadmins detect and diagnose potential data leaks caused by access-control misconfigurations quickly so that they can be fixed timely.

First, this dissertation presents a new type of test programs, ACTESTs, to test access- control configuration changes and a new approach to generate such test programs from existing program code. ACTESTs output the impacts of access-control changes—what requests were denied, but will be allowed after a change, and vice versa. With this, sysadmins can validate if the changed requests are intended or not and identify potential security vulnerabilities. The key challenges this dissertation addressed include making ACTESTs safe to run in production environments and making them performance-efficient. ACTESTs help detect 168 new misconfigurations from 72 Docker images.

Second, this dissertation presents P-DIFF, a practical tool for monitoring access-control behavior to help sysadmins early detect unintended access-control configuration changes and perform postmortem forensic analysis upon security attacks. P-DIFF continuously monitors access logs and infers access-control behavior changes from them. This dissertation devises a novel time-changing decision tree to effectively represent access-control behavior changes, coupled with a new learning algorithm to infer the tree from access logs. Evaluation shows that P-DIFF can detect 76%–100% of access control behavior changes.

Pre-2018 CSE ID: CS2015-1014