UC Berkeley

A problem-first approach to statistics develops statistical methods directly from real world questions and problems. This dissertation illustrates this approach through the development of statistics methods and tools in four disciplines: active transportation, higher education, election auditing and sports. Causal inference and nonparametric methods are emphasized as they avoid typically incorrect parametric assumptions.

The second chapter focuses on active transportation and problems with ensuring dataquality. Sufficiently accurate bicycle and pedestrian counts are useful for improving safety analyses, planning infrastructure, and prioritizing funding. The accuracy of instrumental counts is affected by the instrument’s sensing technology, details of siting and installation, calibration, random error, and malfunctions. Some of these errors cannot be detected without an independent, accurate count to compare to the instrumental count. But some failures can be detected (imperfectly) through their signal in the count data, which has led to a variety of algorithms to clean and interpolate instrumental count data. We present different methods for flagging questionable data and provide a detailed comparison of data cleaning approaches.

Higher education is the focus of the next chapter, and the central research questionis “do female presenters receive more questions or comments than male presenters during academic job talks?” We collect a large dataset of academic job talks from eight UC Berkeley departments from 2013-2019 in order to answer this question. We find that differences in the number, nature, and total duration of audience questions and comments are neither material nor statistically significant. For instance, the median difference (by gender) in the duration of questioning ranges from zero to less than two minutes in the five departments. Moreover, in some departments, candidates who were interrupted more often were more likely to be offered a position, challenging the premise that interruptions are necessarily prejudicial. These results are specific to the departments and years covered by the data, but they are broadly consistent with previous research, which found differences of comparable in magnitude. However, those studies concluded that the (small) differences were statistically significant. We present evidence that the nominal statistical significance is an artifact of using inappropriate hypothesis tests. We show that it is possible to calibrate those tests to obtain a proper P-value using randomization.

Motivated by the permutation test work in the previous chapter, the fourth chapterdevelops a method to construct fast exact/conservative Monte Carlo confidence intervals by inverting exact/conservative Monte Carlo tests about parameters. The method uses a single set of Monte Carlo samples, which both reduces the computational burden and ensures that the problem of finding where the P-value crosses α is well posed. For problems with realvalued parameters, if the P-value is quasiconcave in the parameter, a minor modification of the bisection algorithm quickly finds conservative confidence bounds to any desired degree of accuracy. Additional computational savings are possible for common test statistics in the one-sample and two-sample problem by exploiting the relationship between values of the test statistics for different values of the parameter. Examples across a wide range of disciplines are given to illustrate this new method.

The fifth, sixth, and seventh chapters focus on post-election audits. Post-election auditscan provide convincing evidence that election outcomes are correct—that the reported winner( s) really won—by manually inspecting ballots selected at random from a trustworthy paper trail of votes. Risk-limiting audits (RLAs) control the probability that, if the reported outcome is wrong, it is not corrected before the outcome becomes official. RLAs keep this probability below the specified “risk limit.” Chapter five compares RLAs to a proposed Bayesian alternative, Bayesian audits (BAs). BAs control a weighted average probability of correcting wrong outcomes over a hypothetical collection of elections; the weights come from the prior. RLAs and BAs make different assumptions, use different standards of evidence and offer different assurances. We illustrate these differences using simulations based on real contests. Historically, conducting RLAs of all contests in a jurisdiction has been infeasible, because efficiency is eroded when sampling cannot be targeted to ballot cards that contain the contest(s) under audit. States that conduct RLAs of contests on multi-card ballots or of small contests can dramatically reduce sample sizes by using information about which ballot cards contain which contests—by keeping track of card-style data (CSD). We present a method for using CSD to drastically decrease RLA sample sizes in chapter six. Chapter seven describes an open-source Python implementation of RLAs using CSD for the Hart InterCivic Verity voting system and the Dominion Democracy Suite voting system. The software is demonstrated using all 181 contests in the 2020 general election and all 214 contests in the 2022 general election in Orange County, CA, USA, the fifth-largest election jurisdiction in the U.S., with over 1.8 million active voters.

In the final chapter, we develop a novel method to quantify the impact of injuries on playerperformance in baseball. To quantify this impact we can look at the difference between performance the player would have achieved in the absence of injury and after a given injury. This quantity can be estimated by matching injured players to similar non-injured players. However, matching in observational studies faces complications when units enroll in treatment on a rolling basis (e.g., players are injured at different times). To address this issue, we introduce a new matched design, GroupMatch with instance replacement, allowing maximum flexibility in control selection. Second, we propose a block bootstrap approach for inference in matched designs with rolling enrollment and demonstrate that it accounts properly for complex correlations across matched sets in our new design and several other contexts. Third, we develop a falsification test to detect violations of the timepoint agnosticism assumption, which is needed to permit flexible matching across time.