Automated decision-making systems are increasingly being deployed in areas with high personal and societal impact. This naturally led to growing interest in trustworthy artificial intelligence (AI) and machine learning (ML), encompassing many fields of research including algorithmic fairness, robustness, explainability, privacy, and more. These works share a common theme of questioning and moderating the behavior of automated tools in various real-world settings, which inherently exhibit different uncertainties.

This dissertation explores how probabilistic modeling and reasoning as a framework offer a principled way to handle uncertainties when addressing trustworthy AI issues, in particular by explicitly modeling the underlying distribution of the world. The main contributions are as follows. First, it demonstrates that many problems in trustworthy AI can be cast as probabilistic reasoning tasks of varying complexities. Secondly, it proposes algorithms to learn fair and robust decision-making systems, while handling many sources of uncertainties such as missing or biased labels at training time and missing features at prediction time. The proposed approach relies heavily on probabilistic models that are expressive enough to describe the world underlying the system, whilst being tractable enough to answer various probabilistic queries. The final contribution of this thesis is showing that probabilistic circuits are an effective model for this framework and expanding their reasoning capabilities even further.

Logic and uncertainty form two of the primary pillars of modern artificial intelligence. Seeking to draw insights about what can be gained by understanding both, we explore different contexts where both are crucial. First, we use logical circuits as our underlying machinery, developing a hybrid circuit sampling technique for approximate probabilistic inference, as well as an axiomatized method for enforcing logical constraints when learning a deep neural network. We then explore modifying probabilistic databases, incorporating schema level constraints to overcome lack of data, as well as demonstrating how and why to directly incorporate them with relational machine learning techniques for free efficient inference on well tuned models.

Word embeddings learned from text are well-known to capture relational information. However, extracting such relations and their associated vectors is typically performed manually, to illustrate what knowledge is embedded in the space. We propose an automated approach to mine word embeddings for sets of entities of the same type, as well as relationships that hold between them. Our approach starts from a single seed entity and extracts a relational representation from the surrounding vector space. It does so without any relational supervision. Experiments show that our extraction algorithm outperforms spectral clustering and indeed is able to extract high-quality relations from noisy embeddings.

Machine Learning models are becoming increasingly ubiquitous in our daily lives. Despite their advancements, common challenges such as missing data, outliers, and complex behaviors still hinder their usability and trustworthiness. Handling uncertainty is a fundamental aspect of modern machine learning which can potentially help with these challenges. Specifically, in this thesis, we highlight the strengths of tractable probabilistic models, such as probabilistic circuits, in mitigating these issues.

First, we introduce "Expected Prediction" (EXP) as a new type of probabilistic query and examine its computational complexity for a few families of models. We then demonstrate how computing EXP can help address the additional uncertainty arising from missing data in a principled manner. Next, we illustrate how EXP can be used to generate sufficient explanations for classifier decisions while offering probabilistic guarantees. Finally, we leverage the supermodularity of marginal queries and continuous relaxations like multi-linear extension to scale the detection of the root causes of outliers in high-dimensional settings.

This thesis develops a novel methodology incorporating symbolic knowledge in deep learning.

We constructed a logical-constraint semantic loss, using a neural network's output vectors as variables.

Semantic loss captures the symbolic knowledge and adds previously-lost information to neural networks.

An experimental evaluation shows that it leads the neural network to achieve (near-)state-of-the-art performance on semi-supervised multi-class classification.

Moreover, it is applicable to more complex constraints.

We present a process, at the end of the thesis, of calculating the semantic loss and its derivatives for complex constraints to show its generality.

The work presented in this thesis was published at the International Conference on Machine Learning.

There is a trade-off between expressiveness and tractability in generative modeling. On the one hand, while neural-based deep generative models are extremely expressive, the ways we can query them are limited; on the other hand, while tractable probabilistic models support efficient computation of various probabilistic queries, scaling them up is a major challenge. Probabilistic circuits are a tractable representation of probability distributions allowing for exact and efficient computation of likelihoods and marginals. We study the task of scaling up the learning of probabilistic circuits and then applying them to various applications. On the learning front, we propose a new algorithm for learning the sparse structures of probabilistic circuits that can significantly improve their capacity. On the application front, we further demonstrate the expressiveness and tractability of probabilistic circuits in two downstream applications: genetic sequence modeling and controllable language generation.

On one side, symbolic methods represent our knowledge of the world, and when coupled with probabilistic reasoning, one can infer the uncertainty of unknown facts conditioned on the knowledge of known evidence. On the other side, statistical machine learning finds and infers a predictive function from vast amounts of data. To automate complex decision making, both are crucially needed. While their unification has been long pursued, the two up to now still largely remain disparate from one another. This dissertation demonstrates circuit representations are a promising symbolic-statistical synthesis. In particular, we study circuit representations across the dimensions of tractable probabilistic reasoning with and without logical constraints, structure learning from data, and classifying on image domains, namely, the main tasks across symbolic and statistical methods. And more importantly, we show those dimensions are unified for circuit representations by leveraging the same syntactic properties.

Along with the ubiquitous applications of Artificial Intelligence (AI), the quest for developing trustworthy AI models intensifies. Deep neural networks, while powerful in learning, fall short in reasoning with domain knowledge and offering robustness guarantees. Neurosymbolic AI bridges this gap by melding the learning capabilities of neural networks and reasoning techniques from symbolic AI, thus building models that behave as intended. This dissertation demonstrates my work that addresses the two fundamental challenges in neurosymbolic AI: 1) enabling differentiable learning of deep neural networks under symbolic constraints and 2) performing scalable and reliable probabilistic reasoning over expressive symbolic constraints. It presents how these neurosymbolic approaches achieve trustworthiness through explainability, uncertainty quantification, and domain-knowledge incorporation. These contributions enable broader applications of neurosymbolic AI in various domains including scientific discoveries.

Probabilistic modeling and reasoning are central tasks in artificial intelligence and machine learning. A probabilistic model is a rough description of the world: the model-builder attempts to capture as much detail about the world’s complexities as she can, and when no more detail can be given the rest is left as probabilistic uncertainty. Once constructed, the goal of a model is to perform automated inference: compute the probability that some particular fact is true about the world. It is natural for the model-builder to want a flexible expressive language – the world is a complex thing to describe – and over time this has led to a trend of increasingly powerful modeling languages. This trend is taken to its apex by probabilistic programming languages (PPLs), which enable modelers to specify probabilistic models using the facilities of a full programming language. However, this expressivity comes at a cost: the computational cost of inference is in direct tension with the flexibility of the modeling language, and so it becomes increasingly difficult to design automated inference algorithms that scale to the kinds of systems that model builders want to create.

This thesis focuses on the central question: how can we design effective probabilistic programming languages that profitably trade-off expressivity and tractability for inference? The approach taken here is first to identify and exploit important structure that a probabilistic program may possess. The kinds of structure considered here are discrete program structure and symmetry. Programs are heterogeneous objects, so different parts of programs may exhibit different kinds of structure; in the second part of the thesis I show how to decompose heterogeneous probabilistic program inference using a notion of program abstraction. These contributions enable new applications of probabilistic programs in domains such as text analysis, verification of probabilistic systems, and classical simulation of quantum algorithms.