In recent years, there has been an increasing emphasis on developing generalist agents capable of solving a diverse variety of tasks effectively. We hope that such an agent would be capable of chaining several smaller tasks together, navigating from high-dimensional inputs, and being simple and reliable to train. In this thesis, we will first introduce the current landscape of generalist agents and the state of Deep Reinforcement Learning (RL) and Offline RL. We'll also discuss several major issues underlying these fields, such as training instability and generalization failure. Next, we will explore several proposals for solving such problems, namely through optimization techniques and diffusion. Finally, we will discuss the major challenges and opportunities that lie ahead in the future for training generalist agents.

This thesis examines the ethical implications of artificial intelligence through the lens of physical safety. It scrutinizes the various ways AI systems can instigate unsafe behavior in users, emphasizing the underexplored domain of covertly unsafe language. To improve the safety-related reasoning ability of large language models, we propose FARM to systematically generate rationales attributed to credible sources for physical safety scenarios. Lastly, we close with a broader discussion on AI transparency, delineating its differing research threads and associated considerations such as safety, and call toward a human-centered approach to evaluate future research, centering on the foundational debate of whether humans should trust intelligent systems.

Large language models have advanced the state-of-the-art in natural language processing and achieved success in tasks such as summarization, question answering, and text classification. However, these models are trained on large-scale datasets, which may include harmful information. Studies have shown that as a result, the models can exhibit social biases and generate misinformation after training. This dissertation discusses research on analyzing and interpreting the risks of large language models across the areas of fairness, trustworthiness, and safety.

The first part of this dissertation analyzes issues of fairness related to social biases in large language models. We first investigate issues of dialect bias pertaining to African American English and Standard American English within the context of text generation. We also analyze a more complex setting of fairness: cases in which multiple attributes affect each other to form compound biases. This is studied in relation to gender and seniority attributes.

The second part focuses on trustworthiness and the spread of misinformation across different scopes: prevention, detection, and memorization. We describe an open-domain question-answering system for emergent domains that uses various retrieval and re-ranking techniques to provide users with information from trustworthy sources. This is demonstrated in the context of the emergent COVID-19 pandemic. We further work towards detecting potential online misinformation through the creation of a large-scale dataset that expands misinformation detection into the multimodal space of image and text. As misinformation can be both human-written and machine-written, we investigate the memorization and subsequent generation of misinformation through the lens of conspiracy theories.

The final part of the dissertation describes recent work in AI safety regarding text that may lead to physical harm. This research analyzes covertly unsafe text across various language modeling tasks including generation, reasoning, and detection.

Altogether, this work sheds light on the undiscovered and underrepresented risks in large language models. This can advance current research toward building safer and more equitable natural language processing systems. We conclude with discussions of future research in Responsible AI that expand upon work in the three areas.

Training data has played a major role in the rise of large deep learning models. In particular, the scale and diversity of training data has led to incredible new capabilities in large language models. However, despite the success of such models, a notable gap persists in understanding the important role that data plays in their performance, and how to use this understanding to further improve models. In this work, we advocate for, and demonstrate the effectiveness of, data-centric AI.

In the first part of this dissertation, we aim to better understand language models through their data. First, we design a relation extraction system that outputs human-interpretable intermediate outputs, allowing us to better understand why the system makes its predictions. Next, we delve into the intricate relationship between data and models by studying zero-shot and few-shot transfer learning settings, giving us insights into the interactions that training data has on model performance across diverse tasks.

Based on the lessons from the first part of this dissertation, we next aim to improve the data used to train models. We first demonstrate that data selection can be formulated as a multi-armed bandit problem, where the goal is to optimize a model's training data. We apply the multi-armed bandit formulation first to the few-shot fine-tuning setting, and then to language model pretraining, designing algorithms and rewards that are unique for each problem setting. Finally, we show that for cross-lingual question answering, data augmentation is a strong approach to improving the diversity of training data, leading to improved performance.

Overall, this work aims to improve our understanding of how deep learning models work, using data as the viewpoint. Further, we take this understanding and use it to develop data-efficient and performant models. We conclude the dissertation with discussions of future research in data-centric AI and propose avenues for extending these concepts into new research directions.

Grounding natural language onto real-world perception is a fundamental challenge to empower various practical applications that require human-machine communication.

This Ph.D. dissertation presents our research thrusts on developing intelligent embodied agents that connect language, vision, and actions. It consists of two major directions towards human-machine communication.

First, from visual recognition to cognition, we study using natural language to describe the visual information and express the understanding of the visual world.

We show that scalable learning approaches based on semantics and external knowledge can generate coherent, fine-grained, and generalizable visual descriptions.

We also investigate the automatic evaluation metrics for language generation, and propose an adversarial reward learning method to overcome their limitations in optimizing the policy for human-like visual storytelling. Besides, we introduce a multilingual dataset for video-and-language research, which goes beyond monolingual language grounding in vision.

Second, in order to connect language and vision to actions, we have situated natural language inside interactive environments where communication often occurs.

We systematically study the task of vision-language navigation, aiming to associate human commands with visual perception and navigate the 3D world.

Then we demonstrate novel methods to tackle generalization and data scarcity issues from various perspectives like counterfactual thinking, transfer learning, multitask learning and agnostic learning. Our efforts shed light on generalizing embodied navigation agents to more challenging and practical scenarios.

Finally, we summarize the strengths, weaknesses, and implications of our work, and discuss the future research plan of pushing embodied AI research further.