A key task in natural language generation (NLG) is Referring Expression Generation (REG), in which a set of properties are selected to describe a target referent. Computational cognitive models of REG typically focus on REG-in-context, where the referring expressions are designed to take into account the conversational context into which they are to be generated. However, in practice, these methods only focus on linguistic context of the text into which they are to be inserted. We argue that to develop robust models of naturalistic human referring, REG will need to move beyond linguistic context, and account for cognitive and environmental context as well. That is, we propose that a cognitivist, interactionist, and situated approach to modeling REG is needed. In this paper, we present GAIA, a Givenness Hierarchy theoretic model of REG, and demonstrate the immediate qualitative benefits of this model over the traditional REG model which it extends.

Robots must be able to communicate naturally and efficiently, e.g., using concise referring forms like it, that, and the ⟨N’⟩. Recently researchers have started working on Referring Form Selection (RFS) machine learning algorithms but only evaluating them offline using traditional metrics like accuracy. In this work, we investigated how a cognitive status-informed RFS computational model might fare in actual human-robot interactions in a human-subjects study (N=36). Results showed improvements over a random baseline in task performance, naturalness, understandability, and mental workload. However, the model was not perceived to outperform a simple, naive, non-random baseline (constant use of indefinite noun phrases). We contribute several key research directions for further development of cognitive status-informed RFS models, the inclusion of multi-modality, and further development of testbeds.

Human dialogue is governed by communicative norms thatspeakers are expected to follow in order to be viewed as coop-erative dialogue partners. Accordingly, for language-capableautonomous agents to be effective human teammates they mustbe able to understand and generate language that complieswith those norms. Moreover, these linguistic norms are highlycontext sensitive, requiring autonomous agents to be able tomodel the contextual factors that dictate when and how thosenorms are applied. In this work, we consider three key lin-guistic norms (directness, brevity, and politeness), and exam-ine the extent to which adherence to these norms varies underchanges to three key contextual factors (potential for harm, in-terlocutor authority, and time pressure). Our results, based ona human-subject study involving 5,642 human utterances, pro-vide strong evidence that speakers do indeed vary their adher-ence to these norms under changes to these contextual factors.

Working Memory (WM) is a necessary component for models of human cognition and human-inspired robot cognitive architectures. Different theories explain how the limited capacity of WM should be maintained, including theories of forgetting through decay and interference. Yet, it is unclear how WM models informed by these theories might be used to inform robot cognition, and how they might shape robots' ability to engage in natural, situated, language-based interactions. To resolve this tension, in this work we consider entity-level, feature-based WM systems that can be integrated into robot cognitive architectures to reflect both decay- and interference-based dynamics. We demonstrate how different parameterizations of these WM strategies have fundamentally different error modes in different interaction contexts. We formulate rules that inform the selection of decay and interference parameters to be used in contexts with different factors that are important for language-based interaction.

We present a computational cognitive model of referential choice that models and explains the choice between a wide variety of referring forms using a small set of features important to situated contexts. By combining explainable machine learning techniques, data collected in situated contexts, and recent computational models of cognitive status, we produce an accurate and explainable model of referential choice that provides an intuitive pragmatic account of this process in humans, and an intuitive method for computationally enabling this capability in robots and other autonomous agents.

Gestures play a critical role in human-human and human-robot interaction. In task-based contexts, deictic gestures like pointing are particularly important for directing attention to task-relevant entities. While most work on task-based human-human and human-robot dialogue focuses on closed-world domains, recent research has begun to consider open-world tasks, where task-relevant objects may not be known to the hearer a priori. In open-world tasks, we argue that a more nuanced consideration of gesture is necessary, as interactants may use gestures that bridge traditional gesture categories, in order to navigate the open-world dimensions of their task environment. In this work, we explore the types of gestures used in open-world task contexts, and their frequencies of use. Our results suggest a need to rethink the way that gesture analysis is approached in the study of human-human and human-robot interaction.

Working Memory (WM) plays a key role in natural language understanding and generation. To enable a human-like breadth and flexibility of language understanding and generation capabilities, cognitive systems for language-capable robots should feature a human-like WM system in a similarly central role. However, it is still quite unclear how robotic WM should be designed, as a variety of models of human WM have been proposed in cognitive psychology. Moreover, human reliance on WM during language production is sometimes to help the speaker rather than to help hearers. Thus, it is unclear whether different robotic WM systems might harm certain dimensions of interaction for the sake of the robot speaker's ostensible ease of cognitive processing. In this paper we demonstrate how different models of human WM can be implemented into robot cognitive architectures. Our results suggest that these models can be effective in terms of accuracy, perceived naturalness, and perceived human-likeness.

One of the principle tenets of modern behavioral ethics is thathuman morality is dynamic and malleable. Recent work intechnology ethics has highlighted the role technologies canplay in this process. As such, it is the responsibility oftechnology designers to actively identify and address possi-ble negative consequences of such technological mediation. Inthis work, we examine dialogue systems employed by currentrobotic agents, arguing that they can have deleterious effectson both the human moral ecosystem and human perception ofthe robots, regardless of the robots’ actual ethical competence.We present a preliminary Bayesian analysis of empirical datasuggesting that the architectural status quo of clarification re-quest generation systems may (1) cause robots to unintention-ally miscommunicate their ethical intentions (our two tests forthis yielded Bayes factors of 1319 and 1099) and (2) weakenhumans’ contextual application of moral norms (Bayes fac-tor of 1069).

To enable natural and fluid human-robot interactions, robots need to not only be able to communicate with humans through natural language, but also do so in a way that complies with the norms of human interaction, such as politeness norms. Doing so is particularly challenging, however, in part due to the sensitivity of such norms to a host of different contextual and intentional factors. In this work, we explore computational models of context-sensitive human politeness norms, using explainable machine learning models to demonstrate the value of both speaker intention and task context in predicting adherence with indirect speech norms. We argue that this type of model, if integrated into a robot cognitive architecture, could be highly successful at enabling robots to predict when they themselves should similarly adhere to these norms.

Referring expression understanding and generation are critical for robots to communicate about the world around them. Recently there have been significant advances on the problem of referring expression understanding, also known as reference resolution, with researchers presenting approaches to both incremental reference resolution (i.e., processing referring expressions word by word in real-time as they are spoken) and open-world reference resolution (i.e., resolving references both to known and previously unknown entities). In this work, we combine insights from these approaches to present IPOWER: the first algorithm for performing reference resolution incrementally in open-world environments.