Robots have tremendous potential to help us in our daily lives. However, one key obstacle to facilitating their autonomy is that they lack the ability to perceive novel, or unseen objects. The de-facto solution to this problem is to pre-program robots using a large corpus of prior-known objects in hope that they will understand every object that they encounter. However, if robots need to understand new objects, they must be manually re-programmed to do so, which has proven to be time consuming and expensive, and is fundamentally intractable. Alternatively, a more direct approach to this problem is to leverage a robot's context, e.g., its immediate surroundings, which can be a rich source of information from which to learn about unseen objects in a scalable manner.

The goal of my research is to design algorithms and systems that enable robots to automatically discover unseen objects from their surroundings in a manner that is fast and robust to real-world vision challenges. In this dissertation, I discuss four key contributions of my work. First, I designed Salient Depth Partitioning (SDP), a novel depth-based region cropping algorithm which substantially improves the computation time of existing object detectors by up to 30%, with no discernible change in accuracy. SDP achieves real-time performance and is designed to give robots a better sense of visual attention, guiding them to visual regions that are likely to contain semantically important elements, which are also known as salient. Consequently, SDP can be used as a preprocessing algorithm to improve the computational efficiency of depth-based object detectors on mobile robots.

Second, I demonstrated that object proposal algorithms, a ubiquitous algorithmic component in machine vision systems, do not translate well to real-world contexts, which can negatively impact the performance of robots. I conducted a study to explore how algorithms are influenced by real-world robot vision challenges such as noise, blur, contrast, and brightness. I also investigated their performance on hardware with limited memory, CPU, and GPU resources to mimic constraints faced by mobile robots. To my knowledge, I am the first to investigate object proposal algorithms for robotics applications. My results suggest that object proposal algorithms are not generalizable to real-world challenges, in direct contrast to what is claimed in the computer vision literature. This work contributes to the field by demonstrating the need for better evaluation protocols and datasets, which will lead to more robust unseen object discovery for robots.

Third, I developed Unsupervised Foraging of Objects (UFO), a novel, unsupervised method that can automatically discover unseen salient objects. UFO is substantially faster than existing methods, robust to real-world noise (e.g., noise and blur), and achieves state-of-the-art performance. Unlike existing approaches, UFO leverages object proposals and a parallel discover-prediction paradigm. This allows UFO to quickly discover arbitrary, salient objects on a frame-by-frame basis, which can help robots to engage in scalable object learning. I compared UFO to two of the fastest and most accurate methods (at the time of writing) for unsupervised salient object discovery (Fast Segmentation and Saliency-Aware Geodesic), and found it to be 6.5 times faster, achieving state-of-the-art precision, recall, and accuracy. Furthermore, I show that UFO is robust to real-world perception challenges encountered by robots, including moving cameras and moving objects, motion blur, and occlusion. This work lays the foundation for faster online object discovery for robots which contributes toward future methods that will enable robots to learn about new objects via observation.

Fourth, I designed RaccooNet, a new real-time object proposal algorithm for robot perception. To my knowledge, RaccooNet is the current fastest object proposal algorithm at a runtime of 47.9 fps while also achieving comparable recall performance to the state-of-the-art (e.g., RPN, Guided Anchoring). Additionally, I introduced a novel intersection over union overlap confidence prediction module, which allows RaccooNet to recall more objects using a lesser number of object proposals, thus improving its efficiency. I also designed a faster variant, RaccooNet Mobile, which is over ten times faster than the state-of-the-art (171 fps). Conducting experiments on an embedded device, I demonstrated that my algorithm is suitable for computationally resource-constrained mobile robots. I validated RaccooNet and RaccooNet Mobile on three real-world robot vision datasets (e.g., RGBD-scenes, ARID, and ETH Bahnhof) and showed that they are robust to vision challenges, for example, blur, motion, lighting, object scale. This work contributes to the field by introducing a real-time object proposal algorithm, which will serve as a foundation to new real-time object discovery methods for mobile robots.

Summarizing my doctoral research, my work contributes to building real-time object perception systems that can be deployed on real-world robotic systems that operate in the wild. This work will ultimately lead to more scalable object perception frameworks that can learn directly from the environment, on-the-fly. Moreover, my research will allow roboticists to build smarter robots that will one day become more seamlessly integrated into our daily lives, and become the useful machines that we envisioned for our future.

As robots become more common in our daily lives, they will be expected to interact with and work with teams of people. If a robot has an understanding of the underlying dynamics of a team, then it can recognize, anticipate, and adapt to human motion to be a more effective teammate.

To enable robots to understand team dynamics, I developed a new, non-linear method to detect group synchronization, which takes multiple types of discrete, task-level events into consideration. I explored this method within the context of coordinated action and validated it by applying it to both human-only and mixed human-robot teams. The results suggest that our method is more accurate in estimating group synchronization than other methods from the literature.

Building on this work, I designed a new method for robots to perceive human group behavior in real-time, anticipate future actions, and synthesize their motion accordingly.

I validated this approach within a human-robot interaction scenario, where a robot successfully and contingently coordinated with people in real-time. We found that robots perform better when they have an understanding of team dynamics than they do not.

Moreover, I investigated how the presence and behavior of robots affect group coordination in multi-human, multi-robot teams. The results suggested that group coordination was significantly degraded when a robot joined a human-only group, and was further degraded when a second robot joined the team and employed a different anticipation algorithm from the other robot. These findings suggest that heterogeneous behavior of robots in a multi-human group can play a major role in how group coordination dynamics change.

Furthermore, I designed and implemented algorithms for robots to coordinate with people in tempo-changing environments. These algorithms leveraged a human-like understanding of temporal anticipation and adaptation during the coordination process. I validated the algorithms by applying them in a human-robot drumming scenario. The results suggest that an adaptation process alone enables a robot to achieve human-level performance. Moreover, by combining anticipatory knowledge (anticipation algorithm), along with an adaptation process, a robot can be even better than people in both uniform and single tempo-changing conditions.

My research will enable robots to recognize, anticipate, and adapt to human groups. This work will help enable others in the robotics community to build more fluent and adaptable robots in the future, and provide a necessary understanding for how we design future human-robot teams.

Cognitively assistive robots have great potential to improve the accessibility of healthcare services by extending existing clinical interventions to a person's home. This provides a variety of benefits, including extending the reach of professional services, allowing people to engage with these interventions at their own convenience, and reducing risk of exposure to illness at clinics. However, there are many obstacles to deploying these robots longitudinally and autonomously, particularly for populations with lower technology literacy such as older adults. These obstacles include enabling robots to leverage the expert domain knowledge of clinicians and other stakeholders, contextualizing the robot and intervention to the lives of users, and understanding and adapting to a person's intervention preferences and goals.

The goal of my work is to design robots that can continuously learn from and adapt to people in real-world environments, which I explore in the context of delivering neurorehabilitation to people with cognitive impairments. In this dissertation, I will describe three main contributions of my work.

First, I developed new methods to recognize complex motion reflective of real-world activities to enable robots to accurately understand human intention. Recognizing human activity can help robots understand a person's state and their reactions to its behavior. My work revealed the complementary strengths of two common sensor modalities for recognizing gross and fine motion, which can be leveraged to recognize complex activities and help robots better understand human intention. In addition, I designed a novel deep learning architecture for recognizing fine motion using nonvisual sensors, enabling robots to recognize human activity in dynamic, privacy sensitive settings such as homes.

Second, I developed the first robotic system (JESSIE) which makes control synthesis accessible to novice programmers, allowing clinicians to quickly and easily specify complex robot behaviors through a tangible specification interface. Clinicians can provide robots with valuable domain and personal knowledge which can inform its behavior. My work revealed key insights regarding how robots can learn and adapt to people with cognitive impairments longitudinally at home. JESSIE makes control synthesis more accessible to novice programmers, enabling stakeholders to imbue robots with their domain knowledge and extend the reach of their work.

Third, I developed an autonomous robot (CARMEN) which extends clinical interventions to the home, and longitudinally supports goal progress and motivation. In collaboration with clinicians and people with cognitive impairments, I identified interaction design patterns for translating clinical interventions to robots in order to maintain longitudinal engagement and maximize efficacy. Furthermore, I developed a new framework for roboticists creating longitudinal, robot-delivered health interventions with collaborative goal setting capabilities. My work lays the foundation for enabling robots to support motivation and goal achievement throughout a longitudinal intervention at home.

My research contributes to building robotic systems which can longitudinally personalize their behavior to people in real-world environments. My work will transform how robots longitudinally interact with people, with the ultimate goal of enabling more safe and effective human-robot interaction, particularly for underserved populations.

Robots have the potential to revolutionize the way healthcare is delivered, particularly with regard to improving access to it. In-home robotic assistants could be used to extend care in a consistent, scalable, and personalized way. To realize this vision, we decompose the requirements of such a system, propose a target architecture, and illustrate where more research effort is needed. In particular, we focus on preference inference as a subproblem in which we aim to make direct progress.

Current preference learning techniques lack the ability to infer long-term, task-independent preferences in realistic, interactive, incomplete-information settings. To address this gap, we introduce a novel preference-inference formulation, inspired by assistive robotics applications, in which a robot must infer these kinds of preferences based only on observing the user’s behavior in various tasks. We then propose a candidate inference algorithm based on maximum-margin methods, and evaluate its performance in the context of robot-assisted prehabilitation. We find that the algorithm learns to predict aspects of the user’s behavior as it is given more data, and that it shows strong convergence properties after a small number of iterations. This result moves us towards the vision of more helpful and personal in-home robotic assistants, and demonstrates the tractability of future progress in this area.

Human-Robot Teaming in Safety-Critical Environments: Perception of and Interaction with Groups

Angelique Taylor, Ph.D. CandidateComputer Science and Engineering, UC San Diego amt062@eng.ucsd.edu

The field of robotics is growing at a rapid pace with robot deployments in everyday environments such as hospitals, schools, and retail settings. On average, 70% of people in these environments are in groups: they walk, work, and interact in groups. Recent work in the field has highlighted the importance of designing robots that can interact with groups. To enable robots to fluently assist and interact with groups, they need a high-level understanding of team dynamics, including how to sense groups and engage in intelligent decision making to support them. However, the human-robot interaction (HRI) field has focused on dyadic interaction (i.e., one human and one robot) which does not represent real-world situations where robots might interact with any number of people at a given time.

The goal of my Ph.D. research is to design robotic systems that enable robots to work seamlessly in teams in real-world, safety-critical settings. In this dissertation, I discuss four main contributions of my work. First, I designed the Robot-Centric Group Estimation model (RoboGEM), which enables robots to detect human groups in complex, real-world environments. Prior group perception work tends to: (1) focus on exo-centric perspective approaches, (2) use data captured in well-controlled environments which cannot support real-world operating scenarios, and (3) use supervised learning methods that may potentially fail when robots encounter new situations. In contrast, RoboGEM is unsupervised and works well on ego-centric, real-world data, where both pedestrians and the robot are in motion at the same time. RoboGEM outperforms the current top-performing method by 10% in terms of accuracy, and 50% in terms of recall, and it can be used in real-world environments to enable robots to work in teams.

Second, I expanded the scope of RoboGEM to design RoboGEM 2.0 which enables a robot to track groups over time in crowded environments. RoboGEM 2.0 is based on the intuition that pedestrians are most likely in groups when they have similar trajectories, ground plane coordinates, and proximities. RoboGEM 2.0 leverages deep learning techniques for group data association which enables robots to track groups when ego-motion uncertainty is high. It includes new methods for group tracking that employ Convolutional Neural Network (CNN) feature maps for group data association, and Kalman filtering to track group states over time.

I compared RoboGEM 2.0 to three state-of-the-art methods and showed that it outperforms them in terms of precision, recall, and tracking accuracy. Unlike prior methods that require multiple sensors and substantial computational resources, RoboGEM 2.0 enables robots to detect and track groups of people in real-time from a moving platform using a single RGB-D sensor.

Third, I explored using RoboGEM within a real-world application: teaming in healthcare. This is a dynamic setting in which teams experience coordination, communication, and decision-making challenges, rendering it a well-suited application domain for my work. I was interested in how robots might be used to reduce the degree of preventable patient harm, which in the US, kill over 400,000 patients and injure 5 million patients annually in hospitals alone. Here, nurses are the primary advocate for patients and thus are uniquely positioned to identify and prevent patient harm. However, strict hierarchical structures and asymmetrical power dynamics between physicians and nurses often result in penalties for nurses who speak up to “stop the line” of behavior that causes medical errors.

This inspired my work, which involved collaborating with nurses to envision how robots might empower and support them in clinical teams. For example, our study revealed that nurses want robots to assist with team decision-making, supply delivery, and team “choreography” during surgery and resuscitation procedures. This work provided exciting design concepts for future robot technology in acute settings, which inspired later work in my PhD.

Fourth, I continued my investigation into how robots can support clinical teams by exploring the use of robots in the Emergency Department (ED). The ED is a safety-critical environment in which providers are overburdened, overworked, and have limited resources to do their jobs. To place robots in these complex spaces, robots need to understand many features of the environment in order to operate safely and effectively, including patient acuity to prevent robots from interrupting treatment.

To address this, I developed the Safety-Critical Deep Q-Network (SafeDQN), a new reinforcement learning system that enables robots to socially navigate while taking patient level of acuity into account. The main contribution of this work is a new computational model of patient acuity to enable robots to socially navigate in the ED. I compared SafeDQN to three classic navigation methods, and found that SafeDQN generates the safest, quickest path in a simulated ED environment. Using SafeDQN, mobile robots can fetch and deliver supplies to ED staff in a manner that does not interrupt patient care, thereby less likely to cause patient harm

My Ph.D. research contributes to building real-time robotic systems that can work alongside people in real-world environments. My work enables robots to effectively identify groups, track them over time, and navigate and interact among them in safety-critical, real-world settings. This work will enable more robust, realistic HRI, and support safe operation of mobile robots in human-centered environments.

Preventable patient harm is the root cause of many adverse events in healthcare, and is a leading cause of mortality and morbidity worldwide. One way to address this is through career-long clinical education, often via the use of robotic patient simulator (RPS) systems. These highly realistic human-like physical or virtual platforms enable clinical learners to safely practice their diagnostic, procedural, and social interaction skills without harming real patients. However, most commercial RPS systems lack a realistic depiction of non-verbal facial cues, limiting learner engagement and immersion, which can ultimately lead to incorrect skill transfer and patient harm.

In my PhD, I have worked to address this gap by building new interactive and expressive RPS systems, whose faces are based entirely on real patients, and the system's expressions are realistic. In this dissertation, I will describe the main contributions of my work, including 1) Developing an end-to-end analysis-modeling-synthesis framework that can easily and robustly recognize, model, and synthesize patient-driven facial expressions and clinical cues on the faces of virtual and physical RPS systems, 2) Developing new methods to create accurate computational models of multiple pathologies, including stroke and Bell's Palsy, 3) successfully synthesizing these models on RPS systems, and, 4) designing the RPS as a clinical educational tool tested with clinicians.

My research opens new avenues of exploration in robotics, human-robot interaction, and health technology, and may trigger a new round of relevant technological innovations by creating the next generation of RPS technology. My work will enable roboticists to discover platform-independent methods to control the facial expressions of both robots and virtual agents, yielding new modalities for interaction. Furthermore, disseminating the results of this work to the research community will help both the broader robotics and healthcare communities employ these novel systems in their own application domains. This work serves as a bridge between robotics and healthcare research and practice, and offers promising opportunities to reduce misdiagnoses and bias in healthcare, ultimately reducing preventable patient harm.