UC Merced

Autonomous virtual agents have been employed in different areas, spanning applications from education and training to gaming and e-commerce. In particular, agents of human-like appearance, or virtual human agents, have been greatly improved over the years with the advancement of technologies in machine learning, natural language processing and computer graphics. This dissertation addresses topics related to the design and training of virtual human agents for collaborative tasks.

On the design front, this thesis presents user studies investigating the effect of agent gender and feedback strategies on instructional object manipulation tasks. The obtained findings show that, although agent gender has no significant effect on user preference or performance, users find female agents to be more attractive. Comparing suggestive feedback and correctness feedback strategies, it is found that correctness feedback is preferred by the users and leads to a 65% shorter task completion time.

On the training front, the focus is on collaborative tasks controlled by Deep Reinforcement Learning (DRL). Three important challenges are addressed: sequential multi-phase tasks, collaborative learning of motor tasks in a physics-based environment, and addressing realism and robustness. First, to train agents for collaborative multi-phase tasks the Constrained Multi-agent Soft Actor Critic (C-MSAC) approach is proposed. It is shown that this approach achieves better mean episode reward, generalizability and robustness to disturbance when compared with an unconstrained multi-agent learning baseline. Second, the original inverse kinematics approach is replaced with physics-based control which leads to more natural movements and improves robustness against variations to physical parameters of the environment.

Finally, the Constrained Robust Soft Actor Critic (C-RSAC) approach is proposed to train a single robust agent using the policies obtained from C-MSAC for initialization. C-RSAC uses noise augmentation in the action space of one of the agents to improve robustness of the collaborating agent. It is shown that C-RSAC leads to an improved mean episode reward compared to C-MSAC when collaborating with a noisy agent.

In summary, this thesis investigates several important aspects related to autonomous collaborative virtual human agents such as gender, appearance, feedback strategies, collaborative training, physics-based animation and robustness. The proposed C-MSAC approach for multi-phase multi-agent training and C-RSAC approach for multi-phase single agent robust training represent new contributions to the emerging area of autonomous virtual trainers. Overall the contributions from this thesis inform the design and modeling of collaborative virtual human agents, furthering the goal of enabling these agents to assist humans on various applications of virtual trainers.