UCLA

This thesis advances multi-agent (MA) systems designed to emulate human-like perception. It focuses on the use of first-person partial RGB visual perception alongside limited state observations such as tools, inventory, and body status, to support MA cooperation in a 3D, open-ended environment, specifically that in the game of Minecraft. Central to this exploration are two novel contributions: The first introduces MA-Minecraft, a comprehensive benchmark in the Minecraft environment that challenges agents with RGB image and language inputs in dynamic interaction scenarios. The second presents a novel control strategy, MA offline imitation learning via social Gradient Fields (MAGF), which leverages the same pixel-based partial vision inputs for precise, low-level action execution during MA interactions.Experimental results validate the efficacy of our approach by demonstrating the ability of trained agents to effectively collaborate in complex environments. Together, these contributions aim to enhance strategic decision-making and operational precision in partial observation settings, pushing the boundaries of how embodied agents perceive and interact in complex 3D environments.