UCLA

One of the main challenges of advancing task-oriented learning such as visual task planning andreinforcement learning is the lack of realistic and standardized environments for training and test-ing AI agents. Previously, researchers often relied on ad-hoc lab environments. There have beenrecent advances in virtual systems built with 3D physics engines and photo-realistic rendering forindoor and outdoor environments, but the embodied agents in those systems can only conduct sim-ple interactions with the world (e.g., walking around, moving objects, etc.). Most of the existingsystems also do not allow human participation in their simulated environments. In this work, wedesign and implement a virtual reality (VR) system, VRKitchen, with integrated functions whichi) allow human teachers to perform demonstrations to train agents (i.e., learning from demonstra-tion). ii) allow users to collect multi-modal sensor data to perform other computer vision tasks. Weprovide standardized evaluation benchmarks and data collection tools to facilitate a broad use inresearch on general computer vision research. Especially, we evaluate our collected data on humanattention prediction task.