This Friday, Prof. Yitao Liang from Peking University will give a virtual talk about his research on Vision-Language-Action Models and their applications to open-world games (e.g., Minecraft). Time : May 9, from 10:00 to 11:00 a.m. ET at [ https://princeton.zoom.us/j/97152277086 | https://princeton.zoom.us/j/97152277086 ] We will also stream the talk together at Friend Center 008 . Title: Instruction-Following Control in an Open World: Vision-Language-Action Models and Beyond Abstract: With the emergence of large language models, the debate about whether an instruction-following long-horizon control agent will appear has resurfaced. However, GPT's emerging abilities seem difficult to reproduce in domains that require sequential control. In this talk, we will introduce various efforts by our group and other well-known research labs to develop such a model in open-world game domains (e.g., Minecraft). Due to its extremely high degree of freedom, traditional multi-task data-driven methods are unsustainable (we cannot afford to train on thousands of tasks simultaneously, it is too expensive). One possible direction is to leverage LLMs to achieve flexible task planning. In this talk, we present latest research in this direction, and how to obtain a universal prompt-based generic policy learning. Due to models’s increasingly impressive data fitting capability, how to properly evaluate in open-ended domains has also become a new frontier. We will also discuss how generative evaluation could facilitate a more principled evaluation. Bio: Yitao Liang ( [ mailto:yitaol@pku.edu.cn | yitaol@pku.edu.cn ] ) is an assistant professor and Boya Fellow at Peking University. He obtained his Ph.D. degree in Computer Science from UCLA, advised by Prof. Guy Van den Broeck. His research interests span reasoning, machine learning and AI agents. His work has received recognition from top AI conferences; for example, the best-paper honorable mention from AAMAS 2016, the best paper from RL for Real Life workshop in ICML 2019, a best paper runner-up from the LLD workshop in NeurIPS 2017, a best paper from the TEACH workshop in ICML2023. He regularly serves as area chairs and senior area chairs in top venues e.g., NeurIPS, ICML. Best, Wenzhe Li & Chi Jin