Theodore Sumers will present his General Exam on Monday, April 12, 2021 at 1PM via Zoom.

The members of his committee are: Tom Griffiths (advisor), Karthik Narasimhan, Adele Goldberg

Zoom link: https://princeton.zoom.us/s/5514275793

Everyone is invited to attend the talk, and those faculty wishing to remain for the oral exam are welcome to do so. His abstract and reading list follow below.

Title: Communication as Optimal Control

Abstract: Language is a powerful tool for communication. Humans routinely transmit vast amounts of information with a handful of words-- often by leaving other words unsaid. Yet the unbounded expressivity of language makes it difficult to model. As a result, psychological studies of communication often focus on coordination games with constrained vocabularies, while natural language interfaces assume literal, imperative statements. These paradigms provide experimental and practical control, but limit fully-fledged accounts of naturalistic language use and information transfer from humans to artificial agents.

This exam uses reinforcement learning (RL) to explore human communication in more complex settings. I will first introduce RL-based language games, in which a speaker communicates abstract information (here, the reward function) to guide listener actions. I will show that naturalistic linguistic communication can be modeled with aspect-based sentiment analysis to recover the speaker’s reward function. This is a novel contribution which facilitates intuitive human-agent interactions. I will then propose a general view of communication as optimal control over latent listener belief states. This perspective unifies belief- and action-based models of human communication, and can be viewed as a speaker inducing “optimal rewards” in the listener. Finally, I will close with a set of proposed experiments to elucidate the components of this model. 

Reading List:

Books:

1. Richard Sutton and Andrew Barto. Reinforcement Learning: An Introduction.
2. Michael Tomasello. The Origins of Human Communication.

Papers:

• Communication and Joint Action
• Logic and Conversation (Grice, 1975)
• Understanding and sharing intentions: The origins of cultural cognition (Tomasello et al, 2005)
• Predicting Pragmatic Reasoning in Language Games (Frank & Goodman, 2012)
• Pragmatic Language Interpretation as Probabilistic Inference (Goodman & Frank, 2016)
• Experience Grounds Language (Bisk et al, 2020)
• Human Teaching
• A rational account of pedagogical reasoning: Teaching by, and learning from, examples (Shafto et al, 2014) 
• Faster Teaching via POMDP Planning (Rafferty et al, 2015)
• RL and Rewards
• Where Do Rewards Come From? (Singh et al, 2009)
• Theory of Mind as Inverse Reinforcement Learning (Jara-Ettinger, 2019)
• Pragmatic-Pedagogic Value Alignment (Fisac et al, 2017)
• Deep reinforcement learning from human preferences (Christiano et al, 2017)
• Reward-rational Implicit Choice (Jeon et al 2020) 

• RL and Language
• A Survey of Reinforcement Learning Informed by Natural Language (Luketina et al, 2019)
• Robots that Use Language (Tellex et al, 2020)
• Grounding Language for Transfer in Deep Reinforcement Learning (Narasimhan et al, 2018)