Theodore Sumers will present his General Exam on Monday, April 12, 2021 at
1PM via Zoom.

 

The members of his committee are: Tom Griffiths (advisor), Karthik
Narasimhan, Adele Goldberg

 

Zoom link: https://princeton.zoom.us/s/5514275793

 

Everyone is invited to attend the talk, and those faculty wishing to remain
for the oral exam are welcome to do so. His abstract and reading list follow
below. 

 

Title: Communication as Optimal Control

 

Abstract: Language is a powerful tool for communication. Humans routinely
transmit vast amounts of information with a handful of words-- often by
leaving other words unsaid. Yet the unbounded expressivity of language makes
it difficult to model. As a result, psychological studies of communication
often focus on coordination games with constrained vocabularies, while
natural language interfaces assume literal, imperative statements. These
paradigms provide experimental and practical control, but limit
fully-fledged accounts of naturalistic language use and information transfer
from humans to artificial agents.

 

This exam uses reinforcement learning (RL) to explore human communication in
more complex settings. I will first introduce RL-based language games, in
which a speaker communicates abstract information (here, the reward
function) to guide listener actions. I will show that naturalistic
linguistic communication can be modeled with aspect-based sentiment analysis
to recover the speaker's reward function. This is a novel contribution which
facilitates intuitive human-agent interactions. I will then propose a
general view of communication as optimal control over latent listener belief
states. This perspective unifies belief- and action-based models of human
communication, and can be viewed as a speaker inducing "optimal rewards" in
the listener. Finally, I will close with a set of proposed experiments to
elucidate the components of this model. 

 

Reading List:

 

Books:

1.	Richard Sutton and Andrew Barto. Reinforcement Learning: An
Introduction.
2.	Michael Tomasello. The Origins of Human Communication.

 

Papers:

*	Communication and Joint Action

*	  Logic and
Conversation (Grice, 1975)
*
 Understanding and sharing intentions: The origins of
cultural cognition (Tomasello et al, 2005)
*	 
Predicting Pragmatic Reasoning in Language Games (Frank & Goodman, 2012)
*	 
Pragmatic Language Interpretation as Probabilistic Inference (Goodman &
Frank, 2016)
*	 
Experience Grounds Language (Bisk et al, 2020)

*	Human Teaching

*	  A
rational account of pedagogical reasoning: Teaching by, and learning from,
examples (Shafto et al, 2014) 
*

Faster Teaching via POMDP Planning (Rafferty et al, 2015)

*	RL and Rewards

*	  Where Do
Rewards Come From? (Singh et al, 2009)
*	  Theory
of Mind as Inverse Reinforcement Learning (Jara-Ettinger, 2019)
*	  Pragmatic-Pedagogic Value
Alignment (Fisac et al, 2017)
*	  Deep reinforcement learning from
human preferences (Christiano et al, 2017)
*	  Reward-rational Implicit Choice
(Jeon et al 2020) 

*	RL and Language

*	  A Survey of Reinforcement
Learning Informed by Natural Language (Luketina et al, 2019)
*
 Robots that Use Language (Tellex et al, 2020)
*	  Grounding Language for Transfer
in Deep Reinforcement Learning (Narasimhan et al, 2018)