Human-like intelligence in machines has been of interest to philosophers and engineers for centuries. Today’s deep learning allows computational models to obtain meaningful representations on high-dimensional inputs such as text and images. However, understanding these representations is challenging and why they work remains opaque. Our aim goes in two directions: 1) using computational foundations of human cognition to understand and to improve deep neural networks; 2) reverse-engineering the mind by using success in deep neural networks to understand human cognitive processes. In both directions, Bayesian inference serves as a bridge between the two fields. In the first goal, we present a Bayesian interpretation of the autoregressive objective under several distributional assumptions to show where the optimal content of language model embeddings can be identified. In the second goal, we show how variants of prior distributions in variational auto-encoders, a probabilistic generative model, explain how forming prototypes contributes to semi-supervised categorization.

https://docs.google.com/document/d/13_DPdcPTKklMbzzabiUXY7q-BiLmjnBJGoPZvge5q5o/edit?usp=sharing