Nikunj Saunshi will present his Pre FPO "Towards understanding self-supervised representation learning" on Monday, April 25, 2022 at 1:30pm via Zoom. Zoom link: https://princeton.zoom.us/j/93697691837 His committee is as follows: Prof. Sanjeev Arora (adviser), Prof. Danqi Chen (examiner), Prof. Chi Jin (examiner), Prof. Elad Hazan (reader), Prof. Jason Lee (reader). Title: Towards understanding self-supervised representation learning Abstract: While supervised learning sparked the deep learning boom, it has some critical pitfalls: (1) it requires an abundance of expensive labeled data, and (2) it solves tasks from scratch rather than the human-like approach of leveraging knowledge and skills acquired from prior experiences. Pre-training has emerged as an alternative and effective paradigm, where a model is first trained using easily acquirable data, and later used to solve downstream tasks of interest with much fewer labeled data than supervised learning. Pre-training using unlabeled data, a.k.a. self-supervised learning, has been especially revolutionary, with successes in diverse domains (text, vision, speech, etc.). This raises an interesting and challenging question: why should pre-training on unlabeled data help with seemingly unrelated downstream tasks? In this talk I will present my works that initiate and build theoretical frameworks to study self-supervised learning methods like contrastive learning, language modeling and self-prediction based methods. Central to the framework is the idea that pre-training helps learn low-dimensional representations of data that help solve downstream tasks of interest with linear classifiers, thus requiring little labeled data. A common theme is to mathematically show how appropriate pre-training objectives can extract the downstream signal that is implicitly encoded in the unlabeled data distribution, and how this signal can be decoded from the learned representations using linear classifiers, thus providing a formalization for transference of “skills and knowledge” across tasks. All are welcome to attend.