Xindi Wu will present her General Exam "Data-Centric Multimodal Machine Learning" on Wednesday, January 31, 2024 at 3:00 PM in CS 402 and via zoom.

Zoom link: https://princeton.zoom.us/j/95305696158

Committee Members: Olga Russakovsky (advisor), Szymon Rusinkiewicz, Adji Bousso Dieng

Abstract:

Data is a cornerstone of innovation in multimodal machine learning. With the massive amount of vision-language data available, figuring out how to effectively use it becomes a major challenge. Dataset distillation methods aim to scale down large-scale datasets to significantly smaller sets of synthetic training examples while preserving sufficient information for model training. 

Current dataset distillation methods have shown success in condensing large-scale image datasets into smaller, yet effective, training examples for image classification. However, with the rise in capabilities of vision-language models (VLMs), and especially given the scale of datasets necessary to train these models, the time is ripe to expand dataset distillation methods beyond image classification. This task is challenging, given the nature of vision-language datasets, which typically lack discrete classes and involve complex image-text relationships.

Our research marks a pioneering step in this direction. We take the first steps towards this goal by expanding the idea of trajectory matching to create a distillation method for vision-language datasets. Our method significantly reduces the required dataset size while maintaining the model's performance. We hope that our work lays the groundwork for future research aimed at understanding what is the minimum information required for a vision-language model to achieve comparable performance quickly, thereby building a better understanding of the compositionality of compact visual-linguistic knowledge.

Reading List:

https://docs.google.com/document/d/1A5ShHxlxNgs_E4r5iIs4Y1D1L-nUfp9Z68Vls6mmv7c/edit?usp=sharing

Everyone is invited to attend the talk, and those faculty wishing to remain for the oral exam following are welcome to do so.