Announcement is updated to include Zoom link for virtual attendees.
From: lriehl@cs.princeton.edu On Behalf Of
gradinfo--- via talks
Sent: Wednesday, May 28, 2025 2:10 PM
To: 'talks'
Subject: [talks] Yunyun Wang will present her FPO "Generative Universal
Models for Speech" on Friday, May 30, 2025 at 2:00 PM in CS 302.
Yunyun Wang will present her FPO "Generative Universal Models for Speech" on
Friday, May 30, 2025 at 2:00 PM in CS 302 & Zoom.
Zoom link:
https://princeton.zoom.us/j/9195627075?pwd=a1N5a2VzYy92cEFMVjFTZUZnOGpiUT09
The members of Yunyun's committee are as follows:
Examiners: Adam Finkelstein (Adviser), Szymon Rusinkiewicz, Felix Heide
Readers: Danqi Chen, Zeyu Jin (Adobe Research)
A copy of her thesis is available upon request. Please email
gradinfo@cs.princeton.edu mailto:gradinfo@cs.princeton.edu if you would
like a copy of the thesis.
Everyone is invited to attend her talk.
Abstract follows below:
This thesis presents a comprehensive framework for controllable speech
synthesis through self-supervised generative modeling. We propose Generative
Universal Models for Speech (GUMS), a system that decomposes speech into
disentangled representations-speaker embeddings, acoustic embeddings, and
content representations, and reconstructs it using a synthesis model. Our
approach enables detailed control over speaker voice, environmental
acoustics, speech content, and speaking rate.
We introduce three key representation models. First, GR0 learns global
speaker embeddings by disentangling them from time-varying local content
without requiring speaker labels. Second, we develop content representation
models AIC and GUMS Codec that capture speech content in continuous and
quantized forms, respectively. The AIC model enforces speaker and pitch
invariance through the alteration invariant content loss. GUMS Codec builds
on the speech codec model DAC, incorporating residual vector quantization
along with speaker and pitch conditioning. The result is a highly compact,
discrete, and language-independent representation that is well-suited for
manipulation, control, and efficient transmission.
We then integrate these representations into a high-fidelity speech
synthesis model, DiTVC, based on a Diffusion Transformer architecture. DiTVC
enables direct prompting using target speaker audio instead of relying on
fixed embeddings, allowing for more expressive voice conversion and robust
prosody control. By combining these models, we achieve controllable,
high-quality speech synthesis using unlabeled, in-the-wild data. The unified
framework advances both representation learning and generation, offering an
interpretable, and editable approach to speech synthesis.
