- talks - lists.cs.princeton.edu

Ameet S. Deshpande will present his General Exam "Guiding Attention for Self-Supervised Learning with Transformers" on Thursday, April 29, 2021 at 11AM via Zoom.
by jfarquer＠cs.princeton.edu 06 Apr '21

06 Apr '21

Ameet S. Deshpande will present his General Exam on Thursday, April 29, 2021 at 11AM via Zoom. The members of his committee are as follows: Karthik Narasimhan (advisor), Danqi Chen, Sanjeev Arora Zoom link: <http://princeton.zoom.us/my/ameetsd> http://princeton.zoom.us/my/ameetsd Everyone is invited to attend the talk, and those faculty wishing to remain for the oral exam following are welcome to do so. His abstract and reading list follow below. Title: Guiding Attention for Self-Supervised Learning with Transformers Abstract: Recent advances in self-supervised pre-training have resulted in models with impressive downstream performance on several natural language processing (NLP) tasks. However, this has led to the development of enormous models, which often require days of training on non-commodity hardware (e.g., TPUs, distributed GPUs). Furthermore, studies have shown that it is quite challenging to successfully train these large Transformer models, requiring complicated learning schemes and extensive hyperparameter tuning. In this talk, we present a simple and effective technique for efficient self-supervised learning with bi-directional Transformers. Our approach is motivated by recent studies demonstrating that trained models' self-attention patterns contain a majority of non-linguistic regularities. We propose a computationally efficient auxiliary loss function to guide attention heads to conform to such patterns. Our method is agnostic to the actual pre-training objective. It results in faster convergence of models and better performance on downstream tasks compared to the baselines, achieving state-of-the-art results in low-resource settings. Reading List: Textbooks: 1. <https://web.stanford.edu/~jurafsky/slp3/ed3book_dec302020.pdf> Speech and Language Processing, by Jurafsky and Martin 2. <https://github.com/jacobeisenstein/gt-nlp-class/blob/master/notes/eisenstei n-nlp-notes.pdf> Natural Language Processing, by Jacob Eisenstein Papers: Transformers and pre-training in NLP 1. Vaswani, Ashish, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. "Attention is all you need." 2. Devlin, Jacob, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. "Bert: Pre-training of deep bidirectional transformers for language understanding." 3. Liu, Yinhan, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. "Roberta: A robustly optimized bert pretraining approach." 4. Clark, Kevin, Minh-Thang Luong, Quoc V. Le, and Christopher D. Manning. "Electra: Pre-training text encoders as discriminators rather than generators." Modifying self-attention modules in Transformers 1. Kitaev, Nikita, Łukasz Kaiser, and Anselm Levskaya. "Reformer: The efficient transformer." 2. Child, Rewon, Scott Gray, Alec Radford, and Ilya Sutskever. "Generating long sequences with sparse transformers." 3. Beltagy, Iz, Matthew E. Peters, and Arman Cohan. "Longformer: The long-document transformer." 4. Tay, Yi, Dara Bahri, Donald Metzler, Da-Cheng Juan, Zhe Zhao, and Che Zheng. "Synthesizer: Rethinking self-attention in transformer models." 5. Wang, Sinong, Belinda Li, Madian Khabsa, Han Fang, and Hao Ma. "Linformer: Self-attention with linear complexity." 6. Zaheer, Manzil, Guru Guruganesh, Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontanon, Philip Pham et al. "Big bird: Transformers for longer sequences." Fixing self-attention patterns (Machine Translation) 1. You, Weiqiu, Simeng Sun, and Mohit Iyyer. "Hard-coded gaussian attention for neural machine translation." 2. Raganato, Alessandro, Yves Scherrer, and Jörg Tiedemann. "Fixed encoder self-attention patterns in transformer-based machine translation." Analyzing self-attention 1. Clark, Kevin, Urvashi Khandelwal, Omer Levy, and Christopher D. Manning. "What does bert look at? an analysis of bert's attention." 2. Kovaleva, Olga, Alexey Romanov, Anna Rogers, and Anna Rumshisky. "Revealing the dark secrets of BERT." 3. Voita, Elena, David Talbot, Fedor Moiseev, Rico Sennrich, and Ivan Titov. "Analyzing multi-head self-attention: Specialized heads do the heavy lifting, the rest can be pruned." 4. Michel, Paul, Omer Levy, and Graham Neubig. "Are sixteen heads really better than one?."

1 0

Qipeng Liu will present his Pre FPO "Cryptography in the age of quantum computers" on Monday, April 12, 2021 at 12pm via Zoom.
by jfarquer＠cs.princeton.edu 06 Apr '21

06 Apr '21

Qipeng Liu will present his Pre FPO "Cryptography in the age of quantum computers" on Monday, April 12, 2021 at 12pm via Zoom. Zoom link: <https://princeton.zoom.us/j/6541841088> https://princeton.zoom.us/j/6541841088 Committee: Mark Zhandry (advisor, examiner) Mark Braverman (examiner) Matt Weinberg (examiner) Ran Raz (reader) Gillat Kol (reader) Talk abstract follows below. All are welcome to attend. Abstract: It is widely believed that full-scale quantum computers will eventually be viable. Quantum computers pose threats to many existing cryptosystems (most prominently, Shor's algorithm), while raising the possibility of using quantum mechanical phenomena to achieve never-before-possible capabilities. My talk consists of the following topics: understanding security of existing cryptosystems in a quantum world and developing new schemes with the power of quantum computers. First, I will present my work on quantum query complexity: I will show tight bounds for multi-collision finding problem, and tight time-space tradeoffs for function inversion problem. The latter shows that even with a piece of preprocessed quantum advice, Grover's search can not be sped up. This technique can be extended to prove post-quantum non-uniform security of many existing cryptographic schemes. Second, I will present my work on post-quantum zero-knowledge proof: I will start by showing post-quantum constant-round zero-knowledge protocols with black-box simulators for NP do not exist, unless NP is in BQP; then I will show how to achieve non-interactive zero-knowledge, by proving the famous Fiat-Shamir transformation is secure in the quantum random oracle model. Third, I will present my work on quantum copy-protection, which aims to use the unclonability of quantum states to achieve programs that cannot be copied. Towards this goal, I will show that any unlearnable function can be copy-protected relative to classical oracles. For specific cryptographic functionalities (such as signatures and RPFs), I will show that they can be copy-protected in the plain model.

1 0

Johan Ospina will present his MSE Talk and Thesis "Crowdflow - creating a crowdsourced optical flow dataset" on Tuesday, April 20, 2021 at 3:30PM via Zoom.
by jfarquer＠cs.princeton.edu 01 Apr '21

01 Apr '21

Johan Ospina will present his MSE Talk and Thesis "Crowdflow - creating a crowdsourced optical flow dataset" on Tuesday, April 20, 2021 at 3:30PM via Zoom. Zoom Link: <https://princeton.zoom.us/j/6951559457> https://princeton.zoom.us/j/6951559457 Committee Members: Jia Deng (adviser) and Olga Russakovsky (reader) All are welcome to attend. Title: Crowdflow - creating a crowdsourced optical flow dataset. Abstract: Datasets for training Deep Learning solutions for Optical Flow have been for the most part created in synthetic rendering environments or in highly structured real life environments without much variance of environment. While CGI methods are able to produce accurate training data for learning algorithms, they are limited to either the quality of the rendering algorithms used or the variation of the library of 3D models they have access to. On the other end of the spectrum, the real life datasets that exist are limited in the types of environments they can image. Crowdflow aims to bring a crowdsourced dataset of sparse Optical Flow annotations in a wide variety of real life environments in order to measure how well current state of the art methods perform on a variety of real life scenes.

1 0

Ryan Amos will present his Pre FPO "Consumer protection with longitudinal data collection and analysis" on Thursday, April 1, 2021 at 1pm via Zoom.
by Nicki Mahler 01 Apr '21

01 Apr '21

Ryan Amos will present his Pre FPO "Consumer protection with longitudinal data collection and analysis" on Thursday, April 1, 2021 at 1pm via Zoom. Zoom link: [ https://princeton.zoom.us/j/98956078851?pwd=UWxReS9DcWc3aFlTWEFydTJTVXlnQT09 | https://princeton.zoom.us/j/98956078851?pwd=UWxReS9DcWc3aFlTWEFydTJTVXlnQT09 ] Committee: Examiners: Prateek Mittal (adviser), Ed Felten (adviser), Arvind Narayanan Readers: Jonathan Mayer, Joe Calandrino ( Federal Trade Commission) All are welcome to attend. Title: Consumer protection with longitudinal data collection and analysis Abstract: We'll discuss how we can use longitudinal data to understand issues surrounding consumer protection, particularly issues driven by misaligned incentives in advertising. In the first half, I'll present our work on collecting and analyzing our dataset of one million privacy policies from a twenty year timespan. Our work adds to the growing body demonstrating the inadequacy of the notice-and-choice model of privacy policy regulation. We find that privacy policies have become more difficult to read, and, by comparing our findings to prior web measurements, that they show a troubling lack of transparency. Our work helps contextualize the impact of the GDPR by showing how it stands out against a decade of privacy policy changes. In the second half, I'll present our ongoing work on collecting and analyzing a dataset of Yelp's review recommendation system, which we take as a proxy for fake review filtering. To understand the evolving game between fake reviewers and platforms, we frame our work differently from prior work -- rather than focusing on the reviews themselves, we focus on the platform's reactions to reviews. We find that a substantial proportion of filtered-out reviews are later reclassified, suggesting that Yelp takes an aggressive approach to removing reviews. I'll additionally discuss our ongoing work in this field and questions we want to answer.

1 1

CS Colloquium Speaker: Sarah Dean, today at 4:30pm
by Emily C. Lawrence 29 Mar '21

29 Mar '21

CS Colloquium Speaker Sarah Dean, University of California, Berkeley Host: Elad Hazan Date: Monday, March 29 Time: 4:30pm EST Event page: [ https://www.cs.princeton.edu/events/26052 | https://www.cs.princeton.edu/events/26052 ] Register for Zoom webinar here: [ https://princeton.zoom.us/webinar/register/WN_tGTox7xySEK97iKWXtIOSg | https://princeton.zoom.us/webinar/register/WN_tGTox7xySEK97iKWXtIOSg ] Title: Reliable Machine Learning in Feedback Systems Abstract: Machine learning techniques have been successful for processing complex information, and thus they have the potential to play an important role in data-driven decision-making and control. However, ensuring the reliability of these methods in feedback systems remains a challenge, since classic statistical and algorithmic guarantees do not always hold. In this talk, I will provide rigorous guarantees of safety and discovery in dynamical settings relevant to robotics and recommendation systems. I take a perspective based on reachability, to specify which parts of the state space the system avoids (safety) or can be driven to (discovery). For data-driven control, we show finite-sample performance and safety guarantees which highlight relevant properties of the system to be controlled. For recommendation systems, we introduce a novel metric of discovery and show that it can be efficiently computed. In closing, I discuss how the reachability perspective can be used to design social-digital systems with a variety of important values in mind. Bio: Sarah is a PhD candidate in the Department of Electrical Engineering and Computer Science at UC Berkeley, advised by Ben Recht. She received her MS in EECS from Berkeley and BSE in Electrical Engineering and Math from the University of Pennsylvania. Sarah is interested in the interplay between optimization, machine learning, and dynamics in real-world systems. Her research focuses on developing principled data-driven methods for control and decision-making, inspired by applications in robotics, recommendation systems, and developmental economics. She is a co-founder of a transdisciplinary student group, Graduates for Engaged and Extended Scholarship in computing and Engineering, and the recipient of a Berkeley Fellowship and a NSF Graduate Research Fellowship.

1 0

Reminder: Sarah Dean, tomorrow at 4:30
by Emily C. Lawrence 28 Mar '21

28 Mar '21

CS Colloquium Speaker Sarah Dean, University of California, Berkeley Host: Elad Hazan Date: Monday, March 29 Time: 4:30pm EST Event page: [ https://www.cs.princeton.edu/events/26052 | https://www.cs.princeton.edu/events/26052 ] Register for Zoom webinar here: [ https://princeton.zoom.us/webinar/register/WN_tGTox7xySEK97iKWXtIOSg | https://princeton.zoom.us/webinar/register/WN_tGTox7xySEK97iKWXtIOSg ] Title: Reliable Machine Learning in Feedback Systems Abstract: Machine learning techniques have been successful for processing complex information, and thus they have the potential to play an important role in data-driven decision-making and control. However, ensuring the reliability of these methods in feedback systems remains a challenge, since classic statistical and algorithmic guarantees do not always hold. In this talk, I will provide rigorous guarantees of safety and discovery in dynamical settings relevant to robotics and recommendation systems. I take a perspective based on reachability, to specify which parts of the state space the system avoids (safety) or can be driven to (discovery). For data-driven control, we show finite-sample performance and safety guarantees which highlight relevant properties of the system to be controlled. For recommendation systems, we introduce a novel metric of discovery and show that it can be efficiently computed. In closing, I discuss how the reachability perspective can be used to design social-digital systems with a variety of important values in mind. Bio: Sarah is a PhD candidate in the Department of Electrical Engineering and Computer Science at UC Berkeley, advised by Ben Recht. She received her MS in EECS from Berkeley and BSE in Electrical Engineering and Math from the University of Pennsylvania. Sarah is interested in the interplay between optimization, machine learning, and dynamics in real-world systems. Her research focuses on developing principled data-driven methods for control and decision-making, inspired by applications in robotics, recommendation systems, and developmental economics. She is a co-founder of a transdisciplinary student group, Graduates for Engaged and Extended Scholarship in computing and Engineering, and the recipient of a Berkeley Fellowship and a NSF Graduate Research Fellowship.

1 0

CS Systems Group Speaker: Guanhua Wang, Thursday, April 1 at 3:00pm
by Emily C. Lawrence 25 Mar '21

25 Mar '21

CS System Group Speaker Guanhua Wang, UC Berkeley Host: Wyatt Lloyd Date: Thursday, April 1 Time: 3:00pm ET Event page: [ https://www.cs.princeton.edu/events/26062 | https://www.cs.princeton.edu/events/26062 ] Zoom meeting link: [ https://princeton.zoom.us/j/906163778 | https://princeton.zoom.us/j/906163778 ] Title: Disruptive Research on Distributed ML Systems Abstract: Deep Neural Networks (DNNs) enable computers to excel across many different applications such as image classification, speech recognition and robotics control. To accelerate DNN training and serving, parallel computing is widely adopted. System efficiency is a big issue when scaling out. In this talk, I will make three arguments towards better system efficiency in distributed DNN training and serving. First, Ring All-Reduce for model synchronization is not optimal, but Blink is. By packing spanning trees rather than forming rings, Blink achieves higher flexibility in arbitrary networking environments and provides near-optimal network throughput. Blink is filed as a US patent and is being used by Microsoft. Blink gains lots of attention from industry, such as Facebook (distributed PyTorch team), ByteDance (parent company of TikTok app). Blink was also featured on Nvidia GTC China 2019 and news from Baidu, Tencent, etc. Second, communication can be eliminated via sensAI's class parallelism. sensAI decouples a multi-task model into disconnected subnets, each is responsible for decision making of a single task. sensAI's attribute of low-latency, real-time model serving attracts several Venture Capitals in the Bay Area. Third, Wavelet is more efficient than gang-scheduling. By intentionally adding task launching latency, Wavelet interleaves peak memory usage across different waves of training tasks on the accelerators, and thus it improves both computation and on-device memory utilization. Bio: Guanhua Wang is a final year CS PhD in the RISELab at UC Berkeley, advised by Prof. Ion Stoica. His research lies primarily in the ML+System area including fast collective communication schemes for model synchronization, efficient in-parallel model training and real-time model serving.

1 0

CS Colloquium Speaker: Sarah Dean, Monday, March 29 at 4:30
by Emily C. Lawrence 23 Mar '21

23 Mar '21

CS Colloquium Speaker Sarah Dean, University of California, Berkeley Host: Elad Hazan Date: Monday, March 29 Time: 4:30pm EST Event page: [ https://www.cs.princeton.edu/events/26052 | https://www.cs.princeton.edu/events/26052 ] Register for Zoom webinar here: [ https://princeton.zoom.us/webinar/register/WN_tGTox7xySEK97iKWXtIOSg | https://princeton.zoom.us/webinar/register/WN_tGTox7xySEK97iKWXtIOSg ] Title: Reliable Machine Learning in Feedback Systems Abstract: Machine learning techniques have been successful for processing complex information, and thus they have the potential to play an important role in data-driven decision-making and control. However, ensuring the reliability of these methods in feedback systems remains a challenge, since classic statistical and algorithmic guarantees do not always hold. In this talk, I will provide rigorous guarantees of safety and discovery in dynamical settings relevant to robotics and recommendation systems. I take a perspective based on reachability, to specify which parts of the state space the system avoids (safety) or can be driven to (discovery). For data-driven control, we show finite-sample performance and safety guarantees which highlight relevant properties of the system to be controlled. For recommendation systems, we introduce a novel metric of discovery and show that it can be efficiently computed. In closing, I discuss how the reachability perspective can be used to design social-digital systems with a variety of important values in mind. Bio: Sarah is a PhD candidate in the Department of Electrical Engineering and Computer Science at UC Berkeley, advised by Ben Recht. She received her MS in EECS from Berkeley and BSE in Electrical Engineering and Math from the University of Pennsylvania. Sarah is interested in the interplay between optimization, machine learning, and dynamics in real-world systems. Her research focuses on developing principled data-driven methods for control and decision-making, inspired by applications in robotics, recommendation systems, and developmental economics. She is a co-founder of a transdisciplinary student group, Graduates for Engaged and Extended Scholarship in computing and Engineering, and the recipient of a Berkeley Fellowship and a NSF Graduate Research Fellowship.

1 0

Khiem Ngo will present his Pre-FPO "Tolerating Slowdowns in Replicated State Machines using Copilots" on Tuesday, March 23, 2021 at 3pm via Zoom.
by Nicki Mahler 22 Mar '21

22 Mar '21

Khiem Ngo will present his Pre-FPO "Tolerating Slowdowns in Replicated State Machines using Copilots" on Tuesday, March 23, 2021 at 3pm via Zoom. Zoom link: [ https://princeton.zoom.us/j/98688513083 | https://princeton.zoom.us/j/98688513083 ] Committee Members: Wyatt Lloyd (Advisor) Michael Freedman (Examiner) Ravi Netravali (Examiner) Amit Levy (Reader) Siddhartha Sen (MSR, Reader) All are welcome to attend. Title: Tolerating Slowdowns in Replicated State Machines using Copilots Abstract. Replicated state machines (RSMs) are linearizable, fault-tolerant groups of replicas coordinated by a consensus algorithm. Linearizability gives the RSM the illusion of being a single machine that responds to client commands one by one. Fault-tolerance enables the RSM to continue operating despite the failure of a minority of replicas. RSMs are used throughout large-scale systems, such as distributed databases, cloud storage, and service managers. At such scale, it is common for some machines to be slow. The slowdowns manifest as machines whose latency for responding to other machines is higher than usual. Thus, RSMs should also be slowdown-tolerant, i.e., provide similar performance despite the presence of slow replicas. Unfortunately, no existing consensus protocol is slowdown-tolerant: a single slow replica can sharply increase their latency. This increased latency decreases availability because a service that does not respond in time is not meaningfully available. In this talk, I will first define s-slowdown-tolerance and explain why existing consensus protocols are not slowdown-tolerant. Then I will introduce Copilot replication, the first 1-slowdown-tolerant consensus protocol. Copilot replication uses two pilots to ensure the RSM stays fast, by using proactive redundancy in all stages of processing a client command. Our evaluation shows that Copilot delivers normal latency despite the slowdown of any 1 replica.

1 0

CS Colloquium Speaker: Riad Wahby, today at 12:30pm
by Emily C. Lawrence 22 Mar '21

22 Mar '21

CS Colloquium Speaker Riad Wahby, Stanford University Host: Andrew Appel Date: Monday, March 22 Time: 12:30pm EST Event page: [ https://www.cs.princeton.edu/events/26050 | https://www.cs.princeton.edu/events/26050 ] Register for Zoom webinar here: [ https://princeton.zoom.us/webinar/register/WN_SMKdFMoQRo2Cj3zvU57zyw | https://princeton.zoom.us/webinar/register/WN_SMKdFMoQRo2Cj3zvU57zyw ] Title: Probabilistic Proofs: Theory, Hardware, and Everything in Between Abstract: In the past decade, systems that use probabilistic proofs in real-world applications have seen explosive growth. These systems build upon some of the crown jewels of theoretical computer science---interactive proofs, probabilistically checkable proofs, and zero-knowledge proofs---to solve problems of trust and privacy in a wide range of settings. This talk describes my work building systems that answer questions ranging from "how can we build trustworthy hardware that uses untrusted components?" to "how can we reduce the cost of verifying smart contract execution in blockchains?" Along the way, I will discuss the pervasive challenges of efficiency, expressiveness, and scalability in this research area; my approach to addressing these challenges; and future directions that promise to bring this exciting technology to bear on an even wider range of applications. Bio: Riad S. Wahby is a Ph.D. candidate at Stanford, advised by Dan Boneh and Keith Winstein. His research interests include systems, computer security, and applied cryptography. Prior to attending Stanford, Riad spent ten years as an analog and mixed-signal integrated circuit designer. Riad and his collaborators received a 2016 IEEE Security and Privacy Distinguished Student Paper award; his work on hashing to elliptic curves is being standardized by the IETF.

1 0