- talks - lists.cs.princeton.edu

Tyler Zhu will present his General Exam "Unifying Specialized Visual Encoders for Video Language Models" on Tuesday, January 14, 2025 at 10:00 AM in CS 402 and via zoom.
by CS Grad Department 09 Jan '25

09 Jan '25

Tyler Zhu will present his General Exam " Unifying Specialized Visual Encoders for Video Language Models " on Tuesday, January 14, 2025 at 10:00 AM in CS 402 and via zoom. Zoom link: https://princeton.zoom.us/my/tylerzhu Committee Members: Olga Russakovsky (advisor), Jia Deng , Danqi Chen Abstract: The recent advent of Large Language Models (LLMs) has ushered sophisticated reasoning capabilities into the realm of video through Video Large Language Models (VideoLLMs). However, VideoLLMs currently rely on a single vision encoder for all of their visual processing, which limits the amount and type of visual information that can be conveyed to the LLM. Our method, MERV, Multi-Encoder Representation of Videos, instead leverages multiple frozen visual encoders to create a unified representation of a video, providing the VideoLLM with a comprehensive set of specialized visual knowledge. Spatio-temporally aligning the features from each encoder allows us to tackle a wider range of open-ended and multiple-choice video understanding questions and outperform prior state-of-the-art works on their data mixes. MERV is up to 3.7% better in accuracy than Video-LLaVA across the standard suite video understanding benchmarks, while also having a better Video-ChatGPT score. We also improve upon SeViLA, the previous best on zero-shot Perception Test accuracy, by 2.2%. MERV introduces minimal extra parameters and trains faster than equivalent single-encoder approaches. Finally, we provide qualitative evidence that our model captures domain knowledge from each encoder simultaneously, such as on the motion classification tasks found in Something-Something v2. Our results offer promising directions for future research in utilizing multiple vision encoders for comprehensive video understanding. Reading List: https://docs.google.com/document/d/1jBXhv-IhFRsFsC6HblTb9ZE4nZkQIfGgm2-NAMO… Everyone is invited to attend the talk, and those faculty wishing to remain for the oral exam following are welcome to do so.

1 0

Program in Linguistics Lecture ~ "Prediction and Locality in Language and Language Models" ~ Richard Futrell, University of California, Irvine - December 19 at 4:30pm
by Emily C. Lawrence 17 Dec '24

17 Dec '24

You are invited to attend a Program in Linguistics Lecture on Thursday, December 19 th : Prediction and Locality in Language and Language Models Richard Futrell Department of Language Science University of California, Irvine Thursday, December 19, 2024 4 :30- 6 :00 p.m. 1-S-5 Green Hall I argue that language models share information-processing constraints with humans, arising from the shared core task of language models and the brain: predicting upcoming input. I show that we can use constraints on sequential prediction to explain aspects of human language processing, as reflected in eyetracking and reading time data, as well as various universal properties of languages, as reflected in patterns of word order in massively cross-linguistic text corpora. Furthermore, the same constraints explain language model performance when learning human-like versus non-human-like languages. I argue that this approach points to an information-theoretic explanation for the nature of human language, and links the science of language with neuroscience and machine learning. Richard Futrell is Associate Professor of Language Science at the University of California, Irvine. His research focuses on language in humans and machines.

1 0

Program in Linguistics Lecture ~ "Understanding ambiguity" ~ Gregory Scontras, University of California, Irvine - December 10 at 4:30pm
by Emily C. Lawrence 06 Dec '24

06 Dec '24

You are invited to attend a Program in Linguistics Lecture on Tuesday, December 10 th : Understanding ambiguity Gregory Scontras Department of Language Science University of California, Irvine Tuesday, December 10, 2024 4 :30- 6 :00 p.m. 1-S-5 Green Hall Ambiguity is everywhere, yet it rarely leads to catastrophes in communication. In this talk, I present a computational cognitive model of the social reasoning that we perform when resolving ambiguities. I motivate the model on the basis of a case study in which young children perform quite differently from adults when it comes to understanding utterances with multiple potential meanings. There, we see that pragmatic factors may play a larger role than grammatical processing factors in explaining children’s non-adult-like behavior, and the computational model offers a hypothesis as to why that’s so: pragmatic factors have a larger impact on the truth-value judgment task that researchers use to investigate young children’s language understanding. I then provide additional support for the proposed model of ambiguity resolution, first from adult behavioral experiments, then from the largest-of-its-kind annotated corpus of naturally-occurring ambiguities. Finally, I show how the proposed view of ambiguity resolution as recursive social reasoning offers additional insight into the benefits of having ambiguity in language: by observing how others resolve ambiguity, we learn about the private thoughts they use to do so. Gregory Scontras is a linguist and cognitive scientist specializing in semantics, pragmatics, and their intersection with human cognition. He received his B.A. in linguistics and in philosophy from Boston University, and his Ph.D. in linguistics from Harvard. Scontras was a post-doctoral scholar in the Computation & Cognition Lab at Stanford Psychology. In 2016, he joined the University of California, Irvine, to found the Department of Language Science, where he now directs [ https://www.langsci.uci.edu/scontras/meaning/ | the meaning lab ] . Scontras’s research uses a range of qualitative, experimental, and computational methods, applied to a diverse set of cross-linguistic data — all in service of understanding the mental representations of our speech and the computational system that manipulates them. Ongoing projects investigate the cross-linguistic semantics of measurement and degrees, the development and maintenance of complexity in unbalanced (heritage) bilingualism, the communicative pressures driving cross-linguistic ordering regularities, and the pragmatic reasoning speakers engage in to arrive at meaning in context. ***** Please forward this announcement to faculty and students in your department.

1 0

DeCenter Seminar Series: Tues, Dec 10 at 4:30
by Emily C. Lawrence 04 Dec '24

04 Dec '24

Tuesday, December 10th , at 4:30 PM , the DeCenter Seminar Series continues with Muriel Médard, co-founder of Optimum and NEC Chair of Software Science and Engineering at MIT, where she is a Professor in EECS. Currently on leave from MIT, she will present a seminar titled "The Data Socket - A Canonical Construct for Web 3 and the Coding to Enable It." The event will take place in the Friend Center Convocation Room (113) . Abstract: The success of Web 2 has relied on powerful abstractions and their realization. That of Web 3 requires the same. In Web 2, TCP/IP mimics a virtual dedicated link between nodes, while operating over a heterogeneous, meshed network. For Web 3, the Optimum Data Socket (ODS) provides data access over a virtual dedicated memory instantiated over a permissionless, decentralized, possibly unreliable set of nodes. ODS gives atomic, consistent, and durable data access. Realizing ODS requires deploying new erasure coding techniques, that are infinitely composable, stateless, and optimum in permissionless systems. Bio: Muriel Médard is the co-founder of Optimum, on leave from MIT, where she holds the NEC Chair of Software Science and Engineering for the School of Engineering at MIT, and is a Professor in EECS. She obtained three bachelor’s degrees, her M.S. and Sc.D., all from MIT. Muriel is a Member of the US National Academy of Engineering (elected 2020), a Member of the German National Academy of Sciences Leopoldina (elected 2022), a Fellow of the US National Academy of Inventors (elected 2018), American Academy of Arts and Sciences (elected 2021), and a Fellow of the Institute of Electrical and Electronics Engineers (elected 2008). She holds Honorary Doctorates from the Technical University of Munich (2020), the University of Aalborg (2022) and the Budapest University of Technology and Economics (2023). Muriel was awarded the 2022 IEEE Kobayashi Computers and Communications Award. She received the 2019 Best Paper award for IEEE Transactions on Network Science and Engineering, the 2018 ACM SIGCOMM Test of Time Paper Award, as well as nine conference paper awards. Muriel served as the Editor-in-Chief (EIC) of IEEE Transactions on Information Theory and was EiC of IEEE JSAC. She was president of the IEEE Information Theory Society. Muriel received the inaugural MIT Postdoctoral Association Mentoring Award in 2022, the inaugural MIT EECS Graduate Student Association Mentor Award, voted by the students, in 2013. She set up the Women in the Information Theory Society (WithITS) and Information Theory Society Mentoring Program. She was recognized with the 2017 IEEE Aaron Wyner Distinguished Service Award. Muriel has over seventy US and international patents awarded, the vast majority of which have been licensed or acquired. Muriel has supervised over 40 master students, over 20 doctoral students and over 25 postdoctoral fellows.

1 0

CS Colloquium Speakers: week of March 4
by Emily C. Lawrence 03 Dec '24

03 Dec '24

CS Colloquium Speakers Speaker: Sewon Min, University of Washington Date: Monday, March 4 Time: 12:30pm EST Location: CS 105 Host: Danqi Chen Event page: https://www.cs.princeton.edu/events/26579 Register for live-stream online here: https://princeton.zoom.us/webinar/register/WN_5uVGEpXNTVe8eBNSLpxKtg Title: Rethinking Data Use in Large Language Models Abstract: Large language models (LMs) such as ChatGPT have revolutionized natural language processing and artificial intelligence more broadly. In this talk, I will discuss my research on understanding and advancing these models, centered around how they use the very large text corpora they are trained on. First, I will describe our efforts to understand how these models learn to perform new tasks after training, demonstrating that their so-called in context learning capabilities are almost entirely determined by what they learn from the training data. Next, I will introduce a new class of LMs—nonparametric LMs—that repurpose this training data as a data store from which they retrieve information for improved accuracy and updatability. I will describe my work on establishing the foundations of such models, including one of the first broadly used neural retrieval models and an approach that simplifies a traditional, two-stage pipeline into one. I will also discuss how nonparametric models open up new avenues for responsible data use, e.g., by segregating permissive and copyrighted text and using them differently. Finally, I will envision the next generation of LMs we should build, focusing on efficient scaling, improved factuality, and decentralization. Bio: Sewon Min is a Ph.D. candidate in the Paul G. Allen School of Computer Science & Engineering at the University of Washington. Her research focuses on language models (LMs): studying the science of LMs, and designing new model classes and learning methods that make LMs more performant and flexible. She also studies LMs in information-seeking, legal, and privacy contexts. She is a co-organizer of multiple tutorials and workshops, including most recently at ACL 2023 on Retrieval-based Language Models and Applications and upcoming at ICLR 2024 on Mathematical and Empirical Understanding of Foundation Models. She won a paper award at ACL 2023, received a J.P. Morgan Fellowship, and was named an EECS rising star in 2022. Speaker: Lianmin Zheng, University of California, Berkeley Date: Tuesday, March 5 Time: 12:30pm EST Location: CS 105 Host: Ravi Netravali Event page: https://www.cs.princeton.edu/events/26593 Register for live-stream online here: https://princeton.zoom.us/webinar/register/WN_ghiT2HbhSPqKBouJ-VwzFg Title: Scalable and Efficient Systems for Large Language Models Abstract: Large Language Models (LLMs) have been driving recent breakthroughs in AI. These advancements would not have been possible without the support of scalable and efficient infrastructure systems. In this talk, I will introduce several underlying systems I have designed and built to support the entire model lifecycle, from training to deployment to evaluation. First, I will present Alpa, a system for large-scale model-parallel training that automatically generates execution plans unifying data, operator, and pipeline parallelism. Next, I will discuss efficient deployment systems, covering the frontend programming interface and backend runtime optimizations for high-performance inference. Finally, I will complete the model lifecycle by presenting our model evaluation efforts, including the crowdsourced live benchmark platform, Chatbot Arena, and the automatic evaluation pipeline, LLM-as-a-Judge. These projects have collectively laid a solid foundation for large language model systems, being widely adopted by leading LLM developers and companies. I will conclude by outlining some future directions of machine learning systems, such as co-optimizing across the full stack for building AI-centric applications. Bio: Lianmin Zheng is a Ph.D. student in the EECS department at UC Berkeley, advised by Ion Stoica and Joseph E. Gonzalez. His research interests include machine learning systems, large language models, compilers, and distributed systems. He builds full-stack, scalable, and efficient systems to advance the development of AI. He co-founded LMSYS.org, where he leads impactful open-source large language model projects such as Vicuna and Chatbot Arena, which have received millions of downloads and served millions of users. He also co-organized the Big Model Tutorial at ICML 2022. He has received a Meta Ph.D. Fellowship, an IEEE Micro Best Paper Award, and an a16z open-source AI grant. Speaker: Emma Dauterman, University of California, Berkeley Date: Thursday, March 7 Time: 12:30pm EST Location: CS 105 Host: Amit Levy Event page: https://www.cs.princeton.edu/events/26585 Register for live-stream online here: https://princeton.zoom.us/webinar/register/WN_soNesOKXSFml9MPaTx_1iw Title: Secure systems from insecure components Abstract: In many computer systems today, an attacker that breaks one system component can steal data from millions of users. In this talk, I will present two systems that can withstand component compromise. I will describe (1) a single sign-on system that protects user security and privacy from a compromised single sign-on server, and (2) a secure-hardware-based backup service that protects user backups from compromised secure hardware devices. These systems provide strong security and privacy properties while taking into account practical constraints such as compatibility requirements, hardware limitations, and user expectations. Each splits user secrets across different system components, using new cryptographic tools to provide necessary functionality while protecting user data. Bio: Emma Dauterman is a Ph.D. candidate at UC Berkeley where she is advised by Raluca Ada Popa and Ion Stoica. Her research interests include computer security, systems, and applied cryptography. She has received the Microsoft Research Ada Lovelace fellowship, the NSF graduate research fellowship, and a UC Berkeley EECS excellence award.

1 22

Lahav Lipson will present his FPO "Fast and Robust 3D Reconstruction" on Thursday, November 21, 2024 at 10:00 AM in Friend 202.
by CS Grad Department 18 Nov '24

18 Nov '24

Lahav Lipson will present his FPO " Fast and Robust 3D Reconstruction " on Thursday, November 21, 2024 at 10:00 AM in Friend 202 . Location: Friend 202 The members of Lahav ’s committee are as follows: Examiners: Jia Deng (Adviser), Adam Finkelstein , Szymon Rusinkiewicz Readers: Anirudha Majumdar , Olga Russakovsky A copy of his thesis is available upon request. Please email gradinfo(a)cs.princeton.edu if you would like a copy of the thesis. Everyone is invited to attend his talk. Abstract follows below: 3Dreconstruction from visual data is an important subtask for robotics, autonomous machines, and 3D scene understanding. It involves estimating camera and object motion from images/video, as well as 3D structure. I will introduce an approach known as Optimization Guided Neural Iterations (OGNI), and demonstrate how it can be applied to various 3D reconstruction tasks. In OGNI-based approaches, we mimic classical optimization algorithms by breaking down each task into a series of small revisions predicted by a shallow network. Each revision is supervised independently, and is informed by features conditioned on a running estimate of the solution. This mechanism is surprisingly general, and leads to robust and efficient solutions to 3D reconstruction. Moreover, I introduce several explicit optimization layers which enable one to reformat these challenging problems into easier, low-level vision tasks. On visual SLAM, stereo matching and object pose estimation, I show how this approach leads to state-of-the art accuracy and/or speed. I also discuss potential future directions for this line of work.

1 0

Zexuan Zhong will present his FPO "Language Models with Retrieval Augmentation and Conditional Computation" on Thursday, November 21, 2024 at 4:00 PM in E-Quad E423.
by CS Grad Department 15 Nov '24

15 Nov '24

Zexuan Zhong will present his FPO " Language Models with Retrieval Augmentation and Conditional Computation " on Thursday, November 21, 2024 at 4:00 PM in E-Quad E423 . Location: E-Quad E423 The members of Zexuan ’s committee are as follows: Examiners: Danqi Chen (Adviser), Karthik Narasimhan , Luke Zettlemoyer (University of Washington) Readers: Kai Li , Sanjeev Arora A copy of his thesis is available upon request. Please email gradinfo(a)cs.princeton.edu if you would like a copy of the thesis. Everyone is invited to attend his talk. Abstract follows below: Recent advancements in large language models (LMs) have primarily focused on scaling up model parameters and training tokens to improve performance across various tasks. However, this scaling increases computational costs significantly. Furthermore, conventional parametric LMs are fundamentally incapable of adapting to unseen domains, editing learned knowledge, retaining long-tail knowledge, and are shown to easily leak private data from the training corpus. This thesis explores alternative approaches to scaling LMs while addressing these limitations. Firstly, we study LMs with retrieval augmentation, where LMs leverage an external datastore to make predictions. We develop Trime, a novel end-to-end training approach that learns LMs and the retrieval models jointly. Our results show that Trime significantly enhances LM performance with the same model size and computational budget. These end-to-end trained retrieval-augmented LMs also provide users with effective adaptability to domains unseen during training. Secondly, we focus on a fundamental challenge of LMs: editing facts that are stored in their parameters—a critical problem to address since the world is constantly changing and the knowledge in LMs becomes outdated easily. We examine state-of-the-art knowledge editing methods on LMs and find that existing evaluation paradigms are extremely limited. We propose a novel benchmark MQuAKE, consisting of multi-hop questions that assess whether edited models correctly answer questions where the answer should change as an entailed consequence of edited facts. We demonstrate that existing knowledge editing methods fail on the constructed multi-hop questions. We also propose a simple retrieval-augmented approach that stores all edited facts externally, which outperforms previous methods by a large margin. Thirdly, we investigate training LMs with conditional computation, which is designed for scaling LMs without a substantial increase in computational costs. We focus on mixture-of-experts (MoE), a widely used conditional computation technique which facilitates efficient scaling. However, training the routing network in MoE introduces the challenge of optimizing a non-differentiable, discrete objective. We present a fully differentiable MoE architecture for autoregressive language model pre-training, Lory, which are based on two key techniques: (1) a causal segment routing strategy for high efficiency of expert merging and (2) a similarity-based data 3batching method for better expert specialization. Despite segment-level routing, Lory models achieve competitive performance compared to state-of-the-art MoE models with token-level routing. We further demonstrate that the trained experts of Lory capture domain-level specialization without supervision. Overall, our research illuminates a new paradigm in LM scalability, fundamentally addressing critical limitations and advancing the development of more efficient, effective, adaptable, and updatable language models.

1 0

PPH-NOAA Conference on Data-Driven Precision Health: TOMORROW in the Maeder Hall Auditorium
by Emily C. Lawrence 06 Nov '24

06 Nov '24

[ https://pph.princeton.edu/events/data-driven-precision-health-integrating-g… | Registration ] is still open for the November 7 conference on "Data-Driven Precision Health: Integrating Genomics, Climate, and the Environment.” The conference, a collaboration between Princeton University’s Princeton Precision Health and NOAA’s Geophysical Fluid Dynamics Laboratory and co-sponsored by Princeton’s High Meadows Environmental Institute , will bring together an interdisciplinary community from academia, government, and industry to explore research questions at the intersection of human health, climate, and environment. The event aims to spark discussions and inspire new research directions on future health impacts, with a focus on predicting outcomes and developing equitable risk mitigation strategies. The conference will be held in Maeder Hall, at the Andlinger Center for the Energy and the Environment. Keynote speakers include Jane Burns , Director of the Kawasaki Disease Research Center at UC San Diego and whose work on Kawasaki Disease, environment, and climate has been featured in the [ https://www.nytimes.com/2024/02/27/health/kawasaki-disease-cause.html | NY Times ] , and Juli Trtanj , Director of the NOAA Climate and Health Program and NOAA One Health Lead and co-chair of Global Heat Health Information Network (GHHIN) Management Committee . In addition, Provost Jennifer Rexford and Dean For Research Peter Schiffer will be giving remarks at the conference. Registration ( [ http://pph.princeton.edu/healthandclimate | pph.princeton.edu/healthandclimate ] ) is free but required. Email [ mailto:PrincetonPPH@princeton.edu | PrincetonPPH(a)princeton.edu ] with any questions about the event.

1 0

NextG Distinguished Seminar Series: Alyosha Molnar, Nov. 13, 12:30pm, B205 EQuad
by Emily C. Lawrence 06 Nov '24

06 Nov '24

NextG Distinguished Seminar Series Speaker: Alyosha Molnar, Cornell University Date: Nov 13, 2024 Time: 12:30 pm – 1:30 pm Location: B205 Engineering Quadrangle Title: Solving the “Cocktail Party Problem” for Radios: Circuit and System Techniques for Mitigating Interference Abstract: Even as wireless systems become increasingly important to the functioning of society, they are a victim of their own popularity, with the RF spectrum increasingly load and crowded, creating a sort of “cocktail party problem” for wireless receivers. This can partly be resolved by expanding into new, higher frequency bands that are not yet crowded, but also requires making better use of the frequency bands we already use. In particular, there will be a need for RF and mmWave systems that are flexible enough to make use of whatever spectrum is available, but also sufficiently resilient to strong interference in other bands so as not to be disrupted by them. I will briefly introduce the concept of N-path mixers and filters in modern CMOS, and how their capabilities map nicely onto the emerging needs of flexible, interference tolerant receivers. I will then discuss our more recent work developing techniques to map N-path designs to higher, mmWave frequencies, while maintaining the mixers’ dynamic range without burning excessive power. This will then lead to a discussion of a new style of flexible receiver, leveraging circuit and algorithm co-design to identify and remove interference artifacts from one’s received signal. I will also briefly discuss some of our recent work on tiny, wireless neural implants, (not because its related, just because it is really neat). Bio: Alyosha Molnar received his B.S. in engineering from Swarthmore College in 1997, and M.S. (2003) and Ph.D. (2007) in electrical engineering from the University of California, Berkeley. From 1998 to 2002, he was a design engineer with the RFIC Group at Conexant Systems, Inc., Newport Beach, CA, where he co-led the development of the first cellular direct conversion receiver to be successfully sold on the open market, as well as the first fully integrated quad-band GSM transceiver. In graduate school he worked on one of the first sub-milliwatt radios for “smart dust”, before spending several years in a neuroscience lab studying the biological circuits that underlie early image processing in the mammalian retina. He has been a faculty member with the School of Electrical and Computer Engineering at Cornell University since 2007, doing research spanning RF and mmWave integrated circuits for flexible wireless systems, novel image sensors and associated image processing, neuroscience and neural interface systems and circuits, and microscale autonomous systems. Molnar is the Ilda and Charles Lee Professor of Engineering at Cornell, and is the recipient of the NSF CAREER Award, the DARPA YFA, ISSCC’s Lewis Winner award, and The Darlington Best Paper Award, along with many other research and teaching awards.

1 0

ECE Seminar: Marilyn Wolf TODAY at 4:30pm
by Emily C. Lawrence 04 Nov '24

04 Nov '24

ECE Seminar Speaker: Marilyn Wolf, University of Nebraska–Lincoln Date: Monday, Nov 4 Time: 4:30pm Location: D221, EQuad Host: Niraj Jha [ https://ece.princeton.edu/events/perceptioncontrol-co-design-autonomous-veh… | https://ece.princeton.edu/events/perceptioncontrol-co-design-autonomous-veh… ] Title: Perception/Control Co-Design for Autonomous Vehicles Abstract: Perception is a critical computational task in autonomous vehicles. Autonomous vehicles place stringent and somewhat conflicting demands on perception systems: high accuracy, low latency, and performance on limited computational resources. The conflict between these requirements is particularly acute in the case of unmanned aerial vehicles (UAVs). This talk will describe two related efforts on efficiency and accuracy in perception for autonomous vehicles. Work with Krishna Muuva and Justin Bradley of UNL looks at UAV-UAV tracking. We show that tracking performance saturates above a given level of perceptual accuracy. Work with Deep Samal, Thomas Walton, and Dung Tran of UNL builds a Markovian model for perception errors and shows their effect on tracking. Biography: Marilyn Wolf is Elmer E. Koch Professor of Engineering at the University of Nebraska – Lincoln where she served as Founding Director of the School of Computing. She received her BS, MS, and PhD in electrical engineering from Stanford University in 1980, 1981, and 1984, respectively. She was with AT&T Bell Laboratories from 1984 to 1989. She was on the faculty of Princeton University from 1989 to 2007 and was Farmer Distinguished Chair at Georgia Tech from 2007 to 2019. Her research interests include cyber-physical systems, embedded computing, embedded video and computer vision, and VLSI systems. She has received the IEEE Kirchmayer Graduate Education Award, the IEEE Computer Society Goode Memorial Award, the ASEE Terman Award, the IEEE Circuits and Systems Society Education Award, and the ACM SIGDA Distinguished Service Award. She is a Fellow of the IEEE and ACM and an IEEE Computer Society Golden Core member.

1 0