- talks - lists.cs.princeton.edu

Rui Pan will present his General Exam "Practical Systems for Mitigating Redundant Computation in ML Inference" on May 2, 2024 at 8:45am in CS 302.
by Nicki Mahler 25 Apr '24

25 Apr '24

Rui Pan will present his General Exam "Practical Systems for Mitigating Redundant Computation in ML Inference" on May 2, 2024 at 8:45am in CS 302. Zoom link: [ https://princeton.zoom.us/my/ruipan | https://princeton.zoom.us/my/ruipan ] Committee members: Ravi Netravali (adviser), Kai Li, and Danqi Chen Link to Abstract & Reading list: [ https://docs.google.com/document/d/1Z99Wf5M2MAN0JEQrm3VggIB-N4nXLMu7h_VlDyc… | https://docs.google.com/document/d/1Z99Wf5M2MAN0JEQrm3VggIB-N4nXLMu7h_VlDyc… ]

1 0

Akshara Prabhakar will present her MSE talk "Exploring Reasoning and Interactive Benchmarking of Language Models" on Friday, April 26, 2024 at 3:30pm via Zoom
by Gradinfo 25 Apr '24

25 Apr '24

Akshara Prabhakar will present her MSE talk "Exploring Reasoning and Interactive Benchmarking of Language Models" on Friday, April 26, 2024 at 3:30pm via Zoom zoom: [ https://princeton.zoom.us/j/8646560610 | https://princeton.zoom.us/j/8646560610 ] Adviser: Karthik Narasimhan Reader: Thomas L. Griffiths Title: Exploring Reasoning and Interactive Benchmarking of Language Models Abstract: This thesis aims to deepen our understanding of the reasoning process undertaken in Large Language Models (LLMs) and introduces a challenging benchmark to evaluate language agents' interactive code-generation capabilities. Part 1 delves into untangling the factors affecting the Chain-of-Thought (CoT) reasoning process in LLMs. Despite CoT prompting having demonstrated significant efficacy in enhancing multi-step reasoning capabilities, it has been found to yield biased answers and produce unfaithful explanations, raising interpretability concerns. Additionally, debates persist regarding whether LLMs truly generalize or rely on heuristics. Focusing on the symbolic reasoning task of decoding shift ciphers, we develop a simple probabilistic approach to identify these factors - the probability of the task's expected output (probabilistic effect), what the model has implicitly learned during pre-training (memorization influenced), and the model's attempt to adopt shorter reasoning steps (noisy reasoning) and show the consequent drastic variability in task accuracy. Through a series of experiments, we conclude that LLM behavior exhibits clear hallmarks of both memorization and true reasoning suggesting that CoT resembles a probabilistic, memorization-influenced form of noisy reasoning. Part 2 transitions from LLMs to the emerging domain of language agents, where the LLM is now grounded in a digital environment to aid in decision-making. We introduce InterCode -- an interactive RL environment to benchmark the interactive code generation abilities of language agents with execution-driven feedback. By converting traditional seq2seq datasets into three interactive code environments (Bash, SQL, and Python), we demonstrate InterCode's viability as a testbed by evaluating multiple state-of-the-art LLMs configured with different prompting strategies such as ReAct and Plan & Solve. Our results underscore the benefits of interactive code generation and establish InterCode as a robust, scalable benchmark for advancing code understanding and generation capabilities. Furthermore, we use this framework in the cybersecurity realm; by developing a Capture the Flag task in the InterCode environment, we find that while language agents possess rudimentary cybersecurity knowledge, they are not able to perform multi-step cybersecurity tasks out of the box. Overall, it highlights the evolving landscape of LLMs, emphasizing the need for comprehensive evaluation and understanding of their capabilities across different domains.

1 0

Anirudh Ajith will present his MSE talk "Towards Optimizing In-Context Learning and Ensuring Data Transparency in Large Language Models" on Friday, April 26, 2024 at 1pm in CS 402
by Gradinfo 25 Apr '24

25 Apr '24

Anirudh Ajith will present his MSE talk " Towards Optimizing In-Context Learning and Ensuring Data Transparency in Large Language Models" on Friday, April 26, 2024 at 1pm in CS 402 Adviser: Karthik Narasimhan Reader: Danqi Chen All are welcome to attend. Abstract: Large language models have emerged as a significant development in the fields of natural language processing and artificial intelligence. These models, primarily trained on large corpora of text scraped from webpages and books show emergent abilities enabling strong (and even human-level) performance on a wide range of tasks. These impressive abilities are typically elicited using "in-context learning" prompts that include instructions, annotated demonstrations and unsolved test examples. Recent work has shown that LLM performance remains sensitive to the precise details of the demonstrations, instructions and even formatting used in these prompts. While the optimal selection of demonstrations has been explored, instruction-choice remains understudied with existing analyses confined to narrow subsets of models and tasks, thus reducing the generalizability of their insights. The first part of this thesis introduces the InstructEval suite for the systematic evaluation of instruction selection methods for ICL. We use InstructEval, which includes 13 open-sourced LLMs of varying scales from four model families and covers nine tasks across three categories to evaluate seven popular instruction selection methods over five metrics relevant to ICL. Our experiments reveal that curated manually-written instructions or simple instructions without any task-specific descriptions often elicit superior ICL performance overall than automatic instruction-induction methods, hence pointing to a lack of generalizability among the latter. As the scale of data used to train contemporary LLMs has grown into the trillions of tokens, model developers have also become increasingly reluctant to disclose their data sources. Simultaneously, there have been growing concerns that academic benchmarks used to evaluate LLMs may have been compromised by the leakage of test data into pretraining corpora. Concerns also exist that these LLMs may be trained on sensitive personal information or copyrighted content which could be problematic to generate post-deployment. In the second part of this thesis, we study the pretraining data detection problem: given a piece of text and black-box access to an LLM, can we determine if the model was trained on the provided text? We introduce the WikiMIA benchmark to facilitate this study and propose a new detection method called Min-K% Prob that unlike prior work, does not require reference models, additional training or any knowledge about a model’s pretraining data distribution. In addition to showing that Min-K% Prob outperforms baselines on WikiMIA, we also demonstrate its utility for detecting the leakage of benchmark data and copyrighted content.

1 0

Beining Han will present his General Exam "Learning Open-world Robot Object Rearrangement Skills from Synthetic Data" on Monday, April 29, 2024 at 3:00 PM in CS 402.
by CS Grad Department 24 Apr '24

24 Apr '24

Beining Han will present his General Exam "Learning Open-world Robot Object Rearrangement Skills from Synthetic Data" on Monday, April 29, 2024 at 3:00 PM in CS 402. Committee Members: Jia Deng (advisor), Szymon Rusinkiewicz, Felix Heide Abstract: Abstract: Building robot object rearrangement systems is an important research problem in robotics and embodied AI, with wide applications in industry and service robots. However, it is still very challenging to build such a robot in our open-world that can rearrange any object to anywhere, owing to the complexity of the real-world. Moreover, it is even hard to collect enough real-world data to learn from. In our work, we approach this problem by creating synthetic data with procedural rules and teach the robot rearrangement skills with the data we created. Within this realm, we propose two sub-problems: unknown object relative 6D pose estimation and robot 6D grasping of unknown object in novel environments. We find that these two problems are quite challenging with current methods in vision and robotics, suggesting huge space for improvement. Reading List: https://docs.google.com/document/d/1k_upDVyRe747b6X36aPv6PVUb-Xz4tH6b-cxqd-… Everyone is invited to attend the talk, and those faculty wishing to remain for the oral exam following are welcome to do so.

1 0

Pranay Manocha will present his FPO "Do we need a Reference Signal for Speech Quality Assessment?" on Wednesday, May 1, 2024 at 1:30 PM in CS 401.
by CS Grad Department 24 Apr '24

24 Apr '24

Pranay Manocha will present his FPO "Do we need a Reference Signal for Speech Quality Assessment?" on Wednesday, May 1, 2024 at 1:30 PM in CS 401. Location: CS 401 The members of Pranay’s committee are as follows: Examiners: Adam Finkelstein (Adviser), Szymon Rusinkiewicz, Paul Calamia (Meta Reality Labs Research) Readers: Karthik Narasimhan, Zeyu Jin (Adobe Research) A copy of his thesis is available upon request. Please email gradinfo(a)cs.princeton.edu if you would like a copy of the thesis. Everyone is invited to attend his talk. Abstract follows below: This thesis investigates new metrics for assessing speech quality that aim to align more closely with human auditory perception than current methods. It aims to improve the techniques and understanding of speech quality evaluation. It considers traditional methods that compare speech to a perfect (clean) reference and introduces new approaches for scenarios where such a reference is not available. It also emphasizes the significance of reference signals and explores the necessity for flexible evaluation techniques that can function effectively without an ideal reference. The dissertation describes three main categories of metrics: full-reference (FR), no-reference (NR), and non-matching reference (NMR), providing a detailed comparison of their benefits and limitations. Despite the general preference for FR metrics in situations where a corresponding clean reference signal is available, this research identifies specific circumstances where FR metrics may not be the most effective approach, thereby highlighting the utility and relevance of NMR metrics across different evaluative scenarios. Another contribution of this thesis is the introduction of CoRN, a novel metric formulated through the integration of FR, and NR metrics. This metric builds on an exhaustive analysis of various evaluation metrics, demonstrating its utility in advancing audio quality assessment. Additionally, applying these methods to spatial audio in augmented and virtual reality settings expands the thesis’s contribution to the more general domain of audio quality assessment. This thesis aims to improve the techniques and understanding of speech quality evaluation. This dissertation aims to refine and expand the methodologies and understanding of speech quality evaluation, a crucial step for the evolution of digital communication technologies.

1 0

Yinwei Dai will present his General Exam "From Static to Dynamic: Systems for Dynamic Neural Networks" on May 1, 2024 at 2pm in CS 302 and Zoom.
by Nicki Mahler 24 Apr '24

24 Apr '24

Yinwei Dai will present his General Exam "From Static to Dynamic: Systems for Dynamic Neural Networks" on May 1, 2024 at 2pm in CS 302 and Zoom. Zoom link: [ https://princeton.zoom.us/my/yinweid | https://princeton.zoom.us/my/yinweid ] Committee members: Ravi Netravali(adviser), Wyatt Lloyd, Kai Li Abstract and Reading List: [ https://docs.google.com/document/d/1SOKjywl3Iq9EYf96T2EGmDg8Qc1sTkrrW8NnDe5… | https://docs.google.com/document/d/1SOKjywl3Iq9EYf96T2EGmDg8Qc1sTkrrW8NnDe5… ]

1 0

Ashwini Raina will present his FPO "Rethinking System Design with Awareness for Cross-layer Aspects of Datacenter Storage" on Friday, May 3, 2024 at 10:30am in CS 302.
by Gradinfo 24 Apr '24

24 Apr '24

Ashwini Raina will present his FPO "Rethinking System Design with Awareness for Cross-layer Aspects of Datacenter Storage" on Friday, May 3, 2024 at 10:30am in CS 302. The members of his committee are as follows: Examiners: Michael Freedman (adviser), Wyatt Lloyd, Ravi Netravali Readers: Amit Levy, Asaf Cidon (Columbia University) Storage is a critical piece of infrastructure in modern web applications. In recent years, storage technologies employed in building such systems have undergone significant evolution, bringing about novel cost-performance trade-offs. Concurrently, datacenter storage architectures have become increasingly layered. Software systems designed based on outdated assumptions of datacenter storage often result in poor cost-performance trade-offs or suffer from suboptimal performance. This dissertation proposes a new design approach for systems, one that incorporates the awareness of cross-layer aspects of datacenter storage, and validates the effectiveness of this approach through two systems. The first system is PrismDB, a novel key-value store that exploits two extreme ends of the spectrum of modern NVMe storage technologies (3D XPoint and QLC NAND) simultaneously. In recent years, emerging storage technologies have focused on divergent goals: better performance or lower cost. Correspondingly, data systems that employ these technologies are typically optimized either to be fast (but expensive) or cheap (but slow). PrismDB take a different approach: by architecting a storage engine to natively utilize two tiers of fast and low-cost storage technologies, it shows that a Pareto-efficient balance between performance and cost can be achieved. The second system is Fusion, an object store for analytics that is optimized for query pushdown on erasure-coded data. Computation pushdown is a widely adopted technique to reduce latency of highly selective queries in modern OLAP cloud database running on disaggregated storage. However, existing pushdown solutions are inefficient on erasure-coded storage since the analytics file objects get partitioned across storage nodes. Consequently, the storage system must reassemble the object across nodes before executing the query, leading to significant network latency. Fusion addresses this problem by co-designing its erasure coding and file placement topologies, taking into account popular analytics file formats (e.g., Parquet). It employs a novel stripe construction algorithm that prevents the fragmentation of computable units within an object, and minimizes storage overhead during erasure coding. Overall, this dissertation advocates for designing software systems with an awareness of cross-layer aspects in datacenter storage, and demonstrates the benefits of that approach via two systems: PrismDB and Fusion.

1 0

Wei Luo will present her MSE Thesis talk "SketchProbe: Discovering Vulnerabilities in Sketch-based Applications with Reinforcement Learning" on Wednesday, April 24, 2024 at 2pm in CS 402.
by Gradinfo 23 Apr '24

23 Apr '24

Wei Luo will present her MSE Thesis talk "SketchProbe: Discovering Vulnerabilities in Sketch-based Applications with Reinforcement Learning" on Wednesday, April 24, 2024 at 2pm in CS 402. Advisor: Maria Apostolaki, Reader: Benjamin Eysenbach All are invited to attend. Please see abstract below. Abstract: Sketches are approximate data structures critical to network traffic monitoring. They are specifically designed for environments constrained by memory, enabling the maintenance of accurate statistics within a compact space. This efficiency comes at the cost of estimation error that varies based on the workload encountered by the sketch. Traditionally, sketch-based applications are optimized for average traffic scenarios, and this assumption of standard patterns of network traffic introduces vulnerabilities, as these applications may not be prepared for atypical or adversarial traffic patterns. Manual testing for these vulnerabilities is time-intensive and often impractical due to the vast range of possible traffic scenarios. In this paper, we introduce SketchProbe, a novel system leveraging Reinforcement Learning (RL) to identify adversarial traffic patterns in sketch-based applications. Unlike manual identification, SketchProbe automates the discovery process, ensuring a thorough and efficient exploration of potential failure modes. Our evaluation indicates that SketchProbe effectively identifies critical scenarios that significantly impact the performance of sketch-based applications, which is valuable for improving the robustness of such applications.

2 1

ML in Physics: Joshua Shaevitz, tomorrow at 4:30pm in Jadwin Hall A10
by Emily C. Lawrence 23 Apr '24

23 Apr '24

Dear Colleagues, We are thrilled to invite you to the next ML in Physics Seminar talk. Here are the details: Speaker : Joshua Shaevitz (Professor of Physics and Biophysics at Princeton University) When/Where : Wednesday, April 24th at 4:30pm in Jadwin Hall A10 Title : Supervised and unsupervised algorithms for measuring animal behavior Abstract : For the past decade, my group has worked on supervised and unsupervised algorithms that take video data of behaving animals as inputs and discover stereotyped dynamics in the motion of body parts. We have used these techniques to study the patterns of behavior over an animal’s lifespan, how the brain specifies which behavior to actuate, social behaviors in pairs of animals, and the effect of neurological disorders on behavior and social interactions. I will briefly discuss the types of algorithms we use and then go through a variety of results from different experimental systems including worms, fruit flies, bumblebees, mice, and fish. For more details, including information about how to sign up for our listserv, please visit [ https://ml.physics.princeton.edu/ | https://ml.physics.princeton.edu/ ] . Sincerely, ML in Physics Organizers (Gage DeZoort, Peter Elmer, Boris Hanin, Mariangela Lisanti, and Dan Marlow)

1 0

Prachi Sinha will present her MSE talk "Augmenting Concept-Labeled Datasets for Interpretability Using Counterfactual Generation" TODAY Monday, April 22 at 12:30 PM in CS 402.
by gradinfo＠cs.princeton.edu 23 Apr '24

23 Apr '24

Prachi Sinha will present her MSE talk "Augmenting Concept-Labeled Datasets for Interpretability Using Counterfactual Generation" Monday, April 22 at 12:30 PM in CS 402. Advisor: Vikram Ramaswamy, Reader: Olga Russakovsky Abstract: Concept-based interpretability methods use a pre-defined set of human-understandable concepts to explain a model's output. These explanations are learned by "probing" the model using a concept-labeled dataset and learning how concepts can be linearly combined to predict outputs. However, there are several limitations to current methods. Explanations are highly dependent on the probe dataset, and they may reflect correlations between concepts and outputs, rather than causation. To address these issues, we augment COCO, a concept-labeled dataset, with counterfactual images in which particular semantic concepts have been removed. Using such counterfactual images, we attempt to improve the accuracy of explanations for a scene classification model, and more directly learn the causal relationships between concepts and outputs. We find that explanation accuracy improves in certain cases, but is generally still limited by the probe dataset's alignment with the model, and that counterfactual images elucidate issues with assigning weights to concepts when there are strong correlations between concepts. CS Grad Calendar: https://calendar.google.com/calendar/event?action=TEMPLATE <https://calendar.google.com/calendar/event?action=TEMPLATE&tmeid=NzljZTNoZm 1ibWc1NGU4ZmtodjJkdHRuMmggYWNnMDc5YmxzbzRtczNza2tmZThwa2lyb2dAZw&tmsrc=acg07 9blso4ms3skkfe8pkirog%40group.calendar.google.com> &tmeid=NzljZTNoZm1ibWc1NGU4ZmtodjJkdHRuMmggYWNnMDc5YmxzbzRtczNza2tmZThwa2lyb 2dAZw&tmsrc=acg079blso4ms3skkfe8pkirog%40group.calendar.google.com

1 4