talks

Download

talks@lists.cs.princeton.edu

July 2026

1 participants
2 discussions

Sayash Kapoor will present his FPO "The Missing Science of AI Evaluation" on Jul 21, 2026 at 12pm in 306 Sherrerd Hall
by Gradinfo 15 Jul '26

15 Jul '26

Sayash Kapoor will present his FPO "The Missing Science of AI Evaluation" Date: Jul 21, 2026, 12:00 pm – 2:00 pm Location: 306 Sherrerd Hall [ https://citp.princeton.edu/events/2026/sayash-kapoor-final-public-oral-fpo-… | https://citp.princeton.edu/events/2026/sayash-kapoor-final-public-oral-fpo-… ] Examiners: Arvind Narayanan (Adviser), Peter Henderson, and Brandon Stewart Readers: Arvind Narayanan, Karthik Narasimhan, and Jessica Hullman (Northwestern Univ) AI evaluations inform critical decisions, from the valuations of trillion-dollar companies to policies on regulating AI. Yet, evaluation methods have failed to keep pace with deployment, creating an evaluation crisis where performance in the lab fails to predict real-world utility. In this talk, Kapoor discusses the evaluations crisis in a high-stakes domain: AI-based science. Across dozens of fields, from medicine to political science, he finds that flawed evaluation practices have led to overoptimistic claims about AI’s accuracy, affecting hundreds of published papers. To address these evaluation failures, Kapoor presents a consensus-based checklist that identifies common pitfalls and consolidates best practices for researchers adopting AI, and a benchmark to foster the development of AI agents that can verify scientific reproducibility. AI evaluation failures affect several other applications. Beyond science, he examines how AI agent benchmarks miss many failure modes, and present HAL, a system to address evaluation shortcomings and conduct large-scale agent evaluations. The talk examines inference scaling, a recent technique to boost AI capabilities, and show that many claims of improvement fail to hold under realistic conditions. Finally, Kapoor discusses how better AI evaluation can inform policymaking, drawing on work on open foundation models and engagement with state and federal agencies. Why does the evaluation crisis persist? The AI community has poured enormous resources into building evaluations for models, but not into investigating how models impact the world. To address the crisis we need to build a systematic science of AI evaluation to bridge the gap between benchmark performance and real-world impact.

1 0

Nick Koh will present his FPO "Consequence-Finding in Theories of Arithmeticon" on Wednesday, 7/15/26 at 3pm in CS 402.
by Gradinfo 10 Jul '26

10 Jul '26

Nick Koh will present his FPO "Consequence-Finding in Theories of Arithmeticon" on Wednesday, 7/15/26 at 3pm in CS 402. The members of his committee are as follows: Examiners: Zak Kincaid (adviser), Pravesh Kothari, David Walker Readers: Aarti Gupta and Zak Kincaid Abstract: Given a formula F in a first-order theory, what is the strongest formula of a certain form that is entailed by F? In program analysis and verification, programs are often analyzed through the lens of logic, e.g., by translating fragments of code into logical formulas, and problems like loop summarization and ranking function synthesis can be reduced to instances of consequence-finding. This thesis considers consequence-finding modulo the theory of linear integer-real arithmetic (LIRA) and the theory of linear integer-real rings (LIRR). To aid consequence-finding for these theories, we introduce an algorithmic framework of concept spaces and local abstractions. Concept spaces allow us to frame consequence-finding using representations beyond formulas, while local abstractions formalize a strategy to find consequences by generalizing sampled models of a formula. For LIRA, we consider the problem of finding the strongest conjunction of linear inequalities entailed by an existential formula. This corresponds to the problem of computing the closed convex hull of its set of models. We decompose this into a series of simpler abstraction problems, and compute the convex hull using a combination of local abstractions. For LIRR, we consider the problem of finding the set of all polynomial inequalities and integrality constraints entailed by an existential formula in polynomial integer-real arithmetic. In the standard model of integers, polynomial arithmetic is undecidable, and LIRR is a new theory of polynomial integer-real arithmetic that gives up completeness in exchange for not only decidability, but also complete consequence-finding. We introduce algebraic cones and lattices to represent sets of consequences, and solve the consequence-finding problem using a local abstraction. More broadly, these instances show that concept spaces and local abstractions are useful for consequence-finding in general.

1 0