Sabhya Chhabria will present his MSE talk "Exploring LLM Insight Discovery: InsightBench and MCMC Prompting" Wednesday April 24, at10:00 AM in CS 401 Advisor: Danqi Chen, Reader: Karthik Narasimhan Abstract: Large Language Models (LLMs) have become increasingly adept at existing benchmarks that seek to evaluate their abilities on traditional tasks such as language understanding, reasoning, programming and math questions. In this work, we introduce InsightBench, a new benchmark that seeks to evaluate LLMs on insight problems. Insight problems are those problems where the path to a solution is not immediately obvious. They require some form of creative thinking and the answer usually strikes in an "aha" moment. In this work, we evaluate several popular LLMs (both open and closed sourced) of varying sizes on InsightBench. Our results are able to establish that insight problems (creative problem solving) remain a critical failure point of language models. We further try to bolster LLM performance on certain InsightBench tasks by using Markov Chain Monte Carlo prompting schemes, allowing LLMs to explore large search spaces to solve insight problems. CS Grad Calendar: https://calendar.google.com/calendar/event?action=TEMPLATE&tmeid=MWtlMnFucWZ rNWs4dTZmYWg2c25qZTNxZ24gYWNnMDc5YmxzbzRtczNza2tmZThwa2lyb2dAZw&tmsrc=acg079 blso4ms3skkfe8pkirog%40group.calendar.google.com