Xingyu Zhu will present his General Exam "On the Power of Context-Enhanced Learning in LLMs" on Monday, May 5, 2025 at 1:00 PM in FC 106 and via zoom. Zoom link: https://princeton.zoom.us/my/jupiter Committee Members: Sanjeev Arora (advisor), Danqi Chen, Chi Jin Abstract: We formalize a new concept for LLMs, context-enhanced learning. It involves standard gradient-based learning on text except that the context is enhanced with additional data on which no auto-regressive gradients are computed. This setting is a gradient-based analog of usual in-context learning (ICL) and appears in some recent works. Using a multi-step reasoning task, we prove in a simplified setting that context-enhanced learning can be exponentially more sample-efficient than standard learning when the model is capable of ICL. At a mechanistic level, we find that the benefit of context-enhancement arises from a more accurate gradient learning signal. We also experimentally demonstrate that it appears hard to detect or recover learning materials that were used in the context during training. This may have implications for data security as well as copyright. Reading List: https://docs.google.com/document/d/1_SYs1g8A9cSF__u9dRffQ30ZYAArjijfiyW-IwVw... Everyone is invited to attend the talk, and those faculty wishing to remain for the oral exam following are welcome to do so.