Carlos Jimenez will present his General Exam "Consistency And Robustness Testing for Visual Question Answering" on Friday, May 6, 2022 at 2:30 PM in CS 401 and via zoom. Zoom link: [ https://princeton.zoom.us/j/96002443578 | https://princeton.zoom.us/j/96002443578 ] Committee Members: Karthik Narasimhan (advisor), Danqi Chen, Olga Russakovsky Abstract: We introduce CARETS as a new systematic test suite to measure consistency and robustness of modern visual question answering (VQA) models through a series of fine-grained capability tests. In contrast to existing VQA test sets, CARETS features balanced question generation to create pairs of instances to test models, with each pair focusing on a specific capability such as rephrasing, logical symmetry or image obfuscation. We evaluate six modern VQA systems on CARETS and identify several actionable weaknesses in model comprehension, especially with concepts such as negation, disjunction, or hypernym invariance. We show that even the most sophisticated models are sensitive to aspects such as swapping the order of terms in a conjunction or varying the number of answer choices mentioned in a question; underscoring the need for improved robustness and reliability of state-of-the-art machine learning models. Reading List: [ https://docs.google.com/document/d/1rnOlJVkDNgI7CJsjVN78sEPN-OIa3M9i661f_oly... | https://docs.google.com/document/d/1rnOlJVkDNgI7CJsjVN78sEPN-OIa3M9i661f_oly... ] Everyone is invited to attend the talk, and those faculty wishing to remain for the oral exam following are welcome to do so.