- talks - lists.cs.princeton.edu

Zoe Ashwood will present her FPO "Probabilistic Models for Characterizing Animal Learning and Decision-Making" on Monday, July 25, 2022 at 9:00 AM in PNI 101 and Zoom.
by Louis W. Riehl 18 Jul '22

18 Jul '22

Zoe Ashwood will present her FPO "Probabilistic Models for Characterizing Animal Learning and Decision-Making" on Monday, July 25, 2022 at 9:00 AM in PNI 101 and Zoom. Location: PNI 101 Zoom link: [ https://princeton.zoom.us/j/93691725152?pwd=RlpLWkRObG9lMWs5cTAxZTRRSFVkZz09 | https://princeton.zoom.us/j/93691725152?pwd=RlpLWkRObG9lMWs5cTAxZTRRSFVkZz09 ] The members of Zoe’s committee are as follows: Examiners: Jonathan Pillow (Adviser), Ryan Adams, Tom Griffiths Readers: Barbara Engelhardt, Alexandre Pouget (University of Geneva) A copy of her thesis is available upon request. Please email gradinfo(a)cs.princeton.edu if you would like a copy of the thesis. Everyone is invited to attend her talk. Abstract follows below: A central goal in neuroscience is to identify the strategies used by animals and humans as they make decisions, and to characterize the learning rules that shape these policies in the first place. In this thesis, I discuss three projects aimed at tackling this goal. In the first, we introduce a novel framework, the GLM-HMM, for characterizing mouse and human choice policies during binary decision-making tasks. The GLM-HMM is a hidden Markov model with Bernoulli Generalized Linear Model observations. By fitting the GLM-HMM to hundreds of thousands of decisions, our framework revealed that — contrary to common wisdom — mice and humans use multiple strategies over the course of a single session to perform perceptual decision-making tasks. In the second project, we sought to uncover the learning rules used by mice and rats as they learned to perform this type of task. Our model tracked trial-to-trial changes in the animals’ choice policies, and separated these changes into components explainable by a reinforcement learning rule, and components that remained unexplained. Whereas the average contribution of the conventional REINFORCE learning rule to the policy update for mice learning a common task was just 30%, we found that adding baseline parameters allowed the learning rule to explain 92% of the animals’ policy updates under our model. Finally, I discuss our approach to applying inverse reinforcement learning (IRL) to the trajectories of mice exploring a maze environment. While IRL has been widely applied in robotics and healthcare settings to infer the unknown reward function of an agent, it has yet to be applied extensively in neuroscience. One potential reason for this is that existing IRL frameworks assume that an agent’s reward function is fixed over time. To overcome this limitation, we introduce ‘DIRL’, an IRL framework that allows for time-varying intrinsic rewards. Our method returns interpretable reward functions for two separate cohorts of mice, and provides a novel characterization of exploratory behavior. Overall, we anticipate ‘DIRL’ having broad applicability in neuroscience, and that it could also facilitate the design of biologically-inspired reward functions for training artificial agents to perform analogous tasks. Louis Riehl Graduate Administrator Computer Science Department, CS213 Princeton University (609) 258-8014

1 0

Claudia Roberts will present her Pre FPO "Human-machine Cooperation in Machine-Learning Resistant Real-World Applications" on Thursday, July 14 2:00pm ET via Zoom.
by Nicki Mahler 12 Jul '22

12 Jul '22

Claudia Roberts will present her Pre FPO "Human-machine Cooperation in Machine-Learning Resistant Real-World Applications" on Thursday, July 14 2:00pm ET via Zoom. Zoom link: [ https://princeton.zoom.us/j/2202514985 | https://princeton.zoom.us/j/2202514985 ] Advisor: Arvind Narayanan (examiner) Committee members: Adji Bousso Dieng (reader), Andrés Monroy-Hernández (examiner), Barbara Engelhardt (reader), Matt Salganik (examiner) Title: Human-machine Cooperation in Machine-Learning Resistant Real-World Applications All are welcome to attend. Abstract: Automation tools like machine learning are a necessity in our big data world. However, not all applications lend themselves to machine-learning based automation. Numerous societal problems emerge when we attempt to measure and quantify something that resists measurement and subsequently, develop mathematical intermediaries to attempt to model human behavior. However, using machine-learning backed automation to reduce the scale of many real-world applications is still helpful. In this dissertation, we explore three different ways in which we can use human-machine cooperation in machine-learning resistant applications to keep or increase utility and reduce harm when using these systems in the wild. The first chapter, based on “Friend Request Pending: A Comparative Assessment of Engineering and Social Science Inspired Approaches to Analyzing Complex Birth Cohort Survey Data,” shows how the first mode of human-machine cooperation—explaining things to computers by adding domain knowledge—increases utility of machine learning models in the machine-learning resistant task of grade point average (GPA) prediction. The Fragile Families Challenge is a mass collaboration social science data challenge whose aim is to learn how various early childhood variables predict the long-term outcomes of children. We describe our two-step approach to the Fragile Families Challenge. In step 1, we use a variety of fully automated approaches to predict child academic achievement. We fit 124 models, which involve most possible combinations of 8 model types, 2 imputation strategies, 2 standardization approaches, and 2 automatic variable selection techniques using 2 different thresholds. Then, in step 2, we attempt to improve on the results from step 1 with manual variable selection based on a detailed review of the codebooks. We manually selected 3,694 variables believed to be predictive of academic achievement, using a comprehensive review of student success literature to guide the decision-making process. The best models from step 1 were re-estimated using the manually selected variables. We show that manual variable selection improved the majority of the top 10 models in step 1, but did not improve the best of the top 10. Results indicate that variable selection inspired by social science methodologies can, in most cases, significantly improve models trained completely automatically. The second chapter, based on “Selectively Contextual Bandits,” shows how the second mode of human-machine cooperation—humans and computers working together in decision making—maintains utility while reducing harm of machine learning models in the machine-learning resistant task of image personalization. Personalization is an integral part of most web-service applications and determines which experience to display to each member. A popular algorithmic framework used in industrial personalization systems are contextual bandits, which seek to learn a personalized treatment assignment policy in the presence of treatment effects that vary with the observed contextual features of the members. In order to keep the optimization task tractable, such systems can myopically make independent personalization decisions that can conspire to create a suboptimal experience in the aggregate of the member’s interaction with the web-service. We design a new family of online learning algorithms that benefit from personalization while optimizing the aggregate impact of the many independent decisions. Our approach selectively interpolates between any contextual bandit algorithm and any context-free multi-armed bandit algorithm and leverages the contextual information for a treatment decision only if this information promises significant gains over a decision that does not take it into account. Apart from helping users of personalization systems feel less targeted, simplifying the treatment assignment policy by making it selectively reliant on the context can help improve the rate of learning. We evaluate our approach on several datasets including a video subscription web-service and show the benefits of such a hybrid policy. Finally, the last chapter, based on “On the Bias-Variance Characteristics of LIME and SHAP in High Sparsity Movie Recommendation Explanation Tasks,” shows how the third mode of human-machine cooperation—computers explaining to humans what it’s thinking—can improve utility and reduce harm of machine learning models in the machine-learning resistant task of movie recommendations. We evaluate two popular local explainability techniques, LIME and SHAP, on a movie recommendation task. We discover that the two methods behave very differently depending on the sparsity of the data set. LIME does better than SHAP in dense segments of the data set and SHAP does better in sparse segments. We trace this difference to the differing bias-variance characteristics of the underlying estimators of LIME and SHAP. We find that SHAP exhibits lower variance in sparse segments of the data compared to LIME. We attribute this lower variance to the completeness constraint property inherent in SHAP and missing in LIME. This constraint acts as a regularizer and therefore increases the bias of the SHAP estimator but decreases its variance, leading to a favorable bias-variance trade-off especially in high sparsity data settings. With this insight, we introduce the same constraint into LIME and formulate a novel local explainabilty framework called Completeness-Constrained LIME (CLIMB) that is superior to LIME and much faster than SHAP.

1 0

Princeton AI Club Seminar Series: Yoshua Bengio, July 14 at 4pm
by Emily C. Lawrence 08 Jul '22

08 Jul '22

Princeton AI Club Seminar Speaker : Prof. Yoshua Bengio Title : GFlowNets and AI for Science Date : Thursday, July 14 at 4:00 PM ET Location : Zoom (register to receive the link) Registration : [ https://forms.gle/iFoKEty2BVmp3nKQ9 | https://forms.gle/iFoKEty2BVmp3nKQ9 ] For more info : [ https://aiclub.princeton.edu/seminars/ | https://aiclub.princeton.edu/seminars/ ] Abstract : Machine learning research is expanding its reach, beyond the traditional realm of the tech industry and into the activities of other scientists, opening the door to truly transformative advances in these disciplines. In this talk I will focus on two aspects, modeling and experimental design, that are intertwined in the theory-experiment-analysis active learning loop that constitutes a core element of the scientific methodology. Computers will be necessary to go beyond the currently purely manual research loop and take advantage of high-throughput experimental setups and large-scale experimental datasets. I will introduce a novel machine learning framework called GFlowNets (for “Generative Flow Networks”), related to reinforcement learning, generative modeling and variational methods and conceived as an ML-driven replacement for MCMC. GFlowNets were first used to propose a highly diverse set of molecular candidates and were then incorporated in an active learning framework for efficiently looking for molecules with desirable properties. More recently, we have been exploring how GFlowNets can generate not just molecular graphs but also causal graphs and Bayesian posterior distributions in function space. I will describe our research program to build on these bases and develop machine learning methodologies for efficiently exploring the space of causal theories as well as the space of experiments while characterizing the ambiguities left by finite datasets and non-identifiability, as well as our plans to apply these tools in areas of great societal need like the unmet challenge of antimicrobial resistance. Bio : Recognized worldwide as one of the leading experts in artificial intelligence, Yoshua Bengio is most known for his pioneering work in deep learning, earning him the 2018 A.M. Turing Award, “the Nobel Prize of Computing,” with Geoffrey Hinton and Yann LeCun. He is a Full Professor at Université de Montréal, and the Founder and Scientific Director of Mila – Quebec AI Institute. He co-directs the CIFAR Learning in Machines & Brains program as Senior Fellow and acts as Scientific Director of IVADO. In 2019, he was awarded the prestigious Killam Prize and in 2021, became the second most cited computer scientist in the world. He is a Fellow of both the Royal Society of London and Canada, Knight of the Legion of Honor of France and Officer of the Order of Canada. Concerned about the social impact of AI and the objective that AI benefits all, he actively contributed to the Montreal Declaration for the Responsible Development of Artificial Intelligence.

1 0

Zachary Teed will present his FPO "Optimization Inspired Neural Networks for Multiview 3D Reconstruction" on Friday, July 8, 2022 at 3:00 PM in Friend 202.
by Louis W. Riehl 01 Jul '22

01 Jul '22

Zachary Teed will present his FPO "Optimization Inspired Neural Networks for Multiview 3D Reconstruction" on Friday, July 8, 2022 at 3:00 PM in Friend 202. Location: Friend Center 202 The members of Zachary’s committee are as follows: Examiners: Jia Deng (Adviser), Adam Finkelstein, Felix Heide Readers: Szymon Rusinkiewicz, Kilian Weinberger (Cornell University) A copy of his thesis is available upon request. Please email gradinfo(a)cs.princeton.edu if you would like a copy of the thesis. Everyone is invited to attend his talk. Abstract follows below: 3D reconstruction is usually formulated as an optimization problem. We define a measure of solution quality (objective) and design a search algorithm for finding good solutions (optimizer). Both problems are challenging. The objective needs to adequately capture the complexity of the 3D world while simultaneously being simple enough such that it is tractable using standard techniques. This is in contrast to deep learning where inference is performed using a neural network with learned weights. Despite the success of neural networks on semantic tasks their performance on multiview reconstruction tasks is often less accurate and less generalizable than approaches based purely in optimization. In this thesis, we show that optimization algorithms can be learned from data and used for motion estimation and reconstruction. We design a generic network architecture with interleaves interative updates with domain specific implicit layers. We show that such networks can automate both modeling (objective function design) and optimization. We apply this approach to a three representative multiview problems: optical flow, scene flow, and simultaneous localization and mapping. We push forward performance on these problems while using a general design strategy. Louis Riehl Graduate Administrator Computer Science Department, CS213 Princeton University (609) 258-8014

1 0

Meryem Essaidi will present her Pre FPO "User-Centered Algorithmic Mechanism Design" on Thursday, June 30th 2022, at 10:30am via Zoom.
by Nicki Mahler 28 Jun '22

28 Jun '22

Meryem Essaidi will present her Pre FPO "User-Centered Algorithmic Mechanism Design" on Thursday, June 30th 2022, at 10:30am via Zoom. Zoom link: https://princeton.zoom.us/j/93387293206 The members of her committee are as follows: Prof. Matt Weinberg (adviser) , Prof. Mark Braverman (reader), Prof. Rediet Abebe (reader), Prof. Zeev Dvir (examiner), and Prof. Huacheng Yu (examiner). All are welcome to attend. Title: "User-Centered Algorithmic Mechanism Design" Abstract: In algorithmic mechanism design, classical desiderata are typically seller-centered and aim to protecting them and maximizing their well-being. Examples of such objectives are efficiency, revenue or profit maximization, and strategyproofness. This often creates negative externalities that harm the users purchasing the goods and the people targeted by the market. Examples of such externalities are not treating the users fairly and equally, not protecting against malicious or adversarial sellers, and not maximizing the social welfare or utility of the users. Many frameworks attempt to mitigate these undesirable outputs by either setting constraints on current goals or by redesigning the goals altogether. In our work, we focus on the latter and design frameworks that center user well-being. We ask, how can we design new solutions that prioritize user-centered desiderata while maintaining the preexisting seller-centered desiderata or without too great a cost to the classical seller-centered desiderata such as profit-maximization and strategyproofness? The challenge stems from balancing (sometimes conflicting) interests of both parties and using tools from algorithmic game theory to ensure that all the agents involved will behave as expected and the desired output is achieved. In this thesis, we present three different settings where we study such problems. Project 1. First, in [1] we consider a revenue-maximizing seller with multiple items for sale to a single population of buyers. Our main result shows that for a single population of additive buyers with independent (but not necessarily identically distributed) item values, bundling all items together achieves a constant-factor approximation to the revenue-optimal itemsymmetric mechanism. We motivate this direction via fairness in ad auctions. In ad auction domains the items correspond to views from particular demographics, and recent works have therefore identified a novel fairness constraint: equally-qualified users from different demographics should be shown the same desired ad at equal rates. Prior work abstracts this to the following fairness guarantee: if an advertiser places an identical bid on two users, those two users should view the ad with the same probability. We first propose a relaxation of this guarantee from worst-case to Bayesian settings, which circumvents strong impossibility results from these works, and then study this guarantee through the lens of symmetries, as any item-symmetric auction is also fair (by this definition). Observe that in this domain, bundling all items together corresponds to concealing all demographic data. Project 2. Next, in [2] we consider a revenue-maximizing seller with a single item for sale to multiple buyers with independent and identically distributed valuations. Previous work showed that the only optimal, credible, strategyproof auction is the ascending price auction with reserves which has unbounded communication complexity. In this work, assuming the existence of cryptographically secure commitment schemes, we identify a new single-item auction that is credible, strategyproof, revenue optimal, and terminates in constant rounds in expectation for all distributions with finite monopoly price. Project 3. Finally, in [3] we study a problem inspired by regulated health insurance markets, such as those created by the government in the Affordable Care Act Exchanges or by employers when they contract with private insurers to provide plans for their employees. The market regulator can choose to do nothing, running a Free Market, or can exercise her regulatory power by limiting the entry of providers (decreasing consumer welfare by limiting options, but also decreasing revenue via enhanced competition). We investigate whether limiting entry increases or decreases the utility (welfare minus revenue) of the consumers who purchase from the providers, specifically in settings where the outside option of ”purchasing nothing” is prohibitively undesirable.

1 1

Kevin Lee will be presenting his FPO "The Research-Practice Gap in User Authentication" on June 30th, 2022 at 2pm via Zoom.
by Nicki Mahler 27 Jun '22

27 Jun '22

Kevin Lee will be presenting his FPO "The Research-Practice Gap in User Authentication" on June 30th, 2022 at 2pm via Zoom. His committee members are as follows: Arvind Narayanan (adviser), Examiners: Andrés Monroy-Hernández, and Tom Ristenpart (Cornell); Readers: Jonathan Mayer and Prateek Mittal Zoom: [ https://princeton.zoom.us/j/99377975766 | https://princeton.zoom.us/j/99377975766 ] A copy of his thesis is available upon request. Please email gradinfo(a)cs.princeton.edu if you would like a copy of the thesis. Everyone is invited to attend his talk. Abstract: [ https://docs.google.com/document/d/1unvunt1lEYqoMWlGaH1DQRBq7HwGF5tni5sefrj… | https://docs.google.com/document/d/1unvunt1lEYqoMWlGaH1DQRBq7HwGF5tni5sefrj… ] The gap between user authentication research and practice has led to weaknesses in critical, widely-deployed systems used by millions of people. In these systems, policy and process vulnerabilities—not software vulnerabilities—allow UI-bound, low-tech adversaries to exploit weaknesses to threaten user safety. The disconnect is caused partly by practice failing to heed advice from research. But it is also caused by research not understanding the practical constraints of these systems, while discouraging studies that try to do so. Ultimately, users are the ones who suffer when these weaknesses remain undiscovered. Here, we studied user authentication practices that were not necessarily cutting-edge, but broadly impacted user safety. We identified security policy and process flaws, quantified the risk of harm to users through manual measurements, and called for policy solutions to mitigate the risks. More broadly, we honed a methodology through these studies which can potentially bridge the research-practice gap in user authentication as well as in other topics in information security. First, we studied call center authentication for SIM swap requests at mobile carriers. We found flaws in their authentication policy and processes which could facilitate SIM swap attacks. Furthermore, we found that most websites did not stand up well against SIM swaps, demonstrating that users' accounts could easily be hijacked. Our results have influenced policy changes at carriers and websites, and have motivated ongoing rulemaking by the FCC. Next, we studied security and privacy risks of phone number recycling in the U.S. at mobile carriers. We found that most numbers we sampled were recycled and vulnerable to attacks on previous owners, while carriers had design weaknesses that could facilitate attacks. We have raised awareness about the risks of number recycling at carriers, and have communicated a practical constraint of SMS-based authentication to the research community. Finally, we studied password policies of top websites. Despite well-established recommendations from research, we found few websites actually following them, which could put accounts at risk of password compromise. We hypothesized reasons why these websites were not following best practices, and discussed ways the research community could engage website system administrators to bridge the research-practice gap.

1 0

Princeton AI Club guest speaker: Prof. Emtiyaz Khan - June 27th, at 10 am ET!
by Emily C. Lawrence 21 Jun '22

21 Jun '22

The Princeton AI Club is excited to announce its next guest speaker – [ https://emtiyaz.github.io/ | Prof. Emtiyaz Khan ] from the RIKEN Center for AI Project! His talk will focus on: How can we design AI systems that imitate humans and animals to quickly adapt to their surroundings? If you want to know the answer to this intriguing question, sign-up for his virtual talk [ https://forms.gle/ghtmESmGr2Lr52na9 | here ] , and join us on June 27 th , at 10 am Eastern Time (ET) ! The full details are below. Speaker: Prof. Emtiyaz Khan Date: Monday 27th June 2022, 10:00 AM ET Title: The Bayesian Learning Rule for Adaptive AI Abstract : Humans and animals have a natural ability to autonomously learn and quickly adapt to their surroundings. How can we design AI systems that do the same? In this talk, I will present Bayesian principles to bridge such gaps between humans and AI. I will show that a wide-variety of machine-learning algorithms are instances of a single learning-rule called the Bayesian learning rule. The rule unravels a dual perspective yielding new adaptive mechanisms for machine-learning based AI systems. My hope is to convince the audience that Bayesian principles are indispensable for an AI that learns as efficiently as we do. Reference: The Bayesian Learning Rule, (Preprint) M.E. Khan, H. Rue [ [ https://arxiv.org/abs/2107.04562 | arXiv ] ] [ [ https://twitter.com/EmtiyazKhan/status/1414498922584711171?s=20 | Tweet ] ] Bio: Emtiyaz Khan (also known as Emti) is a team leader at the RIKEN center for Advanced Intelligence Project (AIP) in Tokyo where he leads the Approximate Bayesian Inference Team. He is also an external professor at the Okinawa Institute of Science and Technology (OIST). Previously, he was a postdoc and then a scientist at Ecole Polytechnique Fédérale de Lausanne (EPFL), where he also taught two large machine learning courses and received a teaching award. He finished his PhD in machine learning from University of British Columbia in 2012. The main goal of Emti’s research is to understand the principles of learning from data and use them to develop algorithms that can learn like living beings. For more than a decade, his work has focused on developing Bayesian methods that could lead to such fundamental principles. The approximate Bayesian inference team now continues to use these principles, as well as derive new ones, to solve real-world problems.

1 0

Sergiy Popovych will present his FPO "Self-Supervised Metric Learning for Alignment of Petascale Connectomics Datasets" on Tuesday, June 21, 2022 at 6:00 PM via Zoom.
by Louis W. Riehl 14 Jun '22

14 Jun '22

Sergiy Popovych will present his FPO "Self-Supervised Metric Learning for Alignment of Petascale Connectomics Datasets" on Tuesday, June 21, 2022 at 6:00 PM via Zoom. Location: Zoom link: https://princeton.zoom.us/j/95852144614 The members of Sergiy’s committee are as follows: Examiners: H. Sebastian Seung (Adviser), Karthik Narasimhan, Stephan Saalfeld (Janelia Research Campus) Readers: Kai Li, Viren Jain (Google) A copy of his thesis is available upon request. Please email gradinfo(a)cs.princeton.edu if you would like a copy of the thesis. Everyone is invited to attend his talk. Abstract follows below: The reconstruction of neural circuits from serial section electron microscopy (ssEM) images is being accelerated by automatic image segmentation methods. These methods are often limited by the preceding step of aligning 2D section images to create a 3D image stack. Precise and robust alignment in the presence of image artifacts is challenging, especially as datasets are attaining the petascale. The problem of 3D stack alignment can be divided into two subproblems – image pair alignment and alignment globalization. Image pair alignment is. This dissertation presents a computational pipeline for aligning ssEM images with several key elements. First, a self‐supervised convolutional net are trained via metric learning to encode and align image pairs, followed by iterative finetuning of alignment. Second, a procedure called vector voting is used to remove outliers, further increasing robustness to image defects. Third, for speedup the series is divided into blocks that are distributed to computational workers for alignment. Fourth, the blocks are aligned to each other by composing transformations with decay, which achieves a global alignment without resorting to a time‐consuming global optimization. The pipeline is used to align a female adult fly brain dataset, as well as a cubic millimeter of mouse visual cortex and is publicly available through two open source Python packages. Development of the presented pipeline constitutes a team effort by Seunglab members. Thomas Macrina and Barak Nehoran took lead roles in development of global alignment and vector voting based error recovery. The author of this dissertation Sergiy Popovych has taken a lead role in image pair alignment, seethrough‐based error recovery, as well as the development and architecture of the pipeline software stack. Nico Kemnitz, Manuel Castro, and Dodam Ih have contributed a large variety of software and research components. Louis Riehl Graduate Administrator Computer Science Department, CS213 Princeton University (609) 258-8014

1 0

Yuping Luo will present his FPO "Towards Designing Efficient and Effective Deep Model-based Reinforcement Learning Algorithms" on Tuesday, June 21, 2022 at 1:30 PM via Zoom.
by Louis W. Riehl 14 Jun '22

14 Jun '22

Yuping Luo will present his FPO "Towards Designing Efficient and Effective Deep Model-based Reinforcement Learning Algorithms" on Tuesday, June 21, 2022 at 1:30 PM via Zoom. Location: Zoom link: [ https://princeton.zoom.us/j/94826666865 | https://princeton.zoom.us/j/94826666865 ] The members of Yuping’s committee are as follows: Examiners: Sanjeev Arora (Adviser), Karthik Narasimhan, Tengyu Ma (Stanford University) Readers: Elad Hazan, Jason Lee A copy of his thesis is available upon request. Please email gradinfo(a)cs.princeton.edu if you would like a copy of the thesis. Everyone is invited to attend his talk. Abstract follows below: Recent advances in deep reinforcement learning have demonstrated its great potential for real-world problems. However, two concerns prevent reinforcement learning from being applied: Efficiency and Efficacy. This dissertation studies how to design improving the efficiency and efficacy of reinforcement learning by designing deep model-based algorithms. The access to dynamics models empowers the algorithms to plan, which is key to sequential decision making. Four topics are covered in this dissertation, which are online reinforcement learning, the expressivity of neural networks in deep reinforcement learning, offline reinforcement learning, and safe reinforcement learning. For online reinforcement learning, we present an algorithmic framework with theoretical guarantees by utilizing a lower bound of performance the policy learned in the learned environment can obtain in the real environment, and we also empirically verify the efficiency of our proposed method. For expressivity of neural networks in deep reinforcement learning, we prove that in some scenarios, the model-based approaches can require much less representation power to approximate a near-optimal policy than model-free approaches, and empirically show that this can be an issue in physically simulation environments and a model-based planner can help. For offline reinforcement learning, we devise an algorithm that stays close to the provided expert demonstration set to reduce distribution shift, and we also conduct experiments to demonstrate the efficacy of our methods to improve the success rate for simulated robotics environments. For safe reinforcement learning, we propose a method that uses the learned dynamics model to certify safe states, and our experiments show that our method can learn a decent policy without a single safety violation during training in a set of simple but challenging tasks while baseline algorithms have hundreds of safety violations. Louis Riehl Graduate Administrator Computer Science Department, CS213 Princeton University (609) 258-8014

1 0

Paul Krueger will present his Pre-FPO "Metacognition: learning how to think" on Wednesday, June 8, 2022 at 3:00 PM via Zoom.
by Louis Riehl 07 Jun '22

07 Jun '22

Paul Krueger will present his Pre-FPO "Metacognition: learning how to think" on Wednesday, June 8, 2022 at 3:00 PM via Zoom. Zoom link: https://us04web.zoom.us/j/2902028164?pwd=ZHdhZjZKWm4vK0pZai9nbWpkNlo4Zz09 Committee Members: Tom Griffiths (Examiner, advisor), Ryan Adams (Examiner), Jonathan Cohen (examiner), Nathaniel Daw (reader), Karthik Narasimhan (reader). All are welcome to attend. Title: Metacognition: learning how to think Abstract: Reinforcement learning has been used successfully to account for human decision-making and to produce intelligent behavior in artificial agents. However, people's internal cognitive processes generally remain obscure with these models. The same formalism used to account for external actions can also be applied to cognitive processes themselves. In particular, using the recently developed framework of resource rationality we can describe how people make efficient use of limited cognitive resources. Using this approach, we apply tools from machine learning to derive a normative account of how people ought to think efficiently. As a proof of concept, I will show how this framework can be used to automatically derive decision-making heuristics. I will present results from a classic multi-alternative risky choice task that allows us to trace people's cognitive processes. Using our method, we both re-discovered previously known heuristics, and discovered heuristics that had been overlooked. We found that people did indeed use the same heuristics that our method predicted, and furthermore adapted their heuristics to the structure of the decision environment in a rational way. Our approach bridges classic rationality with research on heuristics and biases by providing a general framework that can be used to apply rationality to any decision-making process while accounting for heuristics, thus providing a more realistic normative standard. In the second part of the talk, I will discuss a new framework for how model-based and model-free reinforcement learning systems can interact cooperatively, and the potential to account for human decision-making with this approach. In particular, this cooperation may be used to mediate the tension between goals and habits. I will close by discussing ongoing work to develop tools to solve meta-MDPs, the formalism used for our resource-rational approach.

1 0