Niranjani Prasad will present her generals exam Thursday, May 25, 2017 at 10am in CS 401. The members of her committee are: Barbara Engelhardt (adviser), Warren Powell, and Sebastian Seung. Everyone is invited to attend her talk, and those faculty wishing to remain for the oral exam following are welcome to do so. Her abstract and reading list follow below. Title: A Reinforcement Learning Approach to Intervention Protocols in the ICU Abstract: The management of routine interventions such as mechanical ventilation and sedation, blood transfusions, or the administration of fluids and vasopressors, constitute a major part of the care of patients in intensive care units (ICUs). Timely and proportionate interventions are crucial to improving patient outcomes and reducing hospital costs, but the effect of these procedures are often poorly understood—particularly when handling heterogenous patient populations—and clinical opinion on best protocols vary. In this work, we focus on the development of decision support for weaning from mechanical ventilation, that leverages available information in the data-intensive ICU setting to predict time to extubation readiness, and recommend a personalized regime of sedation and ventilator support. To this end, we employ off-policy reinforcement learning algorithms to learn an optimal sequence of treatment actions from sub-optimal historical ICU data. We model patient admissions as Markov decision processes, developing tailored representations of the problem state, action space and reward function, and learn treatment policies using Fitted Q-iteration with tree-based regressors and with feedforward neural networks. We demonstrate that this framework shows promised in recommending intervention protocols with improved outcomes, on average outperforming clinical practice in controlling physiological stability and reintubation rates. Reading List: Textbook: Sutton, Richard S., and Andrew G. Barto. Reinforcement learning: An introduction . Vol. 1. No. 1. Cambridge: MIT press, 1998. Papers: Ernst, Damien, Pierre Geurts, and Louis Wehenkel. " Tree-based batch mode reinforcement learning ." Journal of Machine Learning Research 6, 2005. Riedmiller, Martin. " Neural fitted Q iteration–first experiences with a data efficient neural reinforcement learning method. " European Conference on Machine Learning. Springer Berlin Heidelberg, 2005. Mnih, Volodymyr, et al. " Human-level control through deep reinforcement learning ." Nature 518.7540, 2015 . Rasmussen, Carl Edward, and Malte Kuss. " Gaussian Processes in Reinforcement Learning ." NIPS. Vol. 4. 2003. Abbeel, Pieter, and Andrew Y. Ng. " Apprenticeship learning via inverse reinforcement learning ." Proceedings of the twenty-first international conference on Machine learning. ACM, 2004. Escandell-Montero, Pablo, et al. " Optimization of anemia treatment in hemodialysis patients via reinforcement learning ." Artificial intelligence in medicine 62.1, 2014. Guez, Arthur, et al. " Adaptive Treatment of Epilepsy via Batch-mode Reinforcement Learning ." AAAI. 2008. Ernst, Damien, et al. " Clinical data based optimal STI strategies for HIV: a reinforcement learning approach ." Decision and Control, 2006 45th IEEE Conference on. IEEE, 2006. Nemati, Shamim, Mohammad M. Ghassemi, and Gari D. Clifford. " Optimal medication dosing from suboptimal clinical examples: A deep reinforcement learning approach ." Engineering in Medicine and Biology Society (EMBC), IEEE, 2016. Wang, Yingfei, and Warren Powell. " An optimal learning method for developing personalized treatment regimes. " arXiv preprint arXiv: 1607.01462 2016 . Padmanabhan, Regina, Nader Meskin, and Wassim M. Haddad. " Closed-loop control of anesthesia and mean arterial pressure using reinforcement learning ." Biomedical Signal Processing and Control 22, 2015. Wu, Mike, et al. " Understanding vasopressor intervention and weaning: Risk prediction in a public heterogeneous clinical time series database ." Journal of the American Medical Informatics Association, 2016.