Advisor: Arvind Narayanan (examiner)
Committee members: Adji Bousso Dieng (reader), Andrés Monroy-Hernández (examiner), Barbara Engelhardt (reader), Matt Salganik (examiner)
Title: Human-machine Cooperation in Machine-Learning Resistant Real-World Applications
All are welcome to attend.
Abstract:
Automation tools like machine learning are a necessity in our big data world. However, not all applications lend themselves to machine-learning based automation. Numerous societal problems emerge when we attempt to measure and quantify something that resists measurement and subsequently, develop mathematical intermediaries to attempt to model human behavior. However, using machine-learning backed automation to reduce the scale of many real-world applications is still helpful. In this dissertation, we explore three different ways in which we can use human-machine cooperation in machine-learning resistant applications to keep or increase utility and reduce harm when using these systems in the wild.
The first chapter, based on “Friend Request Pending: A Comparative Assessment of Engineering and Social Science Inspired Approaches to Analyzing Complex Birth Cohort Survey Data,” shows how the first mode of human-machine cooperation—explaining things to computers by adding domain knowledge—increases utility of machine learning models in the machine-learning resistant task of grade point average (GPA) prediction. The Fragile Families Challenge is a mass collaboration social science data challenge whose aim is to learn how various early childhood variables predict the long-term outcomes of children. We describe our two-step approach to the Fragile Families Challenge. In step 1, we use a variety of fully automated approaches to predict child academic achievement. We fit 124 models, which involve most possible combinations of 8 model types, 2 imputation strategies, 2 standardization approaches, and 2 automatic variable selection techniques using 2 different thresholds. Then, in step 2, we attempt to improve on the results from step 1 with manual variable selection based on a detailed review of the codebooks. We manually selected 3,694 variables believed to be predictive of academic achievement, using a comprehensive review of student success literature to guide the decision-making process. The best models from step 1 were re-estimated using the manually selected variables. We show that manual variable selection improved the majority of the top 10 models in step 1, but did not improve the best of the top 10. Results indicate that variable selection inspired by social science methodologies can, in most cases, significantly improve models trained completely automatically.
The second chapter, based on “Selectively Contextual Bandits,” shows how the second mode of human-machine cooperation—humans and computers working together in decision making—maintains utility while reducing harm of machine learning models in the machine-learning resistant task of image personalization. Personalization is an integral part of most web-service applications and determines which experience to display to each member. A popular algorithmic framework used in industrial personalization systems are contextual bandits, which seek to learn a personalized treatment assignment policy in the presence of treatment effects that vary with the observed contextual features of the members. In order to keep the optimization task tractable, such systems can myopically make independent personalization decisions that can conspire to create a suboptimal experience in the aggregate of the member’s interaction with the web-service. We design a new family of online learning algorithms that benefit from personalization while optimizing the aggregate impact of the many independent decisions. Our approach selectively interpolates between any contextual bandit algorithm and any context-free multi-armed bandit algorithm and leverages the contextual information for a treatment decision only if this information promises significant gains over a decision that does not take it into account. Apart from helping users of personalization systems feel less targeted, simplifying the treatment assignment policy by making it selectively reliant on the context can help improve the rate of learning. We evaluate our approach on several datasets including a video subscription web-service and show the benefits of such a hybrid policy.
Finally, the last chapter, based on “On the Bias-Variance Characteristics of LIME and SHAP in High Sparsity Movie Recommendation Explanation Tasks,” shows how the third mode of human-machine cooperation—computers explaining to humans what it’s thinking—can improve utility and reduce harm of machine learning models in the machine-learning resistant task of movie recommendations. We evaluate two popular local explainability techniques, LIME and SHAP, on a movie recommendation task. We discover that the two methods behave very differently depending on the sparsity of the data set. LIME does better than SHAP in dense segments of the data set and SHAP does better in sparse segments. We trace this difference to the differing bias-variance characteristics of the underlying estimators of LIME and SHAP. We find that SHAP exhibits lower variance in sparse segments of the data compared to LIME. We attribute this lower variance to the completeness constraint property inherent in SHAP and missing in LIME. This constraint acts as a regularizer and therefore increases the bias of the SHAP estimator but decreases its variance, leading to a favorable bias-variance trade-off especially in high sparsity data settings. With this insight, we introduce the same constraint into LIME and formulate a novel local explainabilty framework called Completeness-Constrained LIME (CLIMB) that is superior to LIME and much faster than SHAP.