Personalization is an integral part of most web-service applications and determines which experience to display to each member. A popular algorithmic framework used in industrial personalization systems are contextual bandits, which seek to learn a personalized treatment assignment policy in the presence of treatment effects that vary with the observed contextual features of the members. In order to keep the optimization task tractable, such systems can myopically make independent personalization decisions that can conspire to create a suboptimal experience in the aggregate of the member’s interaction with the web-service. We design a new family of online learning algorithms that benefit from personalization while optimizing the aggregate impact of the many independent decisions. Our approach selectively interpolates between any contextual bandit algorithm and any context-free multi-armed bandit algorithm and leverages the contextual information for a treatment decision only if this information promises significant gains over a decision that does not take it into account. Apart from helping users of personalization systems feel less targeted, simplifying the treatment assignment policy by making it selectively reliant on the context can help improve the rate of learning. We evaluate our approach on several datasets including a video subscription web-service and show the benefits of such a hybrid policy.

Reading List: 

Methods/Techniques (Contextual Bandits)

• An Empirical Evaluation of Thompson Sampling

• A Contextual-Bandit Approach to Personalized News Article Recommendation

• A Tutorial on Thompson Sampling 

• Unbiased Offline Evaluation of Contextual-bandit-based News Article Recommendation Algorithms 

• Online Evaluation for Information Retrieval

Textbooks

• Reinforcement Learning: An Introduction (Ch 2, 16.7)

• Mining of Massive Datasets (Ch 8, 9)

• Bandit Algorithms for Website Optimization 

Historical Context 

• Foundations and Grand Challenges of Artificial Intelligence 

Motivating

• The Perpetuation of Subtle Prejudice: Race and Gender Imagery in 1990s Television Advertising

• Getting too personal: Reactance to highly personalized email solicitations

• Privacy and human behavior in the age of information

• The personalization privacy paradox: An exploratory study of decision making process for location-aware marketing

• Auditing Search Engines for Differential Satisfaction Across Demographics