Maria Dimakopoulou, Zhengyuan Zhou, Susan Athey, Guido Imbens
arXiv 19 Nov 2017 · Statistics — Machine Learning · 27 citations (OpenAlex)
arXiv:1711.07077 · PDF · DOI · OpenAlex · Extracted main text
Contextual bandit algorithms are sensitive to the estimation method of the outcome model as well as the exploration method used, particularly in the presence of rich heterogeneity or complex outcome models, which can lead to difficult estimation problems along the path of learning. We study a consideration for the exploration vs. exploitation framework that does not arise in multi-armed bandits but is crucial in contextual bandits; the way exploration and exploitation is conducted in the present affects the bias and variance in the potential outcome model estimation in subsequent stages of learning. We develop parametric and non-parametric contextual bandits that integrate balancing methods from the causal inference literature in their estimation to make it less prone to problems of estimation bias. We provide the first regret bound analyses for contextual bandits with balancing in the domain of linear contextual bandits that match the state of the art regret bounds. We demonstrate the strong practical advantage of balanced contextual bandits on a large number of supervised learning datasets and on a synthetic example that simulates model mis-specification and prejudice in the initial training data. Additionally, we develop contextual bandits with simpler assignment policies by leveraging sparse model estimation methods from the econometrics literature and demonstrate empirically that in the early stages they can improve the rate of learning and decrease regret.
appendix boundary found by appendix_command · 67% of the source is main text. Read the extracted text to check this.
The works this paper leans on most, across its whole bibliography — not restricted to papers in our corpus. Ranked by composite intensity, which combines how often a work is mentioned, how many sections mention it, and how much of that falls in the main text rather than the appendix.
| Reference | Intensity | Mentions | Sections | Main text | |
|---|---|---|---|---|---|
| 1 | L. Li, W. Chu, J. Langford, and R. Schapire (2010) A contextual-bandit approach to personalized news article recommendation | 0.950 | 7 | 3 | 86% |
| 2 | S. Agrawal and N. Goyal (2013) Thompson sampling for contextual bandits with linear payoffs | 0.920 | 9 | 4 | 78% |
| 3 | A. Agarwal, D. Hsu, S. Kale, J. Langford, L. Li, and R. Schapire (2014) Taming the monster: A fast and simple algorithm for contextual bandits | 0.894 | 7 | 3 | 71% |
| 4 | O. Chapelle and L. Li (2011) An empirical evaluation of thompson sampling | 0.874 | 5 | 2 | 100% |
| 5 | M. Dudik, J. Langford, and L. Li (2011) Doubly robust policy evaluation and learning | 0.737 | 3 | 2 | 100% |
| 6 | N. Kallus (2017) Balanced policy evaluation and learning | 0.737 | 3 | 2 | 100% |
| 7 | H. Lei, A. Tewari, and S. Murphy (2017) An actor-critic contextual bandit algorithm for personalized mobile health interventions | 0.737 | 3 | 2 | 100% |
| 8 | Wei Chu, Lihong Li, Lev Reyzin, and Robert Schapire (2011) Contextual bandits with linear payoff functions | 0.693 | 10 | 2 | 50% |
| 9 | S. Athey, G. Imbens, and S. Wager (2017) Approximate residual balancing | 0.644 | 2 | 2 | 100% |
| 10 | S. Athey, J. Tibshirani, and S. Wager (2017) Generalized random forests | 0.644 | 2 | 2 | 100% |
Showing the top 10 of 66 scored citations.
arXiv econ.EM papers that cite this one, ranked by how heavily they lean on it.