arXiv 12 Jun 2020 · Statistics — Machine Learning · 2 citations (OpenAlex)
arXiv:2006.06982 · PDF · DOI · OpenAlex · Extracted main text
This study addresses the problem of off-policy evaluation (OPE) from dependent samples obtained via the bandit algorithm. The goal of OPE is to evaluate a new policy using historical data obtained from behavior policies generated by the bandit algorithm. Because the bandit algorithm updates the policy based on past observations, the samples are not independent and identically distributed (i.i.d.). However, several existing methods for OPE do not take this issue into account and are based on the assumption that samples are i.i.d. In this study, we address this problem by constructing an estimator from a standardized martingale difference sequence. To standardize the sequence, we consider using evaluation data or sample splitting with a two-step estimation. This technique produces an estimator with asymptotic normality without restricting a class of behavior policies. In an experiment, the proposed estimator performs better than existing methods, which assume that the behavior policy converges to a time-invariant policy.
appendix boundary found by appendix_command · 53% of the source is main text. Read the extracted text to check this.
The works this paper leans on most, across its whole bibliography — not restricted to papers in our corpus. Ranked by composite intensity, which combines how often a work is mentioned, how many sections mention it, and how much of that falls in the main text rather than the appendix.
| Reference | Intensity | Mentions | Sections | Main text | |
|---|---|---|---|---|---|
| 1 | Kato, M., Ishihara, T., Honda, J., and Narita, Y (2002) Adaptive experimental design for efficient treatment effect estimation: Randomized allocation via contextual bandit algorithm self | 1.000 | 11 | 3 | 100% |
| 2 | Hadad, V., Hirshberg, D. A., Zhan, R., Wager, S., and Athey, S (2019) Confidence intervals for policy evaluation in adaptive experiments, 2019 | 1.000 | 7 | 3 | 100% |
| 3 | Chernozhukov, V., Chetverikov, D., Demirer, M., Duflo, E., Hansen, C… (2018) Double/debiased machine learning for treatment and structural parameters | 0.928 | 5 | 4 | 80% |
| 4 | der Laan, V. and Mark, J. I (2008) The construction and analysis of adaptive group sequential designs | 0.874 | 5 | 2 | 100% |
| 5 | Narita, Y., Yasui, S., and Yata, K (2019) Efficient counterfactual learning from bandit feedback | 0.811 | 4 | 2 | 100% |
| 6 | Kallus, N. and Uehara, M (2019) Intrinsically efficient, stable, and bounded off-policy evaluation for reinforcement learning | 0.737 | 3 | 2 | 100% |
| 7 | Dudḱ, M., Langford, J., and Li, L (2011) Doubly Robust Policy Evaluation and Learning | 0.737 | 3 | 2 | 100% |
| 8 | Hahn, J., Hirano, K., and Karlan, D (2011) Adaptive experimental design using the propensity score | 0.644 | 2 | 2 | 100% |
| 9 | Bickel, P. J., Klaassen, C. A. J., Ritov, Y., and Wellner, J. A (1998) Efficient and Adaptive Estimation for Semiparametric Models | 0.644 | 2 | 2 | 100% |
| 10 | Yang, Y. and Zhu, D (2002) Randomized allocation with nonparametric estimation for a multi-armed bandit problem with covariates | 0.644 | 2 | 2 | 100% |
Showing the top 10 of 39 scored citations.
arXiv econ.EM papers that cite this one, ranked by how heavily they lean on it.
| Citing paper | Intensity | Mentions | Sections | |
|---|---|---|---|---|
| 1 | A Practical Guide of Off-Policy Evaluation for Bandit Problems | 0.000 | 1 | 1 |