Off-Policy Evaluation and Learning for External Validity under a Covariate Shift
Advances in Neural Information Processing Systems (NeurIPS), 2020. Spotlight (top 3%, 280/9054).
- off-policy evaluation
- covariate shift
- external validity
- doubly robust
In one sentence. Off-policy evaluation and learning when the population you will deploy on differs from the population that generated the logs: the paper derives the efficiency bound under covariate shift and gives doubly robust, efficient estimators.
The external validity gap
Off-policy evaluation estimates the expected reward of a new policy from data logged under a different policy, and off-policy learning searches for the policy that maximizes it. Both standardly assume that the covariate distribution of the historical data matches that of the population where the new policy will run.
That assumption is frequently false — the logs come from last quarter's users, or from one market, and the policy is meant for another. This is a covariate shift, and it is exactly the external validity problem in econometrics stated in a bandit vocabulary.
Contributions
- The efficiency bound for off-policy evaluation under a covariate shift.
- Doubly robust and efficient estimators for both evaluation and learning, built from a nonparametric estimator of the density ratio between the historical and evaluation covariate distributions.
- A comparison of alternative estimators and their theoretical properties, with experiments confirming the proposed ones.