Learning bounds for doubly-robust covariate shift adaptation

Learning bounds for doubly-robust covariate shift adaptation

\ and Cong Ma\footnote{E-mail: [email removed]}.}} \affil{Department of Statistics at the University of Chicago}

Introduction

Classical supervised learning assumes that the training and test data are drawn from the same distribution vapnik2013nature, gyorfi2002distribution. In practice, such an assumption is rarely met. For instance, credit models are typically trained on approved customers but deployed on rejected applicants; medical imaging data vary across hospitals due to differences in equipment and protocols koh2021wilds, guan2021domain; and in natural language processing, labeled corpora such as the Wall Street Journal, differ sharply from the domains such as arXiv jiang2007instance. For all these cases, distribution shift between training and test domains undermines predictive performance.

A significant particular case of such a distribution shift is known as the covariate shift shimodaira2000improving, quinonero-candela2008dataset, pan2009survey, where the marginal distribution of covariates $X$ varies across the domains while the conditional distribution of $Y|X$ remains the same. Covariate shift is well-documented in healthcare wei2015health, hajiramezanali2018bayesian, image classification saenko2010adapting, remote sensing tuia2011using, sentiment analysis blitzer2007biographies, and speech and language processing yamada2009covariate, hassan2013acoustic, fei2015social.

The problem of covariate shift adaptation assumes access to labeled samples from a source domain and unlabeled covariates from a target domain, with the goal of learning a predictor with a desirable performance under the target distribution. This problem has been central to the literature of transfer learning and domain adaptation sugiyama2007covariate, sugiyama2007direct, sugiyama2012machine, pan2009survey, kato2023double, especially when the target labels are scarce or costly to obtain.

A core difficulty lies in estimating the covariate density ratio between the source and target domains. The standard approach -- plugging-in an estimated covariate density ratio into an importance-weighted empirical risk minimization sugiyama2007covariate, sugiyama2007direct, sugiyama2008direct, reddi2015doubly -- turns out to be highly sensitive to the estimation errors of the density ratio and performs poorly unless the estimator converges at a nearly parametric rate. To address this, kato2023double suggests a doubly-robust (DR) estimator, which augments the importance-weighting with a pilot regression model and leverages double machine learning techniques chernozhukov2017double, chernozhukov2018double, chernozhukov2022locally, chernozhukov2023simple, foster2023orthogonal. Their results establish the asymptotic normality and $\sqrt{n}$-consistency of their DR estimator under parametric models, even when the pilot estimates converge slowly.

Yet, the literature of covariate shift adaptation has centered exclusively on achieving asymptotic results. It remains unclear how the DR covariate shift adaptation performs in finite-sample regimes. This paper aims to close this gap. Our contributions can be summarized as follows:

Together, these results bridge asymptotic efficiency results and a finite-sample out-of-distribution (OOD) generalization bound, providing a comprehensive theoretical underpinning of the DR covariate shift adaptation.

Related works

We take a moment to discuss subsets of related prior works in covariate shift, doubly-robust estimation, and structure-agnostic estimation framework.

\paragraph{Covariate shift} The study of covariate shift can be dated back to the seminal paper by shimodaira2000improving. This paper investigates the impact of covariate shift under parametric models with the vanilla MLE and proposes the importance-weighting (IW) method, which has a remarkable improvement if the underlying regression model is mis-specified. It also establishes the asymptotic normality for a weighted version of MLE under covariate shift, but no finite-sample learning bounds are provided. Later, sugiyama2005model further extends this work by studying an unbiased estimator under the $L^2$-generalization error. Motivated by these fundamental works, there has been a flurry of follow-up works for parametric covariate shift. mousavi2020minimax introduces a statistical minimax framework and gives lower bounds for out-of-distribution generalization under the regression models of linear and one-hidden layer neural networks. lei2021near takes a closer inspection on the minimax optimal estimator for fixed-design linear regression under covariate shift. zhang2022class studies linear models under covariate shift where the learner has access to a small amount of target labels. In stark contrast, this work focuses on the covariate shift problem where the learner has no access to target labels.

Beyond the cases of parametric covariate shift, cortes2010learning investigate the IW estimator under the framework of statistical learning and provide a non-asymptotic upper bound on the excess target risk for the IW estimator. Also, there has been a strand of recent works on well-specified non-parametric models under covariate shift. kpotufe2021marginal investigates the non-parametric classification problem over the class of H\"{o}lder continuous functions and provides a new fine-grained similarity measure. Within a focus on the class of H\"{o}lder continuous functions, pathak2022new introduces a novel measure of distribution mismatch between the source and target domains. Under the setting of reproducing kernel Hilbert space (RKHS), ma2023optimally, gogolashvili2023importance establish the optimal learning rates of kernel ridge regression (KRR) estimators. In particular, ma2023optimally proves that KRR estimation using a carefully selected regularization parameter is miniax optimal provided that the covariate density ratio is uniformly bounded, and a re-weighting version of the KRR estimator using truncated covariate density ratios is minimax-optimal if the covariate density ratio has a finite second-order moment. On the other hand, wang2023pseudo suggests the strategy of learning a predictive model using pseudo-labels. As our final remark, over-parameterized models, such as high-dimensional models and classes of neural networks, under covariate shift has drawn increasing attention from the researchers byrd2019effect, hendrycks2019benchmarking, hendrycks2021many, tripuraneni2021overparameterization).

\paragraph{Doubly-robust (DR) estimation} Doubly-robust (DR) estimation combines an outcome regression with a model for treatment or selection (e.g., the propensity score), guaranteeing its consistency if at least one is correctly specified. Its foundations lie in the seminal paper by robins1994estimation on semi-parametric theory and influence functions, and were formalized for applications by bang2005doubly. Some implementations include the augmented inverse propensity weighting (AIPW) robins1994estimation, robins1995semiparametric, bang2005doubly and target maximum likelihood estimation (TMLE) van2006targeted, van2011targeted, both of which leverage influence functions to correct bias. A corpus of recent studies integrate modern ML techniques for flexible nuisance estimation together with the Neyman orthogonalization and sample splitting chernozhukov2017double, chernozhukov2018double, van2018targeted, kennedy2024semiparametric for retaining valid inference. The DR estimation framework has expanded to settings such as difference-in-differences sant2020doubly, ning2020doubly, instrumental variables okui2012doubly,lee2023doubly, and censored data bai2013doubly. While the DR methods achieve robustness and potential efficiency, they require careful handling of finite-sample bias kang2007demystifying, funk2011doubly, near-positivity violations cole2008constructing, and model diagnostics bang2005doubly, robins1994estimation, since the correctness of at least one nuisance estimate remains crucial.

\paragraph{Structure-agnostic estimation} The structure-agnostic estimation framework stands for a class of statistical methods for estimating functionals or treatment effects without assuming any parametric or structural models for the underlying data generating process. balakrishnan2023fundamental establishes the fundamental limits for such functional estimation, characterizing the optimal rates achievable when only minimal assumptions -- such as smoothness or boundedness -- are imposed. jin2024structure demonstrates that the DR estimators both for the average treatment effect (ATE) and the average treatment effect on the treated (ATT) attain the minimax optimal rates under the structure-agnostic estimation framework. Their findings underscore the effectiveness of the DR learning in causal inference, particularly when relying on flexible ML algorithms for nuisance estimation. jin2025s further studies the sensitivity of structure-agnostic estimation procedures to noise, highlighting several cases where standard estimators fail to achieve normality or efficiency. Finally, bonvini2024doubly extends the framework by formalizing the DR inference under smoothness conditions. Collectively, these recent works aim to construct a rigorous framework for statistical estimation and inference that minimizes reliance on structural assumptions while achieving near-optimal statistical guarantees.

Problem formulation

Let $\mathbb{X}$ denote the covariate space (a.k.a., the feature space). Consider the source distribution $\mathbb{P} \in \Delta \left( \mathbb{X} \times \mathbb{R} \right)$ and the target distribution $\mathbb{Q} \in \Delta \left( \mathbb{X} \times \mathbb{R} \right)$. Also, let $\mathbb{P}_{X} \in \Delta(\mathbb{X})$ and $\mathbb{Q}_{X} \in \Delta(\mathbb{X})$ denote by the marginal distributions of $X$ under $\mathbb{P}$ and $\mathbb{Q}$, respectively. We further define $\mathbb{P}_{Y \mid X} : \mathbb{X} \to \Delta(\mathbb{R})$ and $\mathbb{Q}_{Y \mid X} : \mathbb{X} \to \Delta(\mathbb{R})$ to be the conditional laws of $Y$ given $X$ under $\mathbb{P}$ and $\mathbb{Q}$: \[ \mathbb{P}_{Y \mid X} \left( \cdot \mid x \right) := \mathbb{P} \left( Y \in \cdot \left| X = x \right. \right) \quad \textnormal{and} \quad \mathbb{Q}_{Y \mid X} \left( \cdot \mid x \right) := \mathbb{Q} \left( Y \in \cdot \left| X = x \right. \right). \]

Thus, the two distributions share the same Bayes regression function $f^* : \mathbb{X} \to \mathbb{R}$, \[ f^*(x) := \mathbb{E}_{\mathbb{P}} \left[ Y \mid X = x \right] = \mathbb{E}_{\mathbb{Q}} \left[ Y \mid X = x \right], \quad x \in \mathbb{X}. \] Here, we emphasize that Assumption (ref) does not require $\mathbb{P}_{Y \mid X} = \mathbb{Q}_{Y \mid X}$; only their Bayes regression functions must coincide. In fact, this assumption is weaker compared to the classical covariate shift model shimodaira2000improving, which posits a full equality of the conditional distributions.

\paragraph{Observational data.} We observe $n_{\mathbb{P}}$ labeled samples from the source distribution $\mathbb{P}$, \[ \mathbf{O}^{\mathbb{P}}_{1:n_{\mathbb{P}}} := \left( O_{i}^{\mathbb{P}} := \big( X_{i}^{\mathbb{P}}, Y_{i}^{\mathbb{P}} \big) : i \in \left[ n_{\mathbb{P}} \right] \right) \sim \mathbb{P}^{\otimes n_{\mathbb{P}}}, \] and $n_{\mathbb{Q}}$ unlabeled target covariates, \[ \mathbf{X}^{\mathbb{Q}}_{1:n_{\mathbb{Q}}} = \left( X_{j}^{\mathbb{Q}} : j \in \left[ n_{\mathbb{Q}} \right] \right) \sim \mathbb{Q}_{X}^{\otimes n_{\mathbb{Q}}}. \] Hence, the labels are available only in the source domain.

\paragraph{Risk and excess risk.} Given a function class $\mathcal{F} \subseteq (\mathbb{X} \to \mathbb{R})$, we define the $\mu$-risk $\mathcal{R}_{\mu} : \mathcal{F} \to \mathbb{R}_{+}$ by \[ \mathcal{R}_{\mu} (f) := \mathbb{E}_{\left( X, Y \right) \sim \mu} \left[ \left\{ Y - f(X) \right\}^2 \right], \quad \mu \in \Delta \left( \mathbb{X} \times \mathbb{R} \right). \] Given any $\mu \in \Delta \left( \mathbb{X} \times \mathbb{R} \right)$, let $f^*_\mu \in \operatorname*{argmin} \left\{ \mathcal{R}_{\mu} (f) : f \in \mathcal{F} \right\}$ denote a $\mu$-risk minimizer over the function class $\mathcal{F}$. The excess $\mu$-risk is then defined by

\paragraph{Goal: covariate shift adaptation.} Our objective is to construct an estimator $\hat f \in \mathcal{F}$ that achieves small excess $\mathbb{Q}$-risk $\mathcal{E}_\mathbb{Q}(\hat f)$ with high probability.

\paragraph{Covariate density ratio.} A central quantity in the study of covariate shift is the covariate density ratio between the source and target distributions. We assume that the marginal distributions $\mathbb{P}_X$ and $\mathbb{Q}_X$ are all absolutely continuous with respect to a $\sigma$-finite reference measure $\mu_{\mathbb{X}}$ on $\mathbb{X}$. Let \[ p_X := \frac{\mathrm{d} \mathbb{P}_X}{\mathrm{d} \mu_{\mathbb{X}}} : \mathbb{X} \to \mathbb{R}_{+} \quad \textnormal{and} \quad q_X := \frac{\mathrm{d} \mathbb{Q}_X}{\mathrm{d} \mu_{\mathbb{X}}} : \mathbb{X} \to \mathbb{R}_{+} \] denote their respective densities with respect to $\mu_{\mathbb{X}}$. The covariate density ratio is then defined as \[ \rho^* (x) := \frac{q_X (x)}{p_X( x)}, \quad x \in \mathbb{X}, \] which is assumed to be finite everywhere throughout this paper.

Doubly-robust (DR) covariate shift adaptation

Re-weighting with respect to the source distribution $\mathbb{P}$ yields an alternative expression of the $\mathbb{Q}$-risk as the $\rho^*$-weighted $\mathbb{P}$-risk:

The importance-weighting (IW) estimator shimodaira2000improving can be obtained by minimizing the empirical analogue of the $\rho^*$-weighted $\mathbb{P}$-risk (ref) over $\mathcal{F}$. Its key limitation is the reliance on the knowledge of the unknown covariate density ratio $\rho^* : \mathbb{X} \to \mathbb{R}_{+}$: a modified estimator obtained by plugging-in an estimate $\hat{\rho} : \mathbb{X} \to \mathbb{R}_{+}$ for the covariate density ratio $\rho^*$ might have high variance and degrade its performance unless the estimation of the covariate density ratio is sufficiently accurate.

The doubly-robust (DR) covariate shift adaptation kato2023double augments the IW method with a pilot regression model, and then subtracts a squared-error correction term to cancel the leading error term incurred by the density ratio estimation. For any given pilot estimates $\hat{\rho} : \mathbb{X} \to \mathbb{R}_{+}$ and $\hat{f}_0 : \mathbb{X} \to \mathbb{R}$ for the covariate density ratio $\rho^* : \mathbb{X} \to \mathbb{R}_{+}$ and the shared Bayes regression function $f^* : \mathbb{X} \to \mathbb{R}$, respectively, let us define the DR empirical risk $\widehat{\mathcal{R}}_{\textsf{DR}} : \mathcal{F} \to \mathbb{R}$ by

and the DR estimator as

Intuitively, the pilot regression model terms $\hat f_0$ makes the risk orthogonal to the first-order errors in $\hat{\rho}$ (and vice-versa), yielding stability even when the pilot estimates converge slowly.

\paragraph{Structure-agnostic estimation.} Throughout this section, the pilot estimates $\hat{\rho} : \mathbb{X} \to \mathbb{R}_{+}$ and $\hat{f}_0 : \mathbb{X} \to \mathbb{R}$ are regarded as black-boxes: the analysis only requires the pilot estimates to achieve certain statistical error rates, not how these estimates are obtained. This structure-agnostic estimation framework balakrishnan2023fundamental, jin2024structure, kennedy2024minimax, bonvini2024doubly, jin2025s reflects practice, where the pilot estimates $\hat{\rho}$ and $\hat{f}_0$ can be obtained by leveraging a growing body of modern ML methods (e.g., LASSO bicker2009simultaneous, wainwright2009sharp, tree-based algorithms syrgkanis2020estimation, wager2018estimation, and deep neural networks chen1999improved, hieber2020nonparametric). Later, our finite-sample guarantees will be directly stated in terms of their estimation errors.

Structure-agnostic learning bounds for DR covariate shift adaptation

This section aims to develop finite-sample structure-agnostic learning guarantees for the doubly-robust (DR) estimator. We first state the standing assumptions, introduce the complexity measure utilized in our analysis, and finally present a high-probability bound on the excess $\mathbb{Q}$-risk of the DR estimator (ref) together with a concrete illustration based on classes of Frobenius-norm-bounded neural networks.

In this section, we consider the structure-agnostic perspective that treat the given pilot estimates $\big( \hat{\rho}, \hat{f}_0 \big)$ as black-boxes; our bounds depend only on their estimation errors measured by the mean-squared error with respect to $\mathbb{P}_{X}$.

Assumptions

We begin by introducing the minimal assumptions under which our non-asymptotic analysis holds.

Uniform convergence and Rademacher complexity guarantees

Now, we turn our attention to analysis of the DR estimator (ref) in finite-sample regimes based on uniform convergence arguments. The key complexity measure is the Rademacher complexity of the $f^*$-shifted version of the function class $\mathcal{F} \subseteq \left( \mathbb{X} \to \mathbb{R} \right)$: \[ \mathcal{F}^* := \left\{ f - f^* : f \in \mathcal{F} \right\} \subseteq \left( \mathbb{X} \to \mathbb{R} \right). \] We first recall the definition of the Rademacher complexity for completeness.

With these preliminary notions in hand, one can state a structure-agnostic high-probability upper bound on the excess $\mathbb{Q}$-risk of the DR estimator (ref) that depends only on the $L^2$-errors of the pilot estimates and the Rademacher complexity of $\mathcal{F}^*$ under $\mathbb{P}_X$ and $\mathbb{Q}_X$.

The proof of Theorem (ref) is deferred to Appendix (ref). Let us make use of the notation \[

\] The leading bias term in (ref) can be rewritten as the product $\textnormal{Err}_{\rho} \cdot \textnormal{Err}_{f}$. This key observation leads to the following two concrete implications:

Since we have trivial bounds $\mathcal{R}_{n_{\mathbb{P}}}^{\mathbb{P}_X} \left( \mathcal{F}^* \right) \leq 2$ and $\mathcal{R}_{n_{\mathbb{Q}}}^{\mathbb{Q}_X} \left( \mathcal{F}^* \right) \leq 2$, one can simplify the excess $\mathbb{Q}$-risk bound (ref) in Theorem (ref) of the DR estimator as follows: With Assumptions (ref)--(ref) in hand, it follows that the DR estimator (ref) achieves

with probability at least $1 - \delta$.

An illustration with Frobenius-norm-bounded neural networks

Let $\mathbb{X} \subseteq \mathbb{R}^{n_0}$ be a bounded domain such that $\sup \left\{ \left\| \mathbf{x} \right\|_{2} : \mathbf{x} \in \mathbb{X} \right\} \leq R$ for some radius $R \in \left( 0, +\infty \right)$. We also consider a collection of $1$-Lipschitz activation functions $\left\{ \sigma_j \in \left( \mathbb{R} \to \mathbb{R} \right) : j \in \mathbb{N} \right\}$ that are positive-homogeneous (i.e., $\sigma_{j} \left( \alpha t \right) = \alpha \sigma_j (t)$ for any $\left( \alpha, t \right) \in \mathbb{R}_{+} \times \mathbb{R}$), and that are applied element-wise. We are mainly interested in a class of real-valued neural networks of depth $d \in \mathbb{N}$ over the domain $\mathbb{X} \subseteq \mathbb{R}^{n_0}$ defined as

where $\boldsymbol{\theta} = \left( \mathbf{W}_1, \cdots, \mathbf{W}_d \right) \in \prod_{j=1}^{d} \mathbb{R}^{n_{j} \times n_{j-1}}$ denotes the model parameter consists of $d$ parameter matrices with $n_d = 1$, and the real-valued neural network $\textsf{NN}_{d} \left( \cdot; \boldsymbol{\theta} \right): \mathbb{X} \to \mathbb{R}$ of depth $d$ is defined to be

Here, $M_{\textsf{F}} : [d] \to \mathbb{R}_{+}$ specifies upper bounds on the Frobenius norm of parameter matrices, and the parameter space $\Theta \left( M_{\textsf{F}} \right) \subseteq \prod_{j=1}^{d} \mathbb{R}^{n_{j} \times n_{j-1}}$ is given by \[

\] A prominent example of the above construction are ReLU networks, where every $\sigma_{j}: \mathbb{R} \to \mathbb{R}$ corresponds to applying the ReLU activation function $\sigma (\cdot) := \max \left\{ 0, \cdot \right\}: \mathbb{R} \to \mathbb{R}_{+}$. Armed with the class $\mathcal{H}_d \left( \mathbb{X}; M_{\textsf{F}} \right)$, let us now introduce the function class of our interest. Let $\eta : \mathbb{R} \to \left[ -1, 1 \right]$ be an $L$-Lipschitz bounded activation function such that $\eta (0) = 0$, and define

For example, the inverse tangent activation function $\frac{2}{\pi} \arctan (\cdot) : \mathbb{R} \to \left[ -1, 1 \right]$ satisfies the desired properties with $L = \frac{2}{\pi}$. One can show that the Rademacher complexity of the $f^*$-shift version of (ref) with respect to any probability measure $\mu \in \Delta (\mathbb{X})$ is of order $\mathcal{O} \left( \frac{1}{\sqrt{n}} \right)$.

The proof of Proposition (ref) can be found in Appendix (ref). With Proposition (ref) in hand, one may conclude that the DR estimator (ref) achieves the following excess $\mathbb{Q}$-risk bound when we select the hypothesis class $\mathcal{F} \subseteq \left( \mathbb{X} \to \left[ -1, 1 \right] \right)$ of our interest as (ref): with probability at least $1 - \delta$, one has

Learning bounds for DR covariate shift adaptation: parametric models

This section closely examines the doubly-robust (DR) covariate shift adaptation when the underlying function class is finite-dimensional and well-specified. Our central takeaway messages are two-fold: (i) with parametric models, fast $1/n$-type rates of convergence are attainable without assuming exact knowledge of the covariate density ratio $\rho^*$; and (ii) the DR estimator achieves these rates regardless of the statistical accuracies of the pilot estimates $\big( \hat{\rho}, \hat{f}_0 \big)$.

\paragraph{Parametric model.} Throughout this section, we impose Assumptions (ref) and (ref) and consider a $d$-dimensional parameterization

with the ground-truth parameter $\boldsymbol{\theta}^* \in \Theta$ such that $f^* (\cdot)= f \left( \cdot; \boldsymbol{\theta}^* \right) \in \mathcal{F}$. For any pilot estimates $\big( \hat{\rho},\hat{f}_0 \big)$, the DR empirical risk specialized to the parameterized model (ref) is

where $\ell (a, b) := (b-a)^2$. We define the DR estimator specialized to the parametric model (ref) as

\paragraph{Regularity and landscape conditions.} We first make the following smoothness assumptions customary in classical analysis of MLE lehmann1999elements, le1956asymptotic, cramer1999mathematical, van2000asymptotic, lehmann2006theory.

Here, we note that Assumption (ref) is satisfied, for example, if the population version of the DR empirical risk (ref) is strongly convex in an open neighborhood of $\boldsymbol{\theta}^* \in \Theta$, and the Hessian of the DR empirical risk (ref) uniformly concentrates on that neighborhood.

Now, we are ready to establish an improved structure-agnostic learning bound for the DR estimator (ref) for parametrized hypothesis classes (ref), which leads to faster rates of convergence. For convenience, let us first recall the classical notion of Fisher information, which plays a critical role as a key quantity to measure the difficulty of parameter estimation. The $\mu$-Fisher information matrix evaluated at $\boldsymbol{\theta} \in \Theta$ is defined as

where $\mu \in \left\{ \mathbb{P}, \mathbb{Q} \right\}$ and $\ell : \mathbb{R} \times \mathbb{R} \to \mathbb{R}_{+}$ is the squared error loss. Then, one can easily observe that

where $\mu_{X} (\cdot) := \mu \left( \cdot \times \mathbb{R} \right) \in \Delta (\mathbb{X})$ refers to the covariate marginal distribution of $\mu \in \left\{ \mathbb{P}, \mathbb{Q} \right\}$. We now present our main result of this section, whose proof is deferred to Section (ref):

\paragraph{Interpretations & key implications} Theorem (ref) shows that, for well-specified parametric models, the DR estimator (ref) achieves a fast and instance-dependent upper bound on the excess $\mathbb{Q}$-risk that decouples the contributions of the source and target samples to the bound: \[ \frac{\textsf{Trace} \left\{ \mathcal{I}_{\mathbb{P}} \left( \boldsymbol{\theta}^* \right) \mathcal{I}_{\mathbb{Q}}^{-1} \left( \boldsymbol{\theta}^* \right) \right\}}{n_{\mathbb{P}}} : \textnormal{contributed by the source data}, \quad \frac{d}{n_{\mathbb{Q}}} : \textnormal{contributed by the target data}, \] up to logarithmic factors. Here, the trace factor $\textsf{Trace} \big\{ \mathcal{I}_{\mathbb{P}} \big( \boldsymbol{\theta}^* \big) \mathcal{I}_{\mathbb{Q}}^{-1} \big( \boldsymbol{\theta}^* \big) \big\}$ quantifies the Fisher information mismatch between $\mathbb{P}$ and $\mathbb{Q}$, and is the only way in which covariate shift affects the leading constant. Notably, the excess $\mathbb{Q}$-risk bound in Theorem (ref) holds without access to the ground-truth covariate density ratio $\rho^*$, and is independent of the statistical accuracies of the pilot estimates $\big( \hat{\rho}, \hat{f}_0 \big)$.

We also discuss some appealing attributes of DR covariate shift adaptation and its fast $1/n$-type convergence guarantee (ref) for well-specified parametric models provided in Theorem (ref):

It would be worth pointing out the trace factor $\textsf{Trace} \big\{ \mathcal{I}_{\mathbb{P}} \big( \boldsymbol{\theta}^* \big) \mathcal{I}_{\mathbb{Q}}^{-1} \big( \boldsymbol{\theta}^* \big) \big\}$, which is different from the trace factors that appears in the excess $\mathbb{Q}$-risk bounds for the vanilla MLE and the weighted MLE of ge2024maximum. However, on the closer look, ge2024maximum assumes the boundedness of the covariate density ratio $\rho^*$, under which their excess $\mathbb{Q}$-risk bound for the weighted MLE (see Theorem 5.2 therein) can be translated to the same trace factor as in the bound (ref) of Theorem (ref).

Discussion

This paper establishes the first finite-sample guarantees for doubly-robust (DR) covariate shift adaptation, complementing the prior asymptotic analysis kato2023double and clarifying the role of pilot estimates, sample allocation, and parametric modeling for the Bayes regression function. The structure-agnostic upper bound (ref) of the DR estimator (ref) shows that the leading bias term scales as the product of statistical error rates for the pilot estimates, providing a non-asymptotic demonstration of the celebrated double robustness phenomenon robins1995semiparametric, robins2008higher: one consistent pilot estimate suffices to obtain the consistency of the one-step corrected estimators, and the joint improvement leads us to multiplicative gains. The decomposition of the DR empirical risk (ref) underscores how the labeled source samples primarily benefit the pilot regression model, while the unlabeled target covariates strengthen the effect of the pilot estimate for the covariate density ratio, offering practical guidance on data collection under budget constraints in the target domain. Within well-specified parametric models, our analysis of the DR estimator (ref) via modern techniques for finite-sample analysis of MLE yields a non-asymptotic fast $1/n$-type convergence guarantee, which is independent of the statistical accuracies of pilot black-box estimates. In this result, the difficulty of learning a predictive model under covariate shift is quantified by the Fisher information mismatch term between the source and target distributions. Together, the findings in this paper demonstrate that the DR covariate shift adaptation combines asymptotic efficiency results with strong finite-sample out-of-distribution generalization bounds.

Acknowledgements

Jeonghwan Lee is partially supported by the Doctoral Overseas Scholarship from the Kwanjeong Educational Foundation. Cong Ma is partially supported by the National Science Foundation via grant DMS-2311127 and the CAREER Award DMS-2443867.