Extracted main text — title through conclusion, appendix excluded. This is what our citation measures are computed over, published so the extraction can be checked by eye.
47,143 characters · 18 sections · 47 citation commands
Double Debiased Covariate Shift Adaptation Robust to Density-Ratio Estimation
\def\spacingset#1{ {#1}} \spacingset{1}
{\it Keywords:} Covariate shift adaptation, density-ratio estimation, double machine learning, semiparametric analysis, transfer learning
\spacingset{1.45}
Suppose that we have access to train data with both covariates and outcomes, while test data only contains covariates. Under a covariate shift, covariate distributions are different between the train and test data. In this setting, our goal is to predict the missing outcomes of the test data, and our interest lies in training regression models robust against a covariate shift\footnote{Our method can also be applied to binary classification by estimating logistic regression models. See Remark (ref).}.
One of the golden rules in prediction is to train regression models by minimizing an empirical risk that approximates the population risk defined over the target test distribution. However, under the covariate shift, the distribution of the observable training data deviates from that of the test data, making it difficult to approximate the empirical risk of the test data from the training data.
The seriousness of the covariate shift problem depends on the correctness of the model specification. We say regression models are correctly specified if the error terms are independent of covariates under both train and test distributions. When regression models are correctly specified, we can still obtain a regression model that asymptotically minimizes the test-data population risk by minimizing a train-data empirical risk. This is because the population risk minimizers of the train and test data are the same if the models are correctly specified. In contrast, when regression models are misspecified, their predictive performance can be severely deteriorated Shimodaira2000. Thus, this issue has catalyzed interest in regression methods robust to covariate shift and model misspecification across various disciplines, including machine learning and statistics.
One of the popular approaches for a covariate shift is importance weighting Shimodaira2000,sugiyama2008direct. Specifically, Shimodaira2000 suggests training regression models by weighing the loss with the ratio of covariate densities between the train and test sets. Here, the weighted average loss over the train data converges to the population risk on the test data. Consequently, we can train the regression model by minimizing an estimate of the test-data risk.
Given that the density ratio is typically unknown, subsequent studies have proposed estimating it from observations sugiyama2008direct. By using density-ratio estimation methods, existing importance weighting methods adopt a two-stage approach: initially estimating the density ratio and then minimizing the estimates of the test-data population risk.
While existing studies report successes of covariate shift adaptation using an estimated density ratio, the performance significantly depends on the accuracy of the density ratio estimation. If the density ratio estimator does not converge to the true density ratio, we cannot obtain regression models that minimize the test-data risk. Even if the density ratio is correctly estimated, the convergence rate of the density ratio estimator affects that of the regression models of interest. If the regression models are parametric models, our interest may lie in the $\sqrt{n}$-convergence rate for the parameters. However, the $\sqrt{n}$-convergence rate may not be obtained because the convergence rate of the density ratio is slower than $\sqrt{n}$. These issues motivate us to develop a covariate shift adaptation method robust to a failure of the density ratio estimation and reduce the bias incurred from the density ratio estimation.
To tackle these issues, we develop a doubly robust (DR) estimator for covariate shift adaptation, which utilizes both density-ratio and conditional expected outcome estimators. First, this estimator is robust to inconsistency of the density ratio; that is, either the density-ratio and conditional expected outcome estimators are consistent, the DR estimator is also consistent. Additionally, to reduce the bias caused by the estimation error of the density ratio, we apply the double machine learning (DML) method proposed by ChernozhukovVictor2018Dmlf. Following this method, we first split observations into subgroups and estimate the density ratio and conditional expected outcomes within them. Then, we add nonparametrically estimated conditional expected outcomes to the conventional empirical test-data risk estimator using the estimated density-ratio function. This procedure of DML enables us to show the asymptotic normality of the DR estimator by achieving a fast convergence rate for estimators of the regression model's parameters.
\paragraph{Organization.} The remainder of the paper is organized as follows. In Section (ref), we define the problem-setting. In Section (ref), we discuss properties of covariate shift adaptation with importance weighting using an estimated density ratio. As we will discuss in Section (ref), when using an estimated density ratio, it is not easy to prove the asymptotic normality of the estimator given by covariate shift adaptation with importance weighting. Therefore, in Section (ref), we introduce a new estimator for covariate shift adaptation, which uses a DR structure and DML. We introduce related work in Section (ref) and Appendix (ref). In Section (ref), we show the performance of the proposed method using real datasets.
We formulate our regression problem. Let $\big(Y, X\big)$ and $\big(\widetilde{Y}, \widetilde{X}\big)$ be two pairs of an outcome and covariates, where $Y, \widetilde{Y} \in\mathbb{R}$, and $X, \widetilde{X}\in\mathcal{X}\subset \mathbb{R}^d (d\geq 1)$. We refer to $\big(Y, X\big)$ as the train data and $\big(\widetilde{Y}, \widetilde{X}\big)$ as the test data. While we can observe $\big(Y, X\big)$ as a pair, we only observe $\widetilde{X}$ for the test data. Our goal is to predict $\widetilde{Y}$ by using realizations of $\big(Y, X\big)$ and $\widetilde{X}$, which is an observable counterpart of $\widetilde{Y}$. We refer to this setting as covariate shift Shimodaira2000. The detailed formulations are shown below.
In our setting, we observe $\{(Y_i, X_i)\}^{n}_{i=1}$ and $\{\widetilde{X}_j\}^{m}_{j=1}$, which are independently and identically distributed (i.i.d.) copies of $\big(Y, X\big)$ and $\widetilde{X}$. They are two stratified observations; that is, for fixed sample sizes $n$ and $m$, the two datasets are independently generated Imbens1996,Wooldridge2001.\footnote{As a different setting, we can consider a situation where there are $n_0$ samples, and following a Bernoulli distribution, samples are classified into two datasets with $n$ and $m$ sample sizes. In this setting, $n$ and $m$ are random variables. We do not consider this setting in our study.} Following the terminology of machine learning, we call $\{(Y_i, X_i)\}^{n}_{i=1}$ the (labeled) train data and $\{\widetilde{X}_j\}^{m}_{j=1}$ the (unlabeled) test data.
For each pair of potential outcomes $\big(Y, X\big)$ and $\big(\widetilde{Y}, \widetilde{X}\big)$, we assume the data-generating process as
where $p(y, x)$ is the joint probability density function of $(Y, X)$ and $q(y, x)$ is the joint joint probability density function of $\big(\widetilde{Y}, \widetilde{X}\big)$.
For $\big(Y, X\big)$ and $\big(\widetilde{Y}, \widetilde{X}\big)$, we make the following boundedness assumption.
We also assume that the probability density functions of $Y$ conditioned on $X$ and $\widetilde{Y}$ conditioned on $\widetilde{X}$ are the same; that is,
where $w(y| x)$ is the conditional probability density function of $Y$ (resp. $\widetilde{Y}$) given $X$ (resp. $\widetilde{X}$), $p(x)$ and $q(x)$ are probability density functions of $X$ and $\widetilde{X}$, respectively. Here, let us define the density ratio $r(x) := \frac{q(x)}{p(x)}$, and we assume the overlap between the supports of $X$ and $\widetilde{X}$.
Furthermore, we denote the conditional expectation function $\mathbb{E}[Y|X = x]$ as
where $\mathbb{E}[Y|X = x]$ takes an expectation with respect to the density $w(y|x)$. Here, the following hold:
where $\varepsilon$ and $\widetilde{\varepsilon}$ are error terms such that $\mathbb{E}[\varepsilon | X ] = 0$ and $\mathbb{E}[\widetilde{\varepsilon} | \widetilde{X}]$ almost surely.
To investigate the relationship between $Y_i$ and $X_i$, we model the conditional expectation $f_0(x)$ by a parametrized regression model $g_{\bm{\beta}}$ with a $k$-dimensional parameter $\bm{\beta}$, and define a set of functions as $\mathcal{G}=\{g_{\bm{\beta}}(x): \bm{\beta}\in\Theta\subset \mathbb{R}^k\}$, where $\Theta$ is some compact parameter space. Then, we regress $X$ on $Y$ to determine a parameter $\bm{\beta}$. For the train and test data, let us define the mean squared error minimizers as
where $\mathbb{E}_p$ and $\mathbb{E}_q$ denote expectations over $p(y, x)$ and $q(y, x)$. To identify the parameters in estimation, we assume uniqueness of $\widetilde{\bm{\beta}}_0$ and differentiability of $g_{\bm{\beta}}$ with respect to $\bm{\beta}$.
Note that when the optimal parameter $\beta_0$ is unique, the corresponding asymptotic distribution is also unique.
We consider two scenarios for the regression problem. In the first scenario, a regression model is correctly specified; that is, $\mathcal{G}$ includes the regression function $f_0(x) = \mathbb{E}[Y|X=x]$. In the second scenario, $\mathcal{G}$ does not include the regression function $f_0(x) = \mathbb{E}[Y|X=x]$. We say that regression models are correctly specified in the former scenario, and regression models are misspecified in the latter scenario.
\paragraph{Models are correctly specified.} If $f_0 \in \mathcal{G}$, then it holds that $\bm{\beta}_0 = \widetilde{\bm{\beta}}_0$, $f_0 = g_{\bm{\beta}_0} = g_{\widetilde{\bm{\beta}}_0}$, and
In this scenario, we can train a model that asymptotically minimizes the test-data risk by minimizing an empirical risk over the train data; that is, without covariate shift adaptation, we can consistently estimate the risk minimizer for the test-data risk.
\paragraph{Models are misspecified.} If $f_0 \notin \mathcal{G}$, then it may not hold that $\bm{\beta}_0 = \widetilde{\bm{\beta}}^*$ and $f_0 = g_{\bm{\beta}_0} = g_{\widetilde{\bm{\beta}}^*}$. If $g_{\bm{\beta}_0} \neq g_{\widetilde{\bm{\beta}}^*}$, then it implies that $g_{\bm{\beta}_0}$ may not perform well for data generated from $q(y, x)$. In this scenario, it is desirable to approximate the test-data risk and train regression models by minimizing it.
Thus, the problem of a covariate shift is serious when models are misspecified. To address this issue, Shimodaira2000 proposes the following covariate shift adaptation via importance weighting.
Our goal lies in predicting a test response variable $\widetilde{Y}_j$, an unobserved counterpart of $\widetilde{X}_j$, based on the parametric regression models estimated from $\{(Y_i, X_i)\}^{n}_{i=1}$ and $\{\widetilde{X}_j\}^{m}_{j=1}$. The problem of predicting a test response variable $\widetilde{Y}_j$ under covariate shift ($p(x) \neq q(x)$) and model misspecification is referred to as covariate shift adaptation.
As we review in the following section, a typical approach for this problem is to employ importance weighting Shimodaira2000. However, while this approach allows us to adapt a covariate shift, it is vulnerable to a failure of density ratio estimation, and its estimator has a slow convergence rate owing to the estimation error of the density ratio.
In this study, we propose a method for covariate shift adaptation that is robust to a failure of density ratio estimation and yields an estimator with a faster convergence rate. We refer to our approach as DR covariate shift adaptation because it returns a consistent estimator if one of the two functions in the estimator is correctly specified, as explained in Section (ref).
For obtaining the risk minimizer under the covariate shift, importance weighting with the density ratio has been used Shimodaira2000,Reddi2015.
To estimate the parameter $\widetilde{\bm{\beta}}_0$, which is the minimizer of the prediction risk under the overate shift, we face the problem that we do not have access to the labeled data and cannot obtain a sample approximation of the test-data population risk $\mathcal{R}(\bm{\beta}) := \mathbb{E}_{q}\left[(Y-g_{\bm{\beta}}\left(X\right))^2\right]$.
Let us define the density ratio between $q$ and $p$ as \[r_0(x) = \frac{q(x)}{p(x)}\] Shimodaira2000 shows that we can obtain the population risk for the test distribution using the following weighted squared risk for the training distribution:
Then, the risk minimizer under covariate shift adaptation with importance weighting is given as
Since we do not know the true density ratio $r_0$, we need to estimate it. A naive approach is to estimate the two densities of the numerator $q(x)$ and denominator $p(x)$ separately, then take the ratio of the estimated densities. For instance, we can separately estimate the two probability densities by kernel density estimation. However, this approach may not work well due to the division between the two estimated densities, which magnifies the estimation errors.
For example, this study employs Least-Squares Importance Fitting (LSIF) in experimental studies, which uses the squared error to estimate the density ratio Kanamori2009. The reason for this choice is that there is an algorithm called unconstrained Least-Squares Importance Fitting (uLSIF) with a computational advantage. We can obtain the closed-form solution just by solving the linear equations. By using the uLSIF, given a hypothesis class $\mathcal{H}$, we obtain $\widehat{r}_{n, m}$, an estimator of $r_0$, by
where $\mathcal{R}:\mathcal{H}\to \mathbb{R}^+$ is a regularization term. Then, we define a regression parameter estimator as
The convergence rates of density ratio estimators are shown by existing studies, such as Sugiyama2012, Kanamori2012, Kato2020nnbr, and zheng2022an. For example, when using kernel regression, the convergence rate is given as follows Kanamori2012:
where $0 < \gamma < 1$ is a constant depending on the functional form of the density ratio. These results imply that the estimation error remains in the estimation of $\mathbb{E}_{q}\left[(Y-g_{\bm{\beta}}\left(X\right))^2\right] = \mathbb{E}_{p}\left[(Y-g_{\bm{\beta}}\left(X\right))^2r_0(X)\right]$ as
We typically consider $\sqrt{n}$-consistency for estimators of $\bm{\beta}_0$. However, this result implies that in the estimation of $\mathbb{E}_{q}\left[(Y-g_{\bm{\beta}}\left(X\right))^2\right]$, there remains non-negligible bias term that does not disappear, which remains in the estimation of $\bm{\beta}_0$.
In the following sections, we propose our estimator that attains (i) covariate shift robustness under misspecified models and (ii) asymptotic normality, which implies $\sqrt{n}$-consistency.
When estimating the density ratio with modern machine learning methods, the estimator often violates the Donsker condition. To show the asymptotic normality of an estimator of $\bm{\beta}_0$ with such a nuisance estimator, we introduce DML ChernozhukovVictor2018Dmlf, which employs a sample-splitting technique called cross-fitting klaassen1987,ZhengWenjing2011CTME.
For brevity, without loss of generality, let $n$ and $m$ be even numbers, and we use two-fold cross-fitting. We can extend our method to a case with $L$-fold cross-fitting. For $n_{(1)} = n_{(2)} = n/2$ and $m_{(1)} = m_{(2)} = m/2$, we randomly separate the samples $\{(Y_i, X_i)\}^{n}_{i=1}$ into two subsets $D_{(1)} = \{(Y_{(i, 1)}, X_{(i, 1)})\}^{n_{(1)}}_{i=1}$ and $D_{(2)} = \{(Y_{(i, 2)}, X_{(i, 2)})\}^{n_{(2)}}_{i=1}$ with sample sizes $n_{(1)}$ and $n_{(2)}$; similarly, we separate samples $\{\widetilde{X}_j\}^{m}_{j=1}$ into two subsets $\widetilde{D}_{(1)} = \{\widetilde{X}_{(j, 1)}\}^{m_{(1)}}_{j=1}$ and $\widetilde{D}_{(2)} = \{\widetilde{X}_{(j, 2)}\}^{m_{(2)}}_{j=1}$ with sample sizes $m_{(1)}$ and $m_{(2)}$.
First, by using $\{(Y_{(i, 2)}, X_{(i, 2)})\}^{n_{(2)}}_{i=1}$ and $\{\widetilde{X}_{(j, 2)}\}^{m_{(2)}}_{j=1}$, we estimate $f_0$ and $r_0$, respectively. We estimate them to satisfy Assumption (ref). Let $\widehat{f}_{(-1)}$ and $\widehat{r}_{(-1)}$ be the estimators.
Then, by using $\{(Y_{(i, 1)}, X_{(i, 1)})\}^{n_{(1)}}_{i=1}$ and $\{\widetilde{X}_{(j, 1)}\}^{m_{(1)}}_{j=1}$, we obtain a sample approximate of the risk as
Similarly, we define another empirical risk as
We show the pseudo-code in Appendix (ref).
By minimizing the two empirical risks, we obtain the DR estimator $\widehat{\bm{\beta}}^{\mathrm{DR}}$ for $\bm{\beta}_0$; that is,
We can also estimate $f_0(x)$ by $\widehat{f}^{\mathrm{DR}}(x) = g_{\widehat{\bm{\beta}}^{\mathrm{DR}}}(x)$.
Let us assume $m=\rho n$ for a fixed $\rho > 0$ and consider asymptotics when $n \to \infty$.
First, we consider the first order condition of Eq. ((ref)), which is given as $h(\widehat{\bm{\beta}}^{\mathrm{DR}}) = \bm{0}$, where $h(\widehat{\bm{\beta}}^{\mathrm{DR}})$ is a $k$-dimensional vector defined as
where $\dot{g}_{\bm{\beta}}(X_i) = \frac{\partial g_{\bm{\beta}}(X_i)}{\partial \bm{\beta}}$. We also define
Then, we make the following assumption.
We make the following assumptions for the convergence rates of estimators of nuisance parameters.
For example, the uLSIF with kernel ridge regression satisfies this convergence rate Kanamori2012. Density-ratio estimators with neural networks under Bregman-divergence minimization also satisfy this convergence rate Kato2020nnbr.
Note that under Assumptions (ref) and (ref), $D^\top = \mathbb{E}_q\left[ \dot{g}_{\bm{\beta}}(\widetilde{X})\dot{g}^\top_{\bm{\beta}}(\widetilde{X})\right]$ holds.
In DML, the combination of Assumption (ref) and the cross-fitting allows us to reduce the bias incurred from the estimation error of $r_0$. This is because we can make the convergence rate of $\left\|\widehat{r}_{(-\ell))} - r_0\right\|_2\left\|\widehat{f}_{(-\ell)} - f_0\right\|_2$ faster than $\left\|\widehat{r}_{(-\ell))} - r_0\right\|_2$ due to $\left\|\widehat{f}_{(-\ell)} - f_0\right\|_2$.
Then, we show the asymptotic distributions as follows. The proof is shown in Appendix (ref).
From this theorem and the Taylor expansion, we can obtain the following point-wise asymptotic normality for $\widehat{f}^{\mathrm{DR}}(x) = g_{\widehat{\bm{\beta}}^{\mathrm{DR}}}(x)$.
Our proposed covariate shift adaptation is DR in the sense that if either the density ratio or the conditional expectation is consistently estimated, the parameter $\widetilde{\bm{\beta}}_0$ is also consistently estimated.
Let us define another risk estimator as follows:
which replaces $\widehat{f}_{(-\ell)}$ and $\widehat{r}_{(-\ell)}$ with $f^\dagger$ and $r^\dagger$, respectively. Then, we have
Using this equation, we confirm that if either $f_0$ or $r_0$ is consistently estimated, we can consistently estimate the test-data population risk $\mathcal{R}(\bm{\beta})$ as shown below. Then, we can consistently estimate $\widetilde{\bm{\beta}}_0$ by minimizing the estimate of $\mathcal{R}(\bm{\beta})$.
\paragraph{Case (i): $f^\dagger = f_0 = g_{\bm{\beta}_0}$.} If $f^\dagger = f_0 = g_{\bm{\beta}_0}$, then $\bm{\beta}_0 - \widehat{\bm{\beta}}^{\dagger} \approx 0$ holds, where we used
where $C > 0$ is a constant that is irrelevant to $\bm{\beta}$. Here,
holds. Note that $r^\dagger$ is some function that may not be $r_0$. Therefore, we can adapt a covariate shift if we consistently estimate $f_0$ and cannot estimate $r_0$ well.
\paragraph{Case (ii): $r^\dagger = r_0$.} Additionally, if $r^\dagger = r_0$, then
holds, where we used the following two equations:
Thus, if either the density ratio or the conditional expectation is consistently estimated, we can consistently estimate the test-data population risk.
{
}
In the previous subsection, to estimate $f_0$ by $g_{\bm{\beta}}$, we assume that another estimator of $f_0$ is obtainable with a certain convergence rate. However, covariate shift adaptation mainly considers a situation where we cannot train high-dimensional, nonparametric, or other complicated models, including neural networks and random forests. This is because covariate shift adaptation is a method for addressing model misspecification, and if we use such models with large $\mathcal{G}$, we may not encounter the problem of model misspecification. Therefore, in this subsection, for a situation where we cannot use another model for estimating $f_0$, we propose another estimator with asymptotic normality only using a model of interest $g_{\bm{\beta}}$ and a density ratio model as a variant of the previously defined DR estimator.
We define the following self-debiased (SDB) estimator: \[\widehat{\bm{\beta}}^{\mathrm{SDB}} := \operatorname*{arg\,min}_{\bm{\beta} \in \Theta}\sum_{\ell\in\{1, 2\}}\widehat{\mathcal{R}}^{\mathrm{SDB}}_{(\ell)}(\bm{\beta}),\] where
and $\widehat{\bm{\beta}}_{(-\ell)} := \operatorname*{arg\,min}_{\bm{\beta} \in \Theta}\frac{1}{n_{(-\ell)}}\sum^{n_{(-\ell)}}_{i=1} \big(Y_{(i, \ell)} -g_{\bm{\beta}}\left(X_{(i, \ell)}\right)\big)^2$; that is, we replace $\widehat{f}_{(-\ell)}$ in the DR estimator with $g_{\widehat{\bm{\beta}}_{(-\ell)}}$. We refer to this estimator as SDB estimator because it conducts DML by using a model of interest $g_{\widehat{\bm{\beta}}_{(-\ell)}}$ itself.
Here, we make the following assumption.
Then, we obtain the following corollary.
This estimator does not have the DR property; that is, we can adapt a covariate shift only when $r_0$ is consistently estimated. In this sense, the property is the same as the conventional covariate shift adaptation. However, unlike it, the SDB estimator reduces the bias caused by the density ratio estimator, and we can show the fast $\sqrt{n}$-convergence rate.
Covariate shift adaptation is widely studied in machine learning and statistics, primarily motivated by addressing model misspecification. A pivotal work is by Shimodaira2000, as discussed in our Section (ref). They advance covariate shift adaptation by introducing an importance weighting method for parametric models under model misspecification. Building upon this, sugiyama2008direct proposes a direct density ratio estimation method for covariate shift adaptation, drawing from Shimodaira2000's formulation.
While Reddi2015 also contemplates DR covariate shift adaptation, their approach diverges from ours. Although they also refer to their method as the DR method, leading to potential naming overlaps, our estimator is traditionally dubbed the DR estimator within semiparametric inference literature. Thus, to maintain consistency, we also refer to our estimator as the DR estimator despite the risk of confusion.
We introduce applications of covariate shift adaptation. Sugiyama2006 and Sugiyama2009 leverage covariate shift adaptation combined with importance weighting to address active learning in the presence of model misspecification. Sugiyama2005 suggests a model selection technique under such conditions. Furthermore, Yamada2010 distinguishes speakers under a covariate shift, matsui2018variable utilizes covariate shift adaptation for medical datasets, KatoUehara2020 executes off-policy evaluation, and Kato2020nnbr classify text data. ramchandran2021ensembling discusses appropriate models under a covariate shift. Batch-normalization can be justified as a covariate shift adaptation Sergey2015. For the benchmark datasets, see koh2021wilds. For the inference, Tibshirani2019 proposed conformal inference under a covariate shift.
Various approaches of direct density ratio estimation have been explored without undergoing density estimation Sugiyama2012. These approaches include the moment matching Huang2007,Gretton2009, the probabilistic classification Qin1998,Cheng2004, the density matching Nguyen2010, and the density-ratio fitting Kanamori2009. Sugiyama2011bregman shows that these methods could be generalized as density ratio matching under the Bregman divergence Bregman1967. Recently, to estimate the density ratio with high-dimensional models, Rhodes2020 proposes the telescoping density-ratio estimation, and Kato2020nnbr proposes using the non-negative Bregman divergence.
This section provides simulation studies to verify the soundness of our proposed method. We compared our method (DR (CF)) with the OLS, weighted least squares for covariate shift adaptation with importance weighting and estimated density ratio by uLSIF Kanamori2009 (WLS), and nonparametric regression by the Kernel ridge regression (NP).
For $X$, let $d = 3$, and its first element be the constant term. Let $\bm{\beta} = (\beta_0\ \beta_1\ \cdots \ \beta_k)$ be parameters, and $\bm{\beta}_0 = (\beta_{0, 0}\ \beta_{1, 0}\ \cdots \ \beta_{k, 0})$ be the true parameters. For $x = (1\ x_1\ x_2)^\top$, we conduct simulation studies for the following two regression functions:
In Model 1, we use $\beta_{0, 0} = \beta_{1, 0} = \cdots = \beta_{5, 0}$ and $\varepsilon^d$ follows the standard normal distribution. In Model 2, we use $(\beta_{0, 0}, \beta_{1, 0}, \beta_{2, 0}) = (0, 2, 3)$, and $Y_i$ is generated from a Bernoulli distribution with probability $f_0(X_i)$.
For each $f_0$ in the OLS, WLS, and DR, we train misspecified (Misspecified) and correctly specified models (Correct). For misspecified models, we use $ g_{\bm{\beta}} = \beta_0 + \beta_1x_1 + \beta_2x_2. $ For correctly specified models, we use the same linear models as $f_0$.
For $X_i = (X_{1, i}\ X_{2, i})^\top$ and $\widetilde{X}_j = (\widetilde{X}_{1, j}\ \widetilde{X}_{2, j})^\top$, we generate $(X_{1, i}\ X_{2, i})^\top$ and $(\widetilde{X}_{1, j}\ \widetilde{X}_{2, j})^\top$ from two-dimensional normal distributions $\mathcal{N}(\bm{\theta}, \Omega)$ and $\mathcal{N}(\widetilde{\bm{\theta}}, \Omega)$, respectively. We use two $\Omega$: $\Omega =
$ and $\Omega =
$, referred to as ``Indep $X$'' and ``Corr $X$'' in Table~\ref{tbl:exp_res}. Here, $\bm{\theta}$ and $\widetilde{\bm{\theta}}$ are generated from a uniform distribution with a support $[-1, 1]$, respectively. We obtain $1,000$ observations and $500$ observations for train and test data. We report the mean squared errors (MSEs) and the standard deviation (parenthesis) in Table (ref). More detailed results and additional experiments are shown in Appendix (ref), including experiments with the SDB estimators.
Our DR outperforms the OLS and the WLS under misspecified models. However, under correctly specified models, the performance is lower than theirs. This deterioration is caused by additional errors by using cross-fitting. Compared to the OLS, WLS, and DML, the NP achieves the lower MSEs, as expected, although we focus on cases where the NP cannot be used (we need to use some restricted models for $\mathcal{G}$). This is because the NP uses a large class for the regression models. Combining Table (ref) with the results in Appendix (ref), we consider that the improvement under misspecified models is mainly due to the existence of nonparametric models in the DR estimator.
In this study, we proposed the DR covariate shift adaptation with importance weighting. Our method is robust to the inconsistency of the density ratio estimation and reduces bias incurred from the estimation error of the density ratio. We show the double robustness and the asymptotic distribution of our proposed estimator. Finally, we confirmed the soundness of our proposed method by simulation studies.