Extracted main text — title through conclusion, appendix excluded. This is what our citation measures are computed over, published so the extraction can be checked by eye.
115,595 characters · 19 sections · 78 citation commands
Econometrics with Pre-Trained Embeddings for Unstructured Data
\etocdepthtag.toc{main}
Keywords: transfer learning, double machine learning, deep learning, average treatment effect, demand estimation
JEL codes: C45, C14, C55, C21
Unstructured data, such as images and text, are increasingly used in empirical economics. In these applications, economists often use off-the-shelf pre-trained deep neural network models trained by computer scientists to extract embeddings and then use those embeddings as generated covariates in a downstream target task.\footnote{In the machine learning literature, embeddings are also commonly referred to as extracted features or learned representations.} Embeddings transform unstructured data into structured numerical representations, typically vectors in a lower-dimensional latent space, that preserve relevant information from the original data. From an econometric perspective, this is a two-step estimation problem (murphy1985estimation,pagan1984econometric). This paper provides sufficient conditions under which this workflow yields valid inference in econometric analyses.
The structured dimensionality reduction can be substantial. A color image is typically represented by the intensity values of its pixels. For example, an image with resolution $224 \times 224$ and three RGB color channels has
raw input values. Similarly, a text document is typically represented as a sequence of tokens drawn from a large vocabulary, corresponding to a vector with more than $30{,}000$ dimensions. These raw vectors are ultra-high-dimensional and lack direct economic interpretation in the sense that individual pixel values or naive aggregations of token vectors generally do not correspond to economically meaningful variables. This makes them difficult to use directly in standard empirical specifications, motivating the use of embeddings as lower-dimensional, structured representations of the same underlying information.
Pre-trained embeddings are increasingly used in empirical economics. In some applications, including embeddings can materially change estimates of parameters of interest, including their sign and statistical significance, relative to specifications that omit them.\footnote{ For example, avivi2024patent uses patent-text embeddings to control application quality when studying gender bias in patent examination. dube2020monopsony estimates the elasticity of task vacancy-filling speed with respect to rewards, using task-description embeddings as controls. bajari2025hedonic, bach2024adventures, and compiani2025demand use product text and image embeddings to control for product quality in hedonic regression and demand estimation. klaassen2024doublemldeep study causal effects with multimodal data. Across these applications, the common assumption is that embeddings summarize economically relevant information in otherwise unstructured inputs and can be used as covariates to control or to impute missing variables. Other applications use product embeddings to measure similarity in product space (magnolfi2025triplet,han2025copyright) and satellite image embeddings to predict local poverty status (jean2016combining). } Since training such representation models from scratch typically requires an extremely large sample and substantial computational resources, economists often rely on pre-trained embeddings as structured summaries of raw inputs.
Despite the empirical popularity of this workflow, its theoretical foundations remain limited. There are two main theoretical difficulties. First, the pre-trained model is usually trained on a different dataset and for a different task than the economic application; thus, it is unclear whether the pre-trained representation is valid for the target task. Second, the embedding function is usually not identified: for example, we can insert a matrix and its inverse between the last hidden layer of a deep neural network and the output layer without changing the output.
To address these difficulties, we first formalize the setup containing a source task for the computer scientist and a target task for the economist. The embedding function is estimated from the source task and then used in the target task. In practice, the economist can access the pre-trained embedding function from the computer scientist via standard deep learning libraries such as PyTorch, but it is difficult or impossible for the economist to observe the source sample used to train the embedding function. Our theory allows for this practical setting by treating the source sample as unobserved for the economist.\footnote{ In the machine learning literature, this is known as a transfer-learning setup in which knowledge learned in the source task is reused in the target task (pan2009survey). }
We then show that two conditions are key to justifying the two-step workflow of using pre-trained embeddings in the target task. First, we assume that the source task and the target task share the same approximately true embedding function by requiring that approximation errors of both tasks are small when a shared embedding function is used for both. This condition formalizes the common (implicit) assumption in the empirical literature that the pre-trained embedding function summarizes the relevant information in the unstructured data for the target task. Second, we impose a transferability condition requiring that a neighborhood of the source task identified set is contained in the corresponding target task neighborhood. If the embedding function is the last hidden layer of a deep neural network and the economist uses a linear model on top of it, the transferability condition requires that the coefficient of the target task lies in the linear span of the source-task coefficients associated with the embedding function in population.
To illustrate the importance of the transferability condition, we compare a partially linear model with unstructured controls under two designs in (ref), where one design has the target nuisance lying in the span of the source tasks and the other design does not. The in-span design produces an estimator centered near the true parameter. By contrast, the out-of-span design exhibits a clear shift, motivating a transferability condition that rules out such non-transferable directions. See (ref) for details of the simulation.
Under these conditions, we show that the convergence rate of transfer learning can be faster than $n^{-1/4}$, which is sufficient for valid inference in the double machine learning (DML) framework of chernozhukov2018double for a size-$n$ target sample. Our convergence rate accounts for randomness in both the source and target samples and depends jointly on the complexities and sample sizes of the source task and the target task. The convergence rate in the target task is determined by the slower of the source-side and target-side rates.
Our theory also provides practical guidance for applied researchers. First, the transferability condition is easier to justify for later hidden layers of a deep neural network than for earlier hidden layers, because earlier hidden layers introduce more complex partial identification issues. Thus, the last hidden layer is preferable as the embedding function if the economist is otherwise indifferent between earlier and last hidden layers. Second, even under the sufficient conditions we impose and when the last hidden layer is selected as the embedding function, the embedding function is still not point identified, and it is only identified up to an invertible linear transformation. Partial identification of the embedding function makes variable selection, such as LASSO, fragile. Instead, we recommend using ridge regression or shallow neural networks for the target model since they are robust to invertible linear transformations of the embeddings.
\paragraph{Related Literature} The literature most closely related to ours can be grouped into three strands. First, our paper contributes to the growing literature on inference with variables generated from unstructured data. fong2021machine, angelopoulos2023prediction, egami2023using, zhang2026debiasing, battaglia2025inference, carlson2025unifying, and ludwig2026large provide a framework to correct bias arising from variables generated from unstructured data in the context of missing data. They typically assume a missing completely at random (MCAR) condition or a known propensity score.\footnote{Except for battaglia2025inference, these papers use a missing-variable-imputation framework to handle missing data in the context of chen2008semiparametric.} As discussed by carlson2025unifying, the known propensity score assumption can be relaxed if we can estimate it with sufficiently fast convergence rates. However, the convergence rate of propensity-score estimators based on unstructured data is not studied in these papers. Beyond missing variable imputation, rambachan2024program, modarressi2025causal, and carlson2025making study settings in which unstructured data are used to construct outcome measures. Our paper departs from this literature by studying a general double machine learning setting in which generated embeddings enter the nuisance learning stage as inputs for both outcome imputation and propensity score estimation.
Second, several recent papers use pre-trained embeddings with theoretical justifications for downstream inference. vafa2025estimating study wage-gap decomposition with foundation models. bach2024adventures show that multimodal embeddings from text and images can improve demand estimation and elasticity analysis. These two papers establish square-root-$n$ consistency for the parameter of interest under high-level conditions. christensen2026unstructured develop a bias correction in demand counterfactuals when product differentiation is proxied by finite-dimensional nuisance parameters reparameterized from the embedding function and economic parameters.\footnote{Such reparameterization is not available for the general DML setup we consider in this paper, which includes (ref).} While the scope of christensen2026unstructured is different from ours, their theory takes convergence of the nuisance parameters as given, and our theory can complement their theory by providing sufficient conditions for the convergence in probability. The closest paper to ours is schulte2025adjustment. They study the estimation of the average treatment effect (ATE) with unstructured confounders and point out the identification issues of the embedding function. However, they ignore source-task estimation error and assume transferability across tasks implicitly. Our paper differs by deriving a general convergence rate theory under explicit transferability conditions and by incorporating source task estimation error. This source task error is essential for valid target task inference.
Third, our transferability condition is related to the transfer learning literature in machine learning. tripuraneni2020theory introduce task diversity as a key condition for learning a shared representation that transfers well to new tasks, and watkins2023optimistic derive optimistic excess-risk bounds for multi-task representation learning under related diversity conditions. The rate in watkins2023optimistic is comparable to our rate, but their theory requires a realizability condition that the risk of the true model is zero, which is hard to satisfy in economic applications. Our paper uses a similar task diversity condition to ensure that the source task representation is valid for the target task, but we provide an econometric characterization of the condition through an identification lens. Moreover, we provide a $n^{-1/4}$ convergence rate for the estimated embedding function, which is directly applicable to econometric inference in the target task. In contrast, the machine learning literature typically studies excess-risk bounds for prediction and often obtains slower convergence rates for the estimated embedding function.
\paragraph{Notation} For deterministic sequences $a_n$ and $b_n$, we write $a_n \lesssim b_n$ if there exists a constant $C>0$ such that $a_n \leq C b_n$ for large $n$, and we write $a_n \asymp b_n$ if $a_n \lesssim b_n$ and $b_n \lesssim a_n$. We define the $L^r(P)$ norm as $\|a\|_{P,r}=\{\sum_{t=1}^T \int |a_t(v)|^r dP(v)\}^{1/r}$ for a function $a:\mathcal{V}\mapsto\mathbb{R}^T$, where $a_t$ is the $t$-th element of $a$, and $\|a\|_{\infty} = \sup_{v\in\mathcal{V}}|a(v)|$ for a function $a:\mathcal{V}\to\mathbb{R}$. For a vector $v\in\mathbb{R}^T$, we define the $\ell^r$ norm as $\|v\|_r=\{\sum_{t=1}^T |v_t|^r\}^{1/r}$, where $v_t$ is the $t$-th element of $v$. For a matrix $A\in\mathbb{R}^{T\times K}$, we denote the $t$-th largest singular value of $A$ by $\sigma_t(A)$, the largest singular value of $A$ by $\sigma_{\max}(A)=\sigma_1(A)$, and the smallest singular value of $A$ by $\sigma_{\min}(A)=\sigma_{\min\{T,K\}}(A)$. We also define the smallest nonzero singular value of $A$ by $\sigma_{\min}^+(A)$. We define the spectral norm (or operator norm) as $\|A\|_{\mathrm{sp}}=\sigma_{\max}(A)$, the Frobenius norm as $\|A\|_F=\sqrt{\sum_{t=1}^T\sum_{k=1}^K A_{t,k}^2}$, and the nuclear norm (or trace norm) as $\|A\|_*=\sum_{t=1}^{\min\{T,K\}}\sigma_t(A)$. We denote the Moore-Penrose pseudo-inverse of a matrix $A\in\mathbb{R}^{T\times K}$ by $A^+\in\mathbb{R}^{K\times T}$. For a square matrix $A\in\mathbb{R}^{T\times T}$, we denote the $t$-th largest eigenvalue of $A$ by $\mu_t(A)$, the largest eigenvalue of $A$ by $\mu_{\max}(A)=\mu_1(A)$, and the smallest eigenvalue of $A$ by $\mu_{\min}(A)=\mu_T(A)$ when all eigenvalues are real.
We first introduce the general DML setup, then four empirical applications, and finally formalize the transfer learning setup for the machine learning components. In the following examples, we consider the setting where the economist has a sample of size $n$ for the target task and the computer scientist provides the pre-trained embeddings $\check{h}(\cdot)$ estimated independently from a different source task. See (ref) for the detailed discussion of the embedding functions.
Let $\psi(W_i;\theta,\gamma,\alpha)$ be a moment function, where $W_i$ is an observation from an i.i.d. sample, $\theta$ is the parameter of interest, and $(\gamma,\alpha)$ is an infinite-dimensional nuisance parameter vector. We consider the moment condition:
where $\theta^*$ is the true value of $\theta$, and $(\gamma^*,\alpha^*)$ is the true value of $(\gamma,\alpha)$. We assume that $\gamma^*$ and $\alpha^*$ are unknown functions of $Z_i$, which is a subvector of $W_i$ containing unstructured or ultra-high-dimensional variables such as images and text. The remaining components of $W_i$ are low-dimensional and structured, such as an outcome variable and a treatment variable.
We assume that the computer scientist provides a pre-trained embedding function $\check{h}(z)$ estimated from a different source task. In this setting, it is common for the economist to estimate the nuisance parameters $(\gamma^*,\alpha^*)$ using the pre-trained embeddings $\check{h}(Z_i)$ as regressors in the target stage models. Our nuisance parameter estimators for $\gamma^*$ and $\alpha^*$ are $\hat{f}_{\gamma}\circ\check{h}(z)$ and $\hat{f}_{\alpha}\circ\check{h}(z)$, respectively, where $\hat{f}_{\gamma}$ and $\hat{f}_{\alpha}$ are the estimated target stage models using the pre-trained embeddings $\check{h}(Z_i)$ as regressors. Here, $\hat{f}_{\gamma}\circ\check{h}(z)$ is the composite function of $\hat{f}_{\gamma}$ and $\check{h}(z)$, defined as $\hat{f}_{\gamma}\circ\check{h}(z)=\hat{f}_{\gamma}(\check{h}(z))$. The DML estimator $\hat{\theta}$ is the solution of the sample moment equation
To use the DML theory, we typically verify three key conditions: (i) Neyman orthogonality with respect to $(\gamma,\alpha)$,\footnote { escanciano2026automatic study locally robust estimation with generated regressors. While their theory suggests that the generated regressors affect inference on the parameter of interest in general, this effect of generated regressors does not arise in our setup. We discuss the relationship to their results in (ref). } (ii) sample splitting, and (iii) the sufficiently fast convergence rate of the nuisance parameters:\footnote{ We implicitly assume that the Neyman orthogonal score function is smooth enough to use a second-order Taylor expansion. See also Assumption 3.2 in chernozhukov2018double and the discussion after that. }
where $\|\cdot\|_{P,2}$ is the $L^2(P)$ norm for the distribution of $W_i$, and the $o_P$ notation is with respect to the source and target samples used to estimate $(\hat{f}_\gamma,\hat{f}_\alpha)$ and $\check{h}$.
The first condition holds by construction of the score, and the second holds by the sample-splitting procedure. Because the source sample is independent of the target sample and is used only to estimate nuisance parameters, the sample-splitting condition holds once the target sample is split between nuisance estimation and parameter estimation ((ref)).
Alternatively, for some machine learning models, we can use the leave-one-out stability condition in chen2022debiased to avoid sample splitting. Since the remaining DML arguments are the same as in chernozhukov2018double, we omit the details. Under additional regularity conditions on the moment function in chernozhukov2018double, the asymptotic normality and consistency of the asymptotic variance estimator are guaranteed. Thus, this paper focuses on the third condition: the convergence rate of the nuisance parameters.
We include four empirical applications to illustrate the DML setup with pre-trained embeddings. Our theory can cover any DML model having the same structure as these examples.
In this subsection, we explain the transfer-learning setup for the DML estimator. We explicitly consider the source task for the computer scientist, together with the target task for the economist.
Hereafter, let $P_{\operatorname{so}}$ and $P_{\operatorname{ta}}$ denote the source and target task distributions, and let $\mathbb{E}_{\operatorname{so}}$ and $\mathbb{E}_{\operatorname{ta}}$ denote the corresponding expectations. Let $P$ and $\mathbb{E}$ denote the joint distribution of the source and target tasks and its expectation, respectively. Thus, the expectations and probabilities used in the above subsections are under the target task $\mathbb{E}_{\operatorname{ta}}$ and $P_{\operatorname{ta}}$ except for the $o_P$ notation, which is with respect to both the source and target tasks.
We consider the setting where a source sample of size $m$ and a target sample of size $n$ share the same approximately true embedding function $h_m$ ((ref)). Our asymptotics let both the source sample size $m$ and the target sample size $n$ go to infinity, where the rate at which $m$ diverges may depend on $n$. Formally, we treat the source sample size $m=m_n$ as a sequence indexed by the target sample size $n$ and let $m_n\to\infty$ as $n\to\infty$. Suppose that these samples are generated by $(S_j,Z_j)\sim_{i.i.d.}P_{\operatorname{so}}$ for $j=1,\dots,m$ and $(Y_i,Z_i)\sim_{i.i.d.}P_{\operatorname{ta}}$ for $i=1,\dots,n$, where $(S_j,Z_j)$ and $(Y_i,Z_i)$ are independent and $Z_j$ and $Z_i$ have different marginal probability distributions. Here, $S_j\in\mathcal{S}\subset\mathbb{R}$ is the outcome for the source task, $Y_i\in\mathcal{Y}\subset\mathbb{R}$ is the outcome for the target task, and $Z_j,Z_i\in\mathcal{Z}\subset\mathbb{R}^d$ denote unstructured or ultra-high-dimensional covariates.
\paragraph{Source Task for Computer Scientist} We focus on the most empirically relevant setting where the source task outcome is multi-valued, i.e., $\mathcal{S}=\{1,2,\dots,T+1\}$ for some integer $T\geq1$. Then, the source task is multi-class classification with $T+1$ classes and the computer scientist trains the pre-trained model $g\circ h(z):\mathcal{Z}\mapsto\mathbb{R}^{T}$ to provide the predicted log-odds for each class $t=1,\ldots,T$ against the base class $T+1$. Let $h:\mathcal{Z}\mapsto\mathcal{V}\subset\mathbb{R}^{K}$ be an embedding function returning $h(z)\in\mathcal{V}$ for some integer $K\geq1$, and let $g=(g_1,\ldots,g_T):\mathcal{V}\mapsto\mathbb{R}^{T}$ be a model for the source task, where $g_t(v)$ is the $t$-th element of $g(v)$ for $t=1,\ldots,T$. We normalize the score of the base class as $g_{T+1}(v)=0$ for every $v\in\mathcal{V}$. Let $\mathcal{G}_m$ denote the function class of $g$, $\mathcal{G}_{t,m}$ denote the function class of $g_t$ for each $t=1,\ldots,T$, and $\mathcal{H}_m$ denote the function class of $h$ for the source task. For example, in a deep neural network model, the model can be decomposed as $g\circ h$, where $h$ is some hidden layer of a deep neural network and $g$ is the remaining transformation from the hidden layer to the output layer. For a given loss function $\ell_{\operatorname{so}}:\mathbb{R}^{T}\times\mathcal{S}\mapsto\mathbb{R}$, let the set of population minimizers be the solution of the population risk minimization:\footnote{ As in farrell2021deep, we assume that minimizer(s) exist for both population and sample risk for simplicity. For the statistical convergence rate theorem, exact existence is not crucial as we can always replace the infimum by a function within a functional class with arbitrarily close risk. }
For a score vector $a_{\operatorname{so}}=(a_{\operatorname{so},1},\ldots,a_{\operatorname{so},T})'\in\mathbb{R}^T$, we similarly normalize the base-class score as $a_{\operatorname{so},T+1}=0$. The most common loss function for multi-class classification is the multinomial logit loss:\footnote{This loss function is also common in the machine learning literature, where it is often called the cross-entropy loss with a softmax output layer.}
The key observation here is that the source task is essentially trained on $T$ different tasks $g_1,\ldots,g_T$ with shared embeddings $h(Z)\in\mathbb{R}^K$ to classify labels with $T+1$ classes.
The computer scientist observes a sample of size $m$, $\{(S_j,Z_j)\}_{j=1}^m\sim P_{\operatorname{so}}^m$, for the source task and estimates the pre-trained model $(\check{g},\check{h})$ by some estimation method, such as empirical risk minimization:
\paragraph{Target Task for Economist} Next, we consider the target task for the economist. The economist will train the target model $f\circ h(z):\mathcal{Z}\mapsto\mathbb{R}$ to predict the target outcome $Y\in\mathcal{Y}$ using learned embeddings from the source task. Let $f:\mathcal{V}\mapsto\mathbb{R}$ be the model for the target task, where $\mathcal{F}_n$ is the function class of $f$ for the target task. Let $\ell_{\operatorname{ta}}:\mathbb{R}\times\mathcal{Y}\mapsto\mathbb{R}$ be the loss function for the target task such as the least squares loss for regression and the logistic loss for binary classification. Given an embedding function $h_m$ from the source task, let $\mathcal{F}_n^{\operatorname{sub}}(h_m)$ denote the set of solutions to the population risk minimization problem:
where $\mathcal{F}_n^{\operatorname{sub}}\subseteq\mathcal{F}_n$ is a subset of a functional class $\mathcal{F}_n$ for the target task, possibly subject to restrictions such as a sparsity restriction for LASSO in a linear model $\mathcal{F}_n$. We often have $\mathcal{F}_n^{\operatorname{sub}}=\mathcal{F}_n$ without additional restrictions, for example for linear regression or shallow neural networks.
The economist observes a sample of size $n$, $\{Y_i,Z_i\}_{i=1}^{n}\sim P_{\operatorname{ta}}^n$, and knows the pre-trained model $\check{h}$ given by the computer scientist, but does not observe the source sample. Since the embedding $\check{h}(z)$ is often high-dimensional, the economist can use various machine learning models for $f$ in the target task. For example, the economist uses a pre-trained convolutional neural network (CNN) to extract high-dimensional embeddings $\check{h}(z)$ (e.g., embeddings of dimension 4{,}096 from a final pooling layer) from ultra-high-dimensional image data (e.g., pixel information with 150{,}528 dimensions). The extracted embeddings are then used as regressors, for example, in a ridge regression of the outcome $Y_i$ on $\check{h}(Z_i)$.
Finally, the economist estimates the target model $\hat{f}$ by empirical risk minimization:\footnote{ Throughout the paper, we impose suitable conditions ensuring that minimizers indexed by random variables exist and are measurable. }
In our theory, we allow small optimization errors for both estimators as shown in (ref) in (ref).
In summary, a computer scientist provides the pre-trained model $(\check{g},\check{h})$ estimated from the source sample $\{(S_j,Z_j)\}_{j=1}^m$, and an economist uses the embeddings $\check{h}(Z_i)$ as regressors to estimate the target function $\hat{f}$ in the target sample $\{(Y_i,Z_i)\}_{i=1}^n$.
Below, we present four examples of the transfer learning setup. Further details of pre-trained deep learning models are provided in (ref).
An autoencoder is a special case of the DNN in (ref), where the source task is the nonparametric regression task with the same input and output, i.e., $S=Z$, and the loss function $\ell_{\operatorname{so}}$ is a distance function between the input and output, such as the mean squared error loss (hinton2006reducing).
See (ref) for details of the functional form of the pre-trained models in the examples above.
This and the next sections present our main theoretical results on transfer learning for DML. For this purpose, we first introduce the true models for the source and target tasks. Suppose that the source and target tasks admit true models satisfying $$a_{\operatorname{so}}^*\in\argmin_{a_{\operatorname{so}}:\text{ square-integrable}}\mathbb{E}_{\operatorname{so}}[\ell_{\operatorname{so}}(a_{\operatorname{so}}(Z),S)]$$ and $$a_{\operatorname{ta}}^*\in\argmin_{a_{\operatorname{ta}}:\text{ square-integrable}}\mathbb{E}_{\operatorname{ta}}[\ell_{\operatorname{ta}}(a_{\operatorname{ta}}(Z),Y)],$$ respectively. For example, suppose that the source loss function $\ell_{\operatorname{so}}$ is the multinomial logit loss, $P_{\operatorname{so}}(S=t\mid Z)>0$ for every $t=1,\ldots,T+1$, $P_{\operatorname{so}}$-a.s., and the corresponding log-odds functions are square-integrable under $P_{\operatorname{so}}$. Under the normalization $a_{\operatorname{so},T+1}^*(Z)=0$ and suitable moment conditions, the true source model is almost surely unique and given by the log-odds of the class probabilities, $a_{\operatorname{so},t}^*(Z) = \log(P_{\operatorname{so}}(S=t\mid Z) / P_{\operatorname{so}}(S=T+1\mid Z))$ for $t=1,\ldots,T$. For the target task, if the loss is the squared loss $\ell_{\operatorname{ta}}(a_{\operatorname{ta}}(Z),Y)=(1/2)\times(Y - a_{\operatorname{ta}}(Z))^2$ and suitable moment conditions are satisfied, the true model is almost surely unique and given by the conditional expectation of the outcome as $a_{\operatorname{ta}}^*(Z)=\mathbb{E}_{\operatorname{ta}}[Y\mid Z]$, and if the loss is the binary logit loss $\ell_{\operatorname{ta}}(a_{\operatorname{ta}}(Z),Y)=-Y\log(\exp(a_{\operatorname{ta}}(Z))/(1+\exp(a_{\operatorname{ta}}(Z))))-(1-Y)\log(1/(1+\exp(a_{\operatorname{ta}}(Z))))$ and suitable moment conditions are satisfied, the true model is almost surely unique and given by the log-odds of the conditional probability as $a_{\operatorname{ta}}^*(Z)=\log(P_{\operatorname{ta}}(Y=1\mid Z) / P_{\operatorname{ta}}(Y=0\mid Z))$. In DML applications, these two loss functions are the most common for the target task.
In this section, we develop a general convergence rate theory for the target model $\hat{f}\circ \check{h}$ by relating it to the source task performance of the pre-trained model $\check{g}\circ\check{h}$ relative to the true model $a_{\operatorname{so}}^*$. In (ref), we introduce two key concepts: an approximately true embedding function and task diversity. In (ref), we present the theorem on the convergence rate of the target model in general form. (ref) discusses sufficient and necessary conditions for task diversity.
We first introduce the concept of an approximately true embedding function, which is a key definition for the transfer learning theory. In general, $a^*_{\operatorname{so}}$ and $a^*_{\operatorname{ta}}$ need not be representable as $g_m \circ h_m$ and $f_n \circ h_m$. This mismatch arises from restrictions on the source classes $\mathcal{G}_m,\mathcal{H}_m$ and the target class $\mathcal{F}_n^{\operatorname{sub}}$ even if there exists an embedding function that captures the common features of the source and target tasks well. The following definition quantifies the approximation error for both source and target tasks.
While we state it as a definition, our result requires that the approximation errors $\epsilon_{\operatorname{so},m}$ and $\epsilon_{\operatorname{ta},n}$ converge to zero as $n\to\infty$ for consistency of the DML estimator in (ref). Decays of $\epsilon_{\operatorname{so},m}$ and $\epsilon_{\operatorname{ta},n}$ are determined by the richness of the function classes $\mathcal{G}_m$ and $\mathcal{F}_n^{\operatorname{sub}}$ and how well the embedding function $h_m$ captures the common features of the source and target tasks. When the function classes $\mathcal{G}_m$ and $\mathcal{F}_n^{\operatorname{sub}}$ are linear, the approximation errors $\epsilon_{\operatorname{so},m}$ and $\epsilon_{\operatorname{ta},n}$ are determined by the linear projection of $a_{\operatorname{so}}^*$ and $a_{\operatorname{ta}}^*$ onto the linear span of the embedding function $h_m$, which is similar to the approximation error in the series regression model (newey1997convergence) using some basis functions instead of the embedding function $h_m$.
Next, we introduce the concept of task diversity, which is another key definition. We define task diversity in a way that makes its connection to identification explicit. The reader can refer to (ref) for a detailed comparison with existing definitions of task diversity in the machine learning literature. For functions $(g_m,h_m)\in\mathcal{R}_m$ and $f_n\in\mathcal{F}_n^{\operatorname{sub}}(h_m)$, define the following approximate identified sets of $h_m$ for $\rho\geq0$:
and
where we assume that the minimum exists for both the source and target tasks for simplicity. Note that $\mathcal{H}_{\operatorname{so},m}(0)$ and $\mathcal{H}_{\operatorname{ta},m}(0)$ are the identified sets of $h_m$ for the source task and the target task, respectively. If $h_m$ is identified, these sets shrink to $\{h_m\}$ as $\rho\downarrow 0$. In general, however, $h_m$ need not be identified, and these sets can contain multiple elements. It is known that the hidden layers of the deep neural network are identified only up to some transformations, such as the permutation of the order of the hidden units, the scaling of the hidden units, and the rotation of the hidden units (grigsby2023hidden). See also (ref) for another example of partial identification of the embedding function for middle hidden layers of a deep neural network.
Thus, the task diversity condition requires that if $h$ is in a near-identified set of the source task, then $h$ is also in a near-identified set of the target task, where the distances are measured by the $L^2(P_{\operatorname{so}})$- and $L^2(P_{\operatorname{ta}})$-norms, respectively.
The name “task diversity” comes from the intuition that if the $T$ source tasks are sufficiently diverse, then any embedding function $h$ close to the source task identified set should also be close to the target task identified set. This intuition and sufficient and necessary conditions for task diversity are formalized in (ref).
Next, we introduce the concept of $\nu_n$-transferability, which is defined in terms of the task diversity condition.
$\nu_n$-transferability at rate $\delta_m$ requires that the $(\rho_{\operatorname{so},m},\rho_{\operatorname{ta},n})$-diversity condition holds for all source-side sequences $\rho_{\operatorname{so},m}=O(\delta_m)$ with $\rho_{\operatorname{ta},n}=\rho_{\operatorname{so},m}/\nu_n$. The quantity $\nu_n$ captures the identification strength of the target task informed by the source task. Larger $\nu_n$ implies that the task $g_m$ is more transferable to $f_n$ since it implies a smaller target task near-identified set given the near-identified set of the source task.
We will relate the sequences $\rho_{\operatorname{so},m}$ to the convergence rates of the source task introduced below. Let $f_n^{\bullet}\in\argmin_{f\in\mathcal{F}_n^{\operatorname{sub}}} \|f\circ \check{h}-f_n \circ h_m\|_{P_{\operatorname{ta}},2}$ be the best approximation of the population target model $f_n\circ h_m$ by the estimated embedding function $\check{h}$. Formally, $f_n^{\bullet}$ depends on $\check{h}$, but we suppress the dependence for notational simplicity.
The first part of this assumption states that the pre-trained model $\check{g}\circ \check{h}$ converges to the population source model $g_m\circ h_m$ at some rate $\delta_{\operatorname{so},m}$. The second part states that the target model $\hat{f}\circ \check{h}$ converges to the population target model $f_n^{\bullet}\circ \check{h}$ at some rate $r_{\operatorname{ta},n}$ given the estimated embedding function $\check{h}$. Note that the $O_P$ notation in the second part is with respect to both the source and target samples since $\check{h}$ depends on the source sample. This assumption is high-level, and we will provide more primitive sufficient conditions in (ref) to avoid assuming this directly.
Now, we assume the $\nu_n$-transferability condition with the convergence rate of the source task $\delta_{\operatorname{so},m}$.
(ref) and the quantity $\nu_n$ play a key role in determining the convergence rate of the target model $\hat{f}\circ \check{h}$ as we will see in the next subsection.
Our main theorem on the convergence rate of the target model is as follows.
This theorem states that the convergence rate of the target model $\hat{f}\circ \check{h}$ to the true model $a_{\operatorname{ta}}^*$ is determined by three components: (i) the convergence rate $r_{\operatorname{ta},n}$ of the target model given the estimated embedding function $\check{h}$, (ii) the convergence rate of the source task adjusted by the transferability strength $\nu_n$, and (iii) the approximation error $\epsilon_{\operatorname{ta},n}$ due to an approximately true embedding function. Thus, the rate of the target model is determined by the slowest of these three components. The target-stage rate $r_{\operatorname{ta},n}$ is achieved when transferability is strong, the source task converges sufficiently fast, and the approximation error is sufficiently small.
The next theorem gives a useful necessary condition based on the source task identified set $\mathcal{H}_{\operatorname{so},m}(0)$. We will revisit this necessary condition in (ref) for the specific case of the LASSO.
In this subsection, we discuss sufficient and necessary conditions for task diversity in (ref).
We first assume that the density ratio of $Z$ in the source and target tasks is bounded away from zero.
(ref) holds only if the support of $Z$ in the target task is included in the support of $Z$ in the source task. We allow $\underline{w}_n$ to decrease slowly to zero as $n$ increases, but not too quickly. Although it is commonly assumed in the machine learning literature (e.g., johansson2019support), the density ratio condition is a strong assumption especially when $Z$ is unstructured data. Relaxing this assumption is beyond the scope of this paper and is an important direction for future research.
This theorem provides a sufficient condition for $(\rho_{\operatorname{so},m},\rho_{\operatorname{ta},n})$-diversity based on the existence of a Lipschitz function $\Gamma$ that relates the target model $f_n\circ h_m$ to the source model $g_m\circ h_m$. The first condition requires that the target model can be approximated by applying the Lipschitz function $\Gamma$ to the source model with some small error $\varrho_n$. The second condition requires that for every embedding function $h$ close to the source task identified set, there exists a target model $f$ that can approximate $\Gamma \circ g^{\bullet}_m \circ h$ with some small error $\varrho_n$. Under these conditions and the density ratio condition in (ref), the task $g_m$ is $(\rho_{\operatorname{so},m},L_{\Gamma}\rho_{\operatorname{so},m}/\underline{w}_n+2\varrho_n)$-diverse over $f_n$ with respect to $h_m$ for every $\rho_{\operatorname{so},m}\geq0$. Note that transferability is stronger if the Lipschitz constant $L_{\Gamma}$ is small, the approximation error $\varrho_n$ is small, and the constant $\underline{w}_n$ related to the infimum of the density ratio is large.
In the extreme case $L_\Gamma=0$ (i.e., $\Gamma$ is constant), we have $(\rho_{\operatorname{so},m},2\varrho_n)$-diversity. Also, if $L_\Gamma$ diverges to infinity, the $(\rho_{\operatorname{so},m},\rho_{\operatorname{ta},n})$-diversity condition does not hold for every finite $\rho_{\operatorname{ta},n}$. In summary, a small $L_\Gamma$ leads to better transferability.
In the next example, we verify the sufficient condition in (ref) when both the computer scientist and the economist use linear models.
(ref) shows that, when the target class contains all linear heads with intercepts, the row-span condition is sufficient for transferability with $\varrho_n=0$. Under the regularity conditions in (ref), the same condition is also necessary for $(0,0)$-diversity. Intuitively, if the coefficients of the source tasks are informative enough to represent the coefficients of the target task, the transferability condition holds. Thus, if we want to guarantee transferability uniformly over all target tasks for every possible $\beta_{\operatorname{ta}}$, this requires at least $K$ source tasks ($T\geq K$). The next corollary states sufficient conditions for general Lipschitz target models, which include the linear target case as a special case.
The corollary implies that, when the source task is linear and the target task is Lipschitz, the $\nu_n$-transferability condition holds if the source task coefficients are informative enough to represent the target task coefficients. Moreover, the transferability measure $\nu_n = \underline{w}_n\sigma_{\min}(B_{\operatorname{so},m})/L_{\mathcal{F}}$ is larger if the source task has a larger minimum singular value $\sigma_{\min}(B_{\operatorname{so},m})$, the target task has a smaller Lipschitz constant $L_{\mathcal{F}}$, and the density ratio of $Z$ in the source and target tasks is bounded away from zero with a larger value of $\underline{w}_n$.
The following theorems provide necessary conditions for task diversity. We start with the linear case as in (ref). When $B_{\operatorname{so},m}$ is rank deficient, there is a direction $v(Z)$ in the embedding space that is not captured by any of the source tasks, but it can be relevant for the target task. Intuitively, if the source tasks are not diverse enough to cover all directions of the target task, transferability generally fails for the uncovered directions of the source task coefficients. See the next theorem for a formal statement.
The assumption $(I_K - vv')h_m + c\in\mathcal{H}_m$ is a technical condition that allows us to construct a function $h$ that attains the same source prediction as $h_m$ but a different target prediction from $h_m$. For some functional classes $\mathcal{H}_m$ such as the class of ReLU activated hidden layer functions, $\mathcal{H}_m$ only contains nonnegative functions, but it satisfies this assumption by choosing $c$ to be sufficiently large if $\mathcal{H}_m$ is sufficiently rich.
The following theorem provides necessary conditions for transferability in a general model with a nonlinear source task under sufficiently rich function classes $\mathcal{H}_m$ and $\mathcal{G}_{m}$ so that the nonlinear analog of the construction $h$ in the proof of (ref) is possible. Note that we do not put any restriction on the function class $\mathcal{F}_n^{\operatorname{sub}}$.
The existence of a distinct $h_0$ in (ref) means that the population source model does not uniquely identify the embedding. Such source nonidentification commonly arises in deep neural networks, where different hidden representations can be paired with compensating source heads. As $(0,0)$-diversity requires $\mathcal{H}_{\operatorname{so},m}(0)\subseteq\mathcal{H}_{\operatorname{ta},m}(0)$, the existence of $\Gamma_0$ is needed to adapt the nonidentified embedding function. The sigma-field inclusion condition is an additional information condition for the target distribution. For the same $h_0$, it states that, up to $P_{\operatorname{ta}}$-null sets, the population source output $g_m\circ h_m$ contains all the information on $h_0(Z)$. Sufficiently rich classes $\mathcal{G}_m$ and $\mathcal{H}_m$ are needed for the construction of a source-equivalent $h_0$ satisfying this condition. Under this additional condition, the existence of $\Gamma$ is necessary for $(0,0)$-diversity.
Under the head-class and regularity conditions in (ref), the necessary conditions show that the corresponding sufficient condition is also necessary in these settings. These theorems suggest that the sufficient condition in (ref) with $\varrho_n=0$, namely $f_n \circ h_m(Z) = \Gamma \circ g_m \circ h_m(Z)$, $P_{\operatorname{ta}}$-a.s., is also necessary for the transferability condition in these cases. However, the next example shows that the transferability condition may not hold even if both $f_n$ and $g_m$ are linear but the pre-training model class $\mathcal{G}_m$ is nonlinear.
From this section, we have seen that the transferability condition is substantially stronger for nonlinear source heads $\mathcal{G}_m$ than for linear source heads $\mathcal{G}_m$. This is because the nonlinear source head $\mathcal{G}_m$ can approximate a much richer class of functions than the linear source head $\mathcal{G}_m$, which makes it more difficult to satisfy the diversity condition. Therefore, if the economist is indifferent between linear and nonlinear source heads, the linear source head is preferable for guaranteeing the transferability condition.
In this section, we provide lower-level primitive conditions for the convergence rates for the transfer-learning problem without the high-level conditions of (ref). We first introduce some notation. For a function class $\mathcal{F}$, let $\mathsf{V}(\mathcal{F})$ be the VC dimension of the set of subgraphs of functions in $\mathcal{F}$.\footnote{The {\it subgraph} of a function $f: \mathcal{X} \rightarrow \mathbb{R}$ is the subset of $\mathcal{X} \times \mathbb{R}$ given by $\{(x,y): y<f(x)\}$. The VC dimension of a set of subgraphs is defined as the largest integer such that there exists a set of points with cardinality equal to the integer that can be shattered by the subgraphs. See van2023weak, Section 2.6 for more details.} To avoid technical complications concerning the measurability of the supremum, we assume that the functional classes $\mathcal{F}_n$, $\mathcal{F}_n^{\operatorname{sub}}$, and $\mathcal{G}_m\circ\mathcal{H}_m$ are pointwise measurable.\footnote{A measurable functional class $\mathcal{F}$ is {\it pointwise measurable} if it contains a countable subset $\mathcal{F}^0$ such that for every $f \in \mathcal{F}$ there exists a sequence $f_j\in\mathcal{F}^0$ with $f_j(v) \rightarrow f(v)$ for every $v$, where the convergence is with respect to the Euclidean norm. See van2023weak, Section 2.3 for more details.}
We start with the convergence rate of the target model given the estimated embedding function $\check{h}$ from the source task. Before we state the theorem on the convergence rate of the target model $\hat{f}\circ \check{h}$, we assume one more condition on the target loss function.
This assumption requires that the excess risk of the target loss function is bounded above and below by the squared $L^2$-norm. This assumption is standard in statistical learning theory and is imposed in many existing works (e.g., farrell2021deep). (ref) verifies the excess-risk comparison and the required Lipschitz comparison over uniformly bounded candidate-score ranges for least squares and binary logistic loss.
The theorem shows that, under standard conditions on the function class $\mathcal{F}_n$ and the target loss function, the convergence rate of the target model $\hat{f}\circ \check{h}$ to the true model $a_{\operatorname{ta}}^*$ is governed by the VC dimension of $\mathcal{F}_n$ and the target sample size $n$.
Next, we provide a theorem on the convergence rate of the source model $\check{g}\circ \check{h}$ to the population source model $g_m\circ h_m$. The rate of convergence for deep learning models is an active area of research and can be improved in future work. Importantly, our transfer learning rate of the target task uses the convergence rate of the source model as an input; thus, any improvement in the convergence rate of the source model can be directly translated into an improved convergence rate of the target model by our theory.
The loss conditions in parts (i)-(ii) are satisfied with $\tau_m = T^{-1}$ if the loss function is the multinomial logistic loss and the Euclidean-norm bound $M$ in (ref) holds uniformly in $m$ and $T$ over all $g\in\mathcal{G}_m$ and $h\in\mathcal{H}_m$ and for $a_{\operatorname{so}}^*$. Part (iii) is a regularity condition on the source-head class rather than the loss function. It rules out discontinuous heads and allows the entropy of the composite class $\mathcal{G}_m\circ\mathcal{H}_m$ to be controlled by the coordinatewise entropies of $\mathcal{G}_m$ and $\mathcal{H}_m$.
The next theorem relates the convergence rate of the source model in (ref) (i) to the VC dimensions of the function classes $\mathcal{H}_m$ and $\mathcal{G}_{m}$.
This theorem states that under some standard conditions on the function classes $\mathcal{H}_m$ and $\mathcal{G}_{m}$ and the source loss function, the convergence rate of the source model $\check{g}\circ \check{h}$ to the population model $g_m\circ h_m$ (or the true model $a_{\operatorname{so}}^*$) is determined by the coordinatewise VC dimensions of $\mathcal{H}_m$ and $\mathcal{G}_{m}$, the logarithmic factors induced by the embedding dimension $K$, the output dimension $T$, and the head Lipschitz constant $L_{\mathcal{G},m}$, the source sample size $m$, and the curvature term $\tau_m$ of the source loss function.
(ref) together imply that, under the transferability assumption, the convergence rate of the target model $\hat{f}\circ \check{h}$ to the true model $a_{\operatorname{ta}}^*$ is governed mainly by five components: the ratio of the VC dimension of the target model class $\mathcal F_n$ to the target sample size $n$; the ratio of the source complexities for $\mathcal H_m$ and $\mathcal G_m$ to the source sample size $m$; the transferability rate $\nu_n$; the approximation error terms $\epsilon_{\operatorname{ta},n}$ and $\epsilon_{\operatorname{so},m}$; and the curvature term $\tau_m$ of the source loss function. For example, $\mathsf{V}(\mathcal{F}_n)=O(K)$ if $\mathcal{F}_n$ is a class of linear models with $K$ regressors, and $\mathsf{V}(\mathcal{F}_n)=O(W_n L_n \log(W_n))$ if $\mathcal{F}_n$ is a class of DNNs with width $W_n$ and depth $L_n$ (bartlett2019nearly, Theorem 7). In particular, the rate improves when both function class complexities are small relative to sample size and the transferability and approximation error terms are negligible. If the complexity of the function classes is smaller than the square root of the sample sizes, and the transferability and approximation error terms are sufficiently small, the target model can achieve a convergence rate faster than $n^{-1/4}$.
The derived rate is faster than the one in tripuraneni2020theory since we effectively use the curvature condition and the boundedness of the loss function. Also, our rate is comparable to the result in watkins2023optimistic although their fast rate assumes $\mathbb{E}_{\operatorname{so}}[\ell_{\operatorname{so}}(a_{\operatorname{so}}^*(Z),S)]=0$, which is a strong and unrealistic assumption in practice. Without this assumption, their rate becomes slower than ours and comparable to the one in tripuraneni2020theory. Importantly, the slow rate in tripuraneni2020theory is not enough to apply the DML framework since it is always slower than $n^{-1/4}$.
We can directly apply the above theorems to derive the convergence rates of transfer learning when the target sample size $n$ is large enough compared to its input dimension $K$. This strategy is taken in, for example, bajari2025hedonic and bach2024adventures. Since the VC dimension is well studied for many specific models such as linear models and DNNs (bartlett2019nearly), the above theorems can be applied to derive the convergence rates of transfer learning for these specific models under some conditions on the transferability and approximation errors. For DNNs, the approximation error can be small if the true model is sufficiently smooth and the DNN has enough width and depth (yarotsky2017error).
For the low-dimensional case, the VC dimension of the function class $\mathcal{F}_n$ is small, and (ref) is directly applicable to derive the convergence rate of the target model $\hat{f}\circ \check{h}$ to the true model $a_{\operatorname{ta}}^*$. An important feature of the VC dimension is that it is a pure functional complexity measure and does not depend on the distribution of the input data. On the other hand, (ref) is not directly applicable to the high-dimensional case since the VC dimension of $\mathcal{F}_n$ is large, and we need to assume some additional structure on the input data, i.e., embeddings, to derive the convergence rate of the target model.
In this section, we focus on the case in which the source model has a linear head as in (ref) and $h_m(Z)\in\mathbb{R}^K$ is the last hidden layer of the source model. In this case, we have a convergence rate of the estimated embedding function $\check{h}$ to an approximately true embedding function $h_m$ up to a linear transformation. Recall that the population source model is $g_m \circ h_m(Z) = \alpha_{\operatorname{so},m} + B_{\operatorname{so},m} h_m(Z)$ for some vector $\alpha_{\operatorname{so},m}\in\mathbb{R}^T$, some matrix $B_{\operatorname{so},m}\in\mathbb{R}^{T\times K}$, and an embedding function $h_m(Z)\in\mathbb{R}^K$. The pre-trained source model is $\check{g}\circ \check{h}(Z) = \check{\alpha}_{\operatorname{so}} + \check{B}_{\operatorname{so}} \check{h}(Z)$ for some vector $\check{\alpha}_{\operatorname{so}}\in\mathbb{R}^T$, some matrix $\check{B}_{\operatorname{so}}\in\mathbb{R}^{T\times K}$, and an embedding function $\check{h}(Z)\in\mathbb{R}^{K}$ estimated from the source task. Define a transformed embedding function $\check{h}^{\mathrm{proj}}$ as the projection of $\check{h}$ onto the row space of $\check{B}_{\operatorname{so}}$ as follows:
The following lemma gives an $L_2$ convergence rate for the projected embedding $\check{h}^{\mathrm{proj}}$ toward an affine transformation $\check{q}+\check{Q}h_m$ of $h_m$, where $\check{q} := \check{B}_{\operatorname{so}}^{+} (\alpha_{\operatorname{so},m} - \check{\alpha}_{\operatorname{so}})$ is the shift term and $\check{Q} := \check{B}_{\operatorname{so}}^{+} B_{\operatorname{so},m}$ is the linear transformation of $h_m$.
If $\check{B}_{\operatorname{so}}$ has full column rank, $\check{B}_{\operatorname{so}}^{+} = (\check{B}_{\operatorname{so}}'\check{B}_{\operatorname{so}})^{-1}\check{B}_{\operatorname{so}}'$ is the left inverse of $\check{B}_{\operatorname{so}}$ and $\check{B}_{\operatorname{so}}^{+}\check{B}_{\operatorname{so}} = I_K$ is the identity matrix; thus, $\check{h}^{\mathrm{proj}} = \check{h}$. On the other hand, we have $\check{B}_{\operatorname{so}}^{+}\check{B}_{\operatorname{so}} \neq I_K$ in general if $\check{B}_{\operatorname{so}}$ does not have full column rank.
We can interpret $\check{h}^{\mathrm{proj}}$ as the projection of $\check{h}$ onto the row space of $\check{B}_{\operatorname{so}}$ since $\check{B}_{\operatorname{so}}^+\check{B}_{\operatorname{so}} = P_{\check{B}_{\operatorname{so}}'} := \check{B}_{\operatorname{so}}'(\check{B}_{\operatorname{so}}\check{B}_{\operatorname{so}}')^+\check{B}_{\operatorname{so}}$ by harville2008matrix, Corollary 20.3.8. Note that $\check{h}^{\mathrm{proj}}$ and $\check{h}$ return the same output for the source task since $\check{B}_{\operatorname{so}}\check{h}^{\mathrm{proj}}(z) = \check{B}_{\operatorname{so}}\check{h}(z)$. The projection preserves the source output while removing components of the estimated embedding in the null space of $\check{B}_{\operatorname{so}}$.\footnote{An indeterminacy can remain. For every constant vector $c\in\mathbb{R}^{K}$, the pairs $(\check{\alpha}_{\operatorname{so}},\check{h}^{\mathrm{proj}})$ and $(\check{\alpha}_{\operatorname{so}}+\check{B}_{\operatorname{so}}c,\check{h}^{\mathrm{proj}}-c)$ produce the same source output algebraically. However, this indeterminacy does not affect the convergence of the target model as long as $\|c\|_2$ is bounded and the target model is nonparametric or includes a slope coefficient.}
(ref) implies that we can estimate $h_m$ only up to a shifted linear transformation represented by $\check{h}^{\mathrm{proj}}$. The matrix $\check{Q}$ describes the linear map from an approximately true embedding function $h_m$ to the component of $\check{h}$ that is identified from the source task. In the special case in which $\check{\alpha}_{\operatorname{so}}=\alpha_{\operatorname{so},m}$, $\check{B}_{\operatorname{so}} = B_{\operatorname{so},m}A^{-1}$ and $\check{h} = A h_m$ for an invertible matrix $A\in\mathbb{R}^{K\times K}$, and $B_{\operatorname{so},m}$ has full column rank, we have $\check{B}_{\operatorname{so}}\check{h} = B_{\operatorname{so},m}A^{-1}A h_m = B_{\operatorname{so},m}h_m$ and the exact equality $\check{h}^{\mathrm{proj}} = \check{Q} h_m$, where $\check{Q} = \check{B}_{\operatorname{so}}^{+} B_{\operatorname{so},m} = A$. Since this holds for every invertible matrix, the embedding function $h_m$ is identified from the source task only up to an invertible linear transformation $A$ such that $A h_m\in\mathcal{H}_m$. This special case is consistent with the discussion in schulte2025adjustment that the embedding function is identified only up to an invertible linear transformation under the restrictive assumption that there is no estimation error for the embedding function. More generally, the matrix $\check{Q}$ captures the identified set of the embedding function $h_m$ from the source task output. This extends the invertible reparameterization partial identification to settings in which $B_{\operatorname{so},m}$ and $\check{B}_{\operatorname{so}}$ may be rank deficient.
We do not recommend using LASSO for the target model for the following reason. As we have seen in (ref), $\mathcal{H}_{\operatorname{so},m}(0)$ includes invertible linear transformations of an approximately true embedding function $h_m$. (ref) suggests that, to ensure $\|\hat{f}_{n,h_m^{\dagger}}\circ h_m^{\dagger} - a^*_{\operatorname{ta}}\|_{P_{\operatorname{ta}},2} = o_{P_{\operatorname{ta}}}(1)$ for any sequence $h_m^{\dagger}\in\mathcal{H}_{\operatorname{so},m}(0)$, it is necessary that $\mathcal{H}_{\operatorname{so},m}(0)\subseteq \mathcal{H}_{\operatorname{ta},m}(\rho_{\operatorname{ta},n})$ for some sequence $\rho_{\operatorname{ta},n}=o(1)$ when the target model has small estimation and approximation errors. That is, for every invertible matrix $Q\in\mathbb{R}^{K\times K}$ and every vector $q\in\mathbb{R}^{K}$ such that $q + Q h_m \in \mathcal{H}_m$, there must exist some $f\in\mathcal{F}_n^{\operatorname{sub}}$ such that $\|f_n\circ h_m - f\circ (q + Q h_m)\|_{P_{\operatorname{ta}},2}$ converges to zero as $n\rightarrow\infty$. Note that linear regression, ridge regression, and DNNs satisfy this condition since they transform the input vector by a linear transformation.
Therefore, our theory recommends avoiding methods that rely on assumptions, such as sparsity, that are not preserved under the unidentified invertible linear transformations of the embedding function $h_m$ (e.g., wainwright2019high, Chapter 7). Suppose the economist uses LASSO for the target model given the estimated embedding function $\check{h}$ from the source task. To use LASSO, the existing theory typically requires an approximate-sparsity assumption. On the other hand, since $\mathcal{H}_{\operatorname{so},m}(0)$ includes invertible linear transformations of an approximately true embedding function $h_m$, Theorem 1 in kolesar2025fragility implies that the approximate sparsity assumption is violated in general. Thus, even if the economist can justify the approximate sparsity assumption for an embedding function $h_m$, the approximate sparsity assumption is hard to justify for other $h\in\mathcal{H}_{\operatorname{so},m}(0)$. Since we cannot identify the embedding function $h_m$ from the source task, any method relying on variable selection can be fragile.
(ref) shows that approximate sparsity is not preserved under the linear reparameterizations left unidentified by the source task. We therefore study an $\ell_2$-penalized linear target model as a tractable alternative.
The scale of a linear source head is not identified separately from the scale of its embedding. Thus, the condition in (ref) is not invariant to observationally equivalent rescalings. Multiplying the embedding by a large constant and dividing the source-head matrix by the same constant can make its smallest positive singular value arbitrarily small. This scale indeterminacy is not a problem for the low-dimensional case since the VC dimension is invariant to scaling. However, the scale indeterminacy is typically a problem for the high-dimensional case since we use the structure on the embeddings and the source head to derive the convergence rate of the target model. The ridge theory below therefore imposes fixed singular value bounds on the source-head matrices used in the analysis.
Fix constants $C_{\alpha,\operatorname{so}}<\infty$ and $0<\underline{\sigma}\leq\bar{\sigma}<\infty$, and let $\Theta_{\operatorname{so},m}$ be a deterministic, nonempty, compact subset of
We assume that the population source-head parameters $(\alpha_{\operatorname{so},m},B_{\operatorname{so},m})$ and the pre-trained source-head parameters $(\check{\alpha}_{\operatorname{so}},\check{B}_{\operatorname{so}})$ belong to $\Theta_{\operatorname{so},m}$ with probability approaching one. The economist can always obtain them through a transformation in (ref) if they want to ensure this condition explicitly.
For each $(\alpha,B)\in\Theta_{\operatorname{so},m}$, define
For this subsection, let $\mathcal{G}_m := \{g_{\alpha,B}:(\alpha,B)\in\mathbb{R}^{T}\times\mathbb{R}^{T\times K}\}$ be the class of linear source heads, so that $\{g_{\alpha,B}:(\alpha,B)\in\Theta_{\operatorname{so},m}\}\subseteq\mathcal{G}_m$. Thus, $\mathcal{G}_m$ may also contain linear heads with parameters outside $\Theta_{\operatorname{so},m}$. For every $(\alpha,B)\in\Theta_{\operatorname{so},m}$, (ref) gives $\|B^+\|_{\mathrm{sp}}\leq\underline{\sigma}^{-1}$. Moreover, on the event that the chosen population and pre-trained source-head parameters belong to $\Theta_{\operatorname{so},m}$, we have $\|\check{Q}\|_{\mathrm{sp}} = \|\check{B}_{\operatorname{so}}^+B_{\operatorname{so},m}\|_{\mathrm{sp}}\leq\bar{\sigma}/\underline{\sigma}$.
Define the projected-embedding class used in the ridge analysis by
For every $\rho\geq0$, let $\mathcal{H}_{\operatorname{so},m}^{\mathrm{proj}}(\rho):=\{h\in\mathcal{H}_m^{\mathrm{proj}}: \min_{g\in\mathcal{G}_{m}}\|g\circ h-g_m\circ h_m\|_{P_{\operatorname{so}},2}\leq \rho\}$. Fix constants $C_{\alpha},C_{\beta}<\infty$, and let
Let $\mathcal{F}_n = \mathcal{F}_n^{\operatorname{sub}} = \{f(v) = \alpha+\beta'v: (\alpha,\beta)\in\Theta_n\}$ and let $f_n(v) = \alpha_{\operatorname{ta},n}+\beta_{\operatorname{ta},n}'v\in\mathcal{F}_n^{\operatorname{sub}}(h_m)$ be the population target model. For a deterministic sequence $\lambda_n>0$, define the ridge estimator on the projected embedding by
We assume the following conditions on the target model and the source parameters.
For any embedding $h$, let
In particular, $\Sigma_{h_m,n}$ denotes the covariance matrix of the population embedding. Define the effective degrees of freedom of the ridge regression based on the embedding $h$ by
where $\mu_1(\Sigma_{h,n})\geq\cdots\geq\mu_K(\Sigma_{h,n})\geq0$ are the eigenvalues of $\Sigma_{h,n}$.
The first two terms in the above rate are the estimation errors of ridge regression, and the last two terms are the transferability and approximation errors. The middle term arises from the difference between $\mathcal{N}_{\check{h}^{\mathrm{proj}},n}(\lambda_n)$ and $\mathcal{N}_{h_m,n}(\lambda_n)$.
For a concrete rate, suppose that, uniformly over $n$ and $j=1,\ldots,K$,
and $\delta_{\operatorname{so},m}/\underline{w}_n+\epsilon_{\operatorname{ta},n}=O(n^{-a/(2a+1)})$. Then $\mathcal{N}_{h_m,n}(\lambda)\lesssim\lambda^{-1/a}$. Choosing $\lambda_n\asymp n^{-a/(2a+1)}$ gives
If $a>1/2$, the above rate is $o_P(n^{-1/4})$, which is sufficient for the DML framework. Thus, ridge is a recommended regularized alternative when the target model is linear and the embedding is nearly identified up to a well-conditioned linear distortion.
In practice, the tuning parameter $\lambda_n$ for ridge is typically selected by $K$-fold cross-validation that directly targets prediction risk; see, for example, hastie2022surprises for a recent detailed discussion of cross-validation for ridge regression, including in over-parameterized settings.
A natural question is whether a similar convergence rate can be achieved by neural network models with regularization such as random dropout or $\ell_2$ regularization.\footnote{ Another line of research in deep learning theory on convergence rates makes use of functional shape assumptions on the function of interest (kohler2021rate,bhattacharya2024deep) or data structure assumptions such as approximate manifold structure (jiao2023deep,schulte2025adjustment). However, their theory relies on the VC-dimension bound derived by bartlett2019nearly, which is always larger than the dimension of the raw input data. Thus, the convergence rate is not informative unless the dimension of the raw input data is treated as fixed relative to the sample size. } We conjecture that the answer is positive. For linear regression, dropout can be interpreted as a form of ridge regularization (srivastava2014dropout, Section 9.1). wei2020implicit demonstrate both theoretically and empirically that dropout can work as regularization for deep neural networks. However, even when $h_m$ is directly observed, the convergence rate of the regularized neural network estimator is not well understood in the literature at the level of generality of farrell2021deep, to the best of our knowledge. Establishing the convergence rate of regularized neural network estimators for the target model in our setting is an important direction for future research.