Ill-Conditioned Orthogonal Scores in Double Machine Learning

Ill-Conditioned Orthogonal Scores in Double Machine Learning

\noindentKeywords: Neyman orthogonality; overlap/positivity; Riesz representer; semiparametric efficiency; weak identification

\noindentJEL Codes: C14, C21, C55

\thispagestyle{plain}

Introduction

Double Machine Learning chernozhukov2018dml permits the application of machine learning methods for nuisance estimation, while preserving $\sqrt{n}$-inference for low-dimensional treatment effects. This holds because Neyman-orthogonal scores reduce first-order sensitivity to nuisance estimation errors and cross-fitting avoids own-observation bias chernozhukov2018dml, kennedy2023semipar. As a result, the corresponding sufficient conditions are typically stated as rate conditions, rather than as parametric smoothness restrictions. In the canonical partially linear regression (PLR) model, for instance, the classical product-rate requirement is $r_n^m r_n^\ell = o_P(n^{-1/2})$, where $r_n^m,r_n^\ell$ denote $L^2$ nuisance-estimation rates chernozhukov2018dml.

These conditions control the entry of nuisance estimation error into the orthogonal score. However, these rate conditions do not track how residual treatment variation after conditioning (the PLR analogue of overlap/positivity) directly impacts finite-sample reliability khanTamer2010overlap,damour2021overlap. Standard DML theory often relies on regularity conditions to keep this residual variation bounded away from zero. We instead make the finite-sample sensitivity induced by low residual treatment variation explicit in the canonical PLR-DML setting through an exact identity and a stochastic bound.\footnote{Our main results are for PLR-DML, where the score is affine in $\theta$ and the identity is exact. Appendix A.1 shows the generic orthogonal-score expansion for nonlinear scores, where a Taylor/linearization step is required.} A simple reportable out-of-fold diagnostic follows as a reporting implication.

An amplification mechanism is transparent in PLR. In this case, the sensitivity of the empirical orthogonal score to the treatment effect $ \theta $ depends on the empirical Jacobian $\widehat{J}_\theta = -\,\widehat{\sigma}_{V}^{2}$, where $V$ is the residualized treatment. If $\widehat{\sigma}_{ V }^{ 2 } $ is small, the score is nearly flat in the $\theta$ direction. Small perturbations of the empirical score equation therefore create large perturbations of $\widehat{\theta}$. This corresponds to ill-conditioning in the numerical sense golubvanloan2013. We capture this channel through $\kappa := { \sigma }_{ D }^{ 2 }/{\sigma}_{ V }^{ 2 } = 1/(1-R^{ 2 }(D\mid X))$. It scales the impact of any residual bias left after orthogonalization and cross-fitting. While standard errors reflect increased sampling variability when $\sigma_{ V }^{ 2 }$ is small, they do not consider the bias-amplification induced by large $ \kappa $. Coverage can fail as a result, even when reported standard errors are only moderately inflated.

Our first main result is an exact finite-sample decomposition for DML in PLR: \[ \widehat{\theta}-\theta_0 \;=\; \widehat{\kappa}\,(S_n' + B_n'). \] Here $\widehat{\kappa}$ is the sample analogue of the condition number, $S_n'$ is the standardized oracle sampling term, and $B_n'$ aggregates the nuisance-driven bias components that remain after orthogonalization and cross-fitting. The identity is exact. In other words, there is no Taylor approximation, because the PLR score is affine in $\theta$. This makes the role of $\widehat{\kappa}$ as a finite-sample amplification factor explicit.

Our second main result converts the exact identity into a stochastic-order bound: \[ \widehat{\theta}-\theta_0 = O_P\!\left(\frac{\sqrt{\kappa_n}}{\sqrt{n}} \;+\; \kappa_n\cdot \mathrm{Rem}_n\right), \qquad \mathrm{Rem}_n := r_n^m r_n^\ell + (r_n^m)^2 + \frac{r_n^m + r_n^\ell}{\sqrt{n}}. \] The first term is the oracle sampling component under conditioning. The second term shows that conditioning enters multiplicatively with the full nuisance remainder. As a consequence, a sufficient condition for the usual $\sqrt{n}$-approximation (oracle term dominates the remainder) is $\kappa_n\cdot \mathrm{Rem}_n = o(n^{-1/2})$. When overlap is stable so that $\kappa_n$ is bounded, this reduces to familiar remainder-rate restrictions. If overlap is weak and $\kappa_n$ is large, the same nuisance errors can be amplified into first-order bias.

Two theoretical connections improve interpretation. First, we link conditioning to semiparametric efficiency geometry by proving $\kappa = \sigma_D^2 \|\alpha_0\|_{L^2}^2$, where $\alpha_0$ is the Riesz representer chernozhukov2022riesz. Second, we formalize weakening overlap through a triangular-array framework in which $\kappa_n\to\infty$. This shows that standard $\sqrt{n}$-asymptotics may fail even when nuisance rates are favorable. The theory also yields an estimable reporting implication. We propose reporting the condition number, \[ \widehat{\kappa}_{\mathrm{oof}} = \frac{1}{1-\widehat{R}^2_{+}(D\mid X)}, \qquad \widehat{R}^2_{+} := \max\{0,\widehat{R}^2_{\mathrm{oof}}\}, \] a scale-invariant measure of residual treatment variation, where $\widehat{R}^2_{\mathrm{oof}}$ is an out-of-fold predictive $R^2$. When multiple first-stage learners are compared, $\widehat{\kappa}_{\mathrm{oof}}$ is computed learner-by-learner.\footnote{Out-of-fold $R^2$ avoids in-sample overfitting bias and aligns the conditioning diagnostic with the cross-fitting used in DML. In PLR, the amplification factor in the bound satisfies $\kappa=\sigma_D^2/\sigma_V^2=1/\{1-R^2(D\mid X)\}$, so $\widehat{\kappa}_{\mathrm{oof}}=1/\{1-\widehat R^2_{+}\}$ is a plug-in proxy for the same conditioning object.} This operationalizes the overlap/conditioning component that enters the bound and helps interpret cross-learner sensitivity predicted by the theory. Importantly, $\widehat{\kappa}_{\mathrm{oof}}$ concerns the conditioning of the orthogonalized score, but it is not a test of unconfoundedness and does not by itself validate causal identification. Analogous to weak instruments, where weak first-stage amplifies bias and motivates strength diagnostics staigerstock1997weakiv,stockyogo2005weakiv,moreira2003conditional, limited residual treatment variation in DML implies large $\kappa$, which amplifies residual nuisance bias in the orthogonal score equation.

The paper proceeds as follows. Section (ref) situates our contribution in the literature. Section (ref) develops the exact decomposition and stochastic-order bound. Section (ref) characterizes conditioning regimes and rate implications. Section (ref) presents Monte Carlo evidence validating the theoretical predictions. Section (ref) provides an empirical illustration. Section (ref) concludes.

Related Work

chernozhukov2018dml formalize DML as Neyman-orthogonal, cross-fitted score estimation, building on Robinson's partialling-out construction for PLR robinson1988plr and influence-function theory newey1990semipar. Subsequent work develops finite-sample guarantees for orthogonal-score estimators chernozhukov2023simple. kennedy2023semipar reviews influence-function--based semiparametric inference (including cross-fitted one-step/TMLE constructions and doubly robust structure) and chernozhukov2021rieszregression develops a geometric debiasing view via Riesz representers. We build on this literature, but shift attention to a typically implicit PLR regularity.

Standard orthogonal-score DML analyses pair nuisance-rate conditions with a nondegeneracy condition for the score equation. In PLR, this amounts to requiring residualized treatment variation so the score Jacobian is not close to singular chernozhukov2018dml. When this curvature deteriorates, inference becomes a weak-identification/ill-conditioning problem. Estimating equations flatten and small singular values act as error amplifiers kaji2021weakid,hanmccloskey2019singular,breunig2020illposed. Yet in applied DML this stability component is rarely tracked or reported alongside the final estimate. We make conditioning an explicit channel in PLR-DML by isolating a condition-number factor $\kappa$ that governs amplification of the orthogonal-score remainder as residual treatment variation shrinks. This links overlap-driven fragility in DML to classical collinearity diagnostics belsley1980diagnostic.

Overlap matters through residualized treatment variation, which governs both identification strength and finite-sample sensitivity. For binary treatment, efficiency theory implies sensitivity increases as propensity scores approach 0 or 1 hahn1998, motivating strict-overlap conditions and practical remedies such as trimming crumpetal2009 and overlap weighting lietal2018overlap. When overlap fails, effects may be only partially identified khanTamer2010overlap, and overlap can deteriorate with covariate dimension, creating tension between flexible adjustment and identification strength damour2021overlap. We use this overlap literature primarily to clarify the mechanism by which weak overlap undermines PLR-DML, while leaving the analysis of overlap-adjusted estimators to future work.

Modern causal estimators rely on regularized, data-adaptive learners for nuisance components, so finite-sample orthogonal-score remainders need not be negligible even with cross-fitting. High-dimensional and debiased approaches for learned nuisances are developed in bellonietal2014highdim, and sufficient conditions for valid semiparametric inference with flexible learners are studied in farrellliangmisra2021. Related robustness ideas are central in doubly robust and targeted learning methods kangschafer2007,bangrobins2005,kennedy2023semipar. These results primarily govern how small nuisance error must be. We complement them by tracking how that error is scaled by the sample geometry of the PLR score equation. Because this scaling depends on the particular out-of-fold nuisance fits, our diagnostic is inherently specification-specific for each learner.

Theoretical Results

This section introduces notation and assumptions for PLR--DML and develops our finite-sample characterization of how limited residual treatment variation affects inference.\footnote{We keep the main bound in stochastic-order form because fully nonasymptotic Gaussian approximation requires additional constants and concentration tools, such as finite-sample analyses for sample-splitting/DML-type estimators.} We present (i) an explicit link between overlap strength and a condition number $\kappa$, (ii) an exact finite-sample identity for $\widehat{\theta}-\theta_0$, and (iii) a stochastic-order bound that yields an operational sufficiency condition for $\sqrt{n}$-valid inference. We write $\kappa$ for the population condition number under a fixed law $P$, and $\kappa_n$ under a triangular-array sequence $(P_n)_{n\ge1}$.

Data Structure

Assumption map. Lemma (ref) uses only $\mathbb{E}[V\varepsilon] = 0$ and $\sigma_V^2 > 0$. The exact decomposition (Theorem (ref)) is algebraic under PLR and does not require rate conditions. Finite-sample bounds (Theorem (ref)) additionally use moment bounds, nuisance $L^2$ rates, and residual-variance stability. Regime analysis allows $\sigma_{V,n}^2 \to 0$ via the triangular array setup.

The Partially Linear Regression Model

The PLR model robinson1988plr specifies:

where $\theta_0 \in \mathbb{R}$ is the target parameter, $g_0: \mathcal{X} \to \mathbb{R}$ is a nuisance function, $m_0(X) := \mathbb{E}[D \mid X]$ is the treatment regression (conditional mean), and $V := D - m_0(X)$ is the treatment residual satisfying $\mathbb{E}[V \mid X] = 0$ by construction.

Target Parameter and Identification

Causal Setup and Identification

Causal interpretation. The estimand $\theta_0$ is causal under standard potential-outcome conditions: consistency ($Y = Y(D)$), conditional exogeneity ($\mathbb{E}[\varepsilon \mid D, X] = 0$), and overlap ($\mathrm{Var}(D \mid X) > 0$). In the PLR framework, these conditions imply the orthogonal moment $\mathbb{E}[V \cdot (Y - g_0(X) - \theta_0(D - m_0(X)))] = 0$ chernozhukov2018dml. Our analysis takes this causal target as given and studies how finite-sample inference behaves as overlap weakens (via $\sigma_V$) and $\kappa$ grows. Thus, $\kappa$ measures inferential fragility, not identification validity.

We distinguish (i) strong overlap (fixed-$\kappa$) and (ii) weakening overlap (triangular array):

To analyze regimes where $\kappa$ may grow, we introduce a triangular array framework:

Define $\ell_0(X) := \mathbb{E}[Y \mid X]$ and outcome residual $U := Y - \ell_0(X)$. Under Assumption (ref), $\ell_0(X) = \theta_0 m_0(X) + g_0(X)$.

Variance Components and the Condition Number

Define the variance components:

The population $R^2$ for treatment explained by covariates is:

Cross-Fitting and the DML Estimator

Cross-fitted residuals: $\widehat{V}_i := D_i - \widehat{m}^{(-k)}(X_i)$, $\widehat{U}_i := Y_i - \widehat{\ell}^{(-k)}(X_i)$ for $i \in I_k$. Define errors:

The PLR score is:

where $\eta = (\ell, m)$. At true values, $\psi(W; \theta_0, \eta_0) = V\varepsilon$.

The proof is in Appendix A.2. Neyman orthogonality implies the first-order effect of nuisance perturbations on the population moment vanishes. Combined with cross-fitting, the leading sample terms involving $\Delta^m, \Delta^\ell$ are mean-zero and of order $(r_n^m + r_n^\ell)/\sqrt{n}$, while the systematic remainder is second order (e.g., $r_n^m r_n^\ell$ and $(r_n^m)^2$).

The Score Jacobian

Connection to the Riesz Representer

The Exact Finite-Sample Decomposition

Define the oracle sampling term:

Variance Inflation versus Bias Amplification

The exact decomposition shows that $\widehat{\kappa}$ multiplies both the oracle sampling component and the nuisance-induced bias, but the inferential consequences are different. On the one hand, variance inflation arises through the oracle term $\widehat{\kappa}\,S_n'$, since $S_n'$ scales with $\sigma_V$ and, under the efficiency bound in Theorem (ref), the relevant benchmark variance is $V_{\mathrm{eff}}=\sigma_\varepsilon^2/\sigma_V^2$. In this case, the usual standard errors track the increased sampling variability induced by limited residual treatment variation, so coverage can remain approximately nominal. On the other hand, bias amplification arises through $\widehat{\kappa}\,B_n'$, because any remaining nuisance error is multiplied by $\kappa$.

Stochastic-Order Implications of the Finite-Sample Identity

Efficiency Bound Connection

Conditioning Regimes and Rate Implications

The regimes below classify how overlap weakens along the triangular array. Plugging regime-specific $\kappa_n$ growth into Theorem (ref) yields the corresponding rate implications. Proofs for the regime-specific rate statements are provided in Appendix A.3. From this point, we analyze sequences of DGPs under Assumption (ref), replacing Assumption (ref) with $\mathrm{Var}(D_n \mid X_n = x) \geq \underline{\sigma}_n^2$ where $\underline{\sigma}_n^2 \downarrow 0$.

We interpret $\widehat{\kappa}_{\mathrm{oof}}$ through qualitative conditioning regimes that guide reporting practices rather than impose strict cutoffs. In well-conditioned scenarios, the orthogonal score exhibits sufficient curvature to ensure that the estimating equation is well-posed. In this case, residual nuisance error is not substantially amplified. As conditioning deteriorates, the same nuisance error becomes increasingly magnified. Consequently, the sensitivity of cross-learner and nuisance specifications serves as direct, observable indicator of estimator fragility and should be reported. In severely ill-conditioned contexts, the sampling term may remain small while an amplified systematic component dominates the estimator. Under such circumstances, conventional confidence intervals tend to under-cover.

A recommended workflow involves computing $\widehat{\kappa}_{\mathrm{oof}}$ for each learner and reporting these values alongside a multi-learner sensitivity summary, as demonstrated in the empirical illustration. Notably, $\widehat{\kappa}_{\mathrm{oof}}$ functions as a diagnostic for the conditioning of the orthogonalized score, rather than as evidence for unconfoundedness or causal validity. When $\widehat{\kappa}_{\mathrm{oof}}$ is elevated, estimates should be regarded as fragile. Selection of a single preferred learner should be avoided. Overlap adjustments represent natural complementary responses in applied research, although a formal analysis of these remedies is beyond the scope of this paper.

Monte Carlo Evidence

We validate the theoretical predictions via a corrupted oracle design that isolates the amplification mechanism. Predictions P1--P3 follow from Theorems (ref)--(ref):

We generate $n = 2{,}000$ observations from the PLR model with $\theta_0 = 1$, $X \in \mathbb{R}^{10}$ drawn from $N(0, \Sigma)$ with AR(1) correlation $\Sigma_{jk} = 0.5^{|j-k|}$ and $U, \varepsilon \stackrel{\text{i.i.d.}}{\sim} N(0, 1)$. The treatment equation is $D = m_0(X) + \sigma_U U$ where $m_0(X) = \beta^\top X$ with $\beta_j = 0.7^j$ and $\sigma_U$ calibrated to target $R^2(D|X) \in {0.50, 0.75, 0.90, 0.95, 0.97, 0.99}$, yielding $\kappa \in {2, 4, 10, 20, 33, 100}$. The corrupted oracle injects multiplicative bias: $\widehat{m}(X) = m_0(X)(1 + \delta)$ and $\widehat{\ell}(X) = \ell_0(X)(1 + \delta)$ for $\delta \in {0, 0.02, 0.05, 0.10, 0.20}$.

Figure (ref) visualizes the bias-amplification mechanism. For fixed nuisance error magnitude, increasing conditioning (higher $\kappa$) magnifies the resulting estimation error. Panel (a) confirms Prediction P1. For each fixed $\delta$, bias increases monotonically with $\kappa$, and the approximately parallel lines across $\delta$ levels illustrate the multiplicative structure. Each unit increase in $\log \kappa$ shifts $\log|\text{Bias}|$ by roughly the same amount regardless of $\delta$. Panel (b) confirms Prediction P2. When bias is plotted against the single index $\kappa\times\delta$, the points align along a single monotone trend, and the pooled log--log regression

captures this organization. The estimated slope below one is consistent with the finite-sample theory. The remainder term in Theorem (ref) contains both linear and quadratic components (e.g., $r_n^m r_n^\ell+(r_n^m)^2$), and the presence of a nonzero noise floor at $\delta=0$ mechanically flattens the log--log relationship for small values of $\kappa\times\delta$. Table (ref) further reports separate elasticities, reinforcing the finite-sample amplification mechanism.

Figure (ref) shows that coverage remains near nominal in well-conditioned regimes, but collapses sharply as $\kappa$ increases for fixed $\delta$, consistent with P3. At $\delta = 0$, coverage ranges from 0.94 to 0.97 across $\kappa$ (see Table (ref) for the full grid). Table (ref) reports the structural $\kappa$ values associated with each $R^2$ regime and confirms $\kappa$ is invariant to injected bias. Coverage decreases primarily because $|\text{Bias}|/\text{SE}$ increases with $\kappa \times \delta$, not because SE inflates. This is a silent failure where CIs are narrow but miss $\theta_0$.

Panel (b) reveals why coverage fails: the ratio $|\text{Bias}|/\text{SE}$ crosses 1.0 precisely when coverage drops below 80%. This confirms that undercoverage is due to bias amplification, not variance inflation. Sample-size sensitivity is summarized in Table (ref).

The Monte Carlo evidence confirms all three predictions. A sign-structure experiment in Table (ref) validates the cross-term structure of Lemma (ref). Opposite-sign nuisance errors produce $3.7\times$ larger bias and near-zero coverage at high $\kappa$, confirming that asymmetric regularization is more damaging than symmetric bias. Robustness to nonlinear DGPs appears in Table (ref).

When $\kappa$ is large, small systematic nuisance errors can dominate sampling noise. In this case, bias is amplified beyond what standard errors capture. The single-index $\kappa \times \delta$ organizes when coverage collapses, making $\kappa$ a fragility predictor. Instability across learner implementations is itself informative about estimation sensitivity. For practice, the main implication is reporting-oriented rather than threshold-based. Large estimated $\kappa$ should be interpreted as a fragility flag that motivates transparent sensitivity analysis.

Empirical Application

We demonstrate our proposed reporting workflow in the LaLonde lalonde1986 benchmark, using the NSW experimental sample and the NSW--PSID observational comparison design studied by dehejiawahba1999. The purpose of this section is diagnostic and interpretation rather than causal adjudication. In particular, the observational benchmark cannot disentangle implementation fragility induced by weak residualized treatment variation (a conditioning problem) from genuine violations of causal assumptions (an identification problem). Our goal is instead to show how the out-of-fold conditioning diagnostic $\hat\kappa_{\mathrm{oof}}$ behaves in a standard applied PLR--DML pipeline. Throughout we use $K=5$ cross-fitting and run PLR--DML across six learners (OLS, Lasso, Ridge, RF, GBM, MLP).\footnote{ For simplicity, we use the same learner class for both $\widehat m(X)$ and $\widehat \ell(X)$. Mixed learner choices are straightforward but not pursued here.}

Table (ref) reports $\hat\theta$, standard errors, confidence intervals, the out-of-fold first-stage fit $\hat R^2_{\mathrm{oof}}(D\mid X)$ and the implied conditioning diagnostic $\hat\kappa_{\mathrm{oof}} = 1/(1-\hat R^2_+)$ with $\hat R^2_+=\max\{0,\hat R^2_{\mathrm{oof}}(D\mid X)\}$. In the experimental NSW sample, $\hat R^2_{\mathrm{oof}}(D\mid X)$ is near zero, so $\hat\kappa_{\mathrm{oof}}\approx 1$ for all learners and estimates are comparatively stable across nuisance specifications. In the observational NSW--PSID sample, more flexible learners achieve higher out-of-fold first-stage fit, implying larger $\hat\kappa_{\mathrm{oof}}$. These same specifications exhibit some larger cross-learner dispersion, including sign reversals, despite confidence interval widths that do not mechanically expand one-for-one with dispersion. This is the qualitative empirical pattern predicted by our exact PLR identity and bound.\footnote{In many observational settings $\hat R^2_{\mathrm{oof}}(D\mid X)$ can be materially higher than in Table (ref), so $\hat\kappa_{\mathrm{oof}}$ may be much larger and the amplification mechanism correspondingly more pronounced. We use the LaLonde benchmark because it admits comparison to an experimental benchmark estimate. As noted above, the observational comparison does not disentangle conditioning-driven fragility from identification failures.} As residualized treatment variation weakens, amplification can make otherwise second-order implementation differences (learner class, tuning, splitting, and regularization) economically consequential for $\hat\theta$.

Accordingly, we interpret $\hat\kappa_{\mathrm{oof}}$ narrowly as a learner-specific proxy for orthogonal-score conditioning. Our recommendation is procedural and reporting-oriented: (a) pre-specify a menu of reasonable nuisance learners and tuning rules (including cross-fitting and tuning conventions); (b) compute $\hat\kappa_{\mathrm{oof}}$ learner-by-learner from the same out-of-fold residuals that enter the cross-fitted score; and (c) report $\hat\kappa_{\mathrm{oof}}$ together with a compact multi-learner sensitivity summary, such as a forest plot (see Figure (ref)). When $\hat\kappa_{\mathrm{oof}}$ is elevated and dispersion is large, nominal standard errors can understate uncertainty because amplification may dominate sampling noise.

Conclusion

We show that, in PLR-DML, conditioning of the orthogonal score is a first-order amplification channel for finite-sample error. The exact identity $\hat\theta-\theta_0=\widehat{\kappa}(S_n' + B_n')$ makes the multiplicative role of the condition number explicit, and the stochastic bound $\hat\theta-\theta_0 = O_P(\sqrt{\kappa_n/n}+\kappa_n\mathrm{Rem}_n)$ yields an operational sufficiency condition: $\kappa_n\mathrm{Rem}_n=o(n^{-1/2})$. For practice, we recommend reporting $\hat\kappa_{\mathrm{oof}}:=1/\{1-\max(0,\hat R^2_{\mathrm{oof}}(D\mid X))\}$ alongside estimates. A large $\hat\kappa_{\mathrm{oof}}$ indicates fragility. In this case, narrow CIs should be interpreted cautiously and multi-learner sensitivity summaries are recommended.

Our results are exact for PLR. In more general orthogonal-score problems, the diagnostic framework extends by linearizing the score equation and tracking both (i) the score Jacobian (the stability object) and (ii) the associated linearization remainder. Appendix A.1 provides the generic orthogonal-score identity that anchors this template. Developing weak-conditioning-robust inference procedures, and extending the analysis to heterogeneous treatment effects and other orthogonal-score settings, are promising directions.

Disclosure Statement

The author has no conflicts of interest to declare.

Data Availability Statement

Replication code and data are available at \url{https://github.com/gsaco/dml-diagnostic}.

Acknowledgments

I used Grammarly for language editing. All remaining errors are my own.